What makes a great test?
Ideally tests would be fast, reliable, easy to own, easy to maintain, etc. Let’s make a list that we can use when creating, improving, or evaluating our tests.
|At least 1000 test cases per second per CPU core.
|No network, no disk, no sleeping.
|Multiple runs, with no changes, always give the same results.
|No network, no race conditions, no system prerequisites.
|Can run any 2 tests at the same time with no interference
|Order independent, leaves system unchanged.
|When a test fails, it is immediately obvious why.
|Tests are well-named and specific. Assertions are rich and informative.
|A human familiar with the domain (even non-programmers) can read the test name and test body and understand why this test case matters.
|Code under test must use in-domain names and expose the right level of detail.
|If I wonder “is there a test for the for this requirement?” I can easily find the corresponding test case (or see that it does not exist).
|May break the 1 test :: 1 class relationship
|Does not impede refactoring
|Does not depend on implementation details. Changing code in ways that doesn’t affect visible behavior is not made harder by the need to update tests.
|No mocks (except for TDA.)
|Only runs what it’s trying to test
|No test helpers. No mocks. No setup that’s not about this test case.
|1 defect == 1 test failure
|If I introduce a single defect, only a single test should fail
|No n-tier architecture, no “core/common” code that is unowned or owned by a platform team.
|A passing test indicates that something of business value is correct and present in the system. A failing test tells us that something of business value is missing or broken.
|Easy to write
|The test framework and tooling and language and norms all make it easy to create a new test without a lot of rigamarole.
(You may notice a lot of overlap with “FIRST-ness”.)
Written on July 8, 2021