What makes a great test?
Ideally tests would be fast, reliable, easy to own, easy to maintain, etc. Let’s make a list that we can use when creating, improving, or evaluating our tests.
|Fast||At least 1000 test cases per second per CPU core.||No network, no disk, no sleeping.|
|Reliable||Multiple runs, with no changes, always give the same results.||No network, no race conditions, no system prerequisites.|
|Independent||Can run any 2 tests at the same time with no interference||Order independent, leaves system unchanged.|
|Diagnosable||When a test fails, it is immediately obvious why.||Tests are well-named and specific. Assertions are rich and informative.|
|Readable||A human familiar with the domain (even non-programmers) can read the test name and test body and understand why this test case matters.||Code under test must use in-domain names and expose the right level of detail.|
|Organized||If I wonder “is there a test for the for this requirement?” I can easily find the corresponding test case (or see that it does not exist).||May break the 1 test :: 1 class relationship|
|Does not impede refactoring||Does not depend on implementation details. Changing code in ways that doesn’t affect visible behavior is not made harder by the need to update tests.||No mocks (except for TDA.)|
|Only runs what it’s trying to test||No test helpers. No mocks. No setup that’s not about this test case.|
|1 defect == 1 test failure||If I introduce a single defect, only a single test should fail||No n-tier architecture, no “core/common” code that is unowned or owned by a platform team.|
|Relevant||A passing test indicates that something of business value is correct and present in the system. A failing test tells us that something of business value is missing or broken.|
|Easy to write||The test framework and tooling and language and norms all make it easy to create a new test without a lot of rigamarole.|
(You may notice a lot of overlap with “FIRST-ness”.)
Written on July 8, 2021