What makes a great test?
Ideally tests would be fast, reliable, easy to own, easy to maintain, etc. Let’s make a list that we can use when creating, improving, or evaluating our tests.
| Name | Definition | Notes |
|---|---|---|
| Fast | At least 1000 test cases per second per CPU core. | No network, no disk, no sleeping. |
| Reliable | Multiple runs, with no changes, always give the same results. | No network, no race conditions, no system prerequisites. |
| Independent | Can run any 2 tests at the same time with no interference | Order independent, leaves system unchanged. |
| Diagnosable | When a test fails, it is immediately obvious why. | Tests are well-named and specific. Assertions are rich and informative. |
| Readable | A human familiar with the domain (even non-programmers) can read the test name and test body and understand why this test case matters. | Code under test must use in-domain names and expose the right level of detail. |
| Organized | If I wonder “is there a test for this requirement?” I can easily find the corresponding test case (or know that it does not exist). | May break the 1 test :: 1 class relationship |
| Does not impede refactoring | Does not depend on implementation details. Changing code in ways that doesn’t affect visible behavior is not made harder by the need to update tests. | No mocks (except for TDA.) |
| Only runs what it’s trying to test | No test helpers. No mocks. No setup that’s not about this test case. | |
| 1 defect == 1 test failure | If I introduce a single defect, only a single test should fail | No n-tier architecture, no “core/common” code that is unowned or owned by a platform team. |
| Relevant | A passing test indicates that something of business value is correct and present in the system. A failing test tells us that something of business value is missing or broken. | |
| Easy to write | The test framework and tooling and language and norms all make it easy to create a new test without a lot of rigamarole. |
(You may notice a lot of overlap with “FIRST-ness”.)
Written on July 8, 2021