What is test coverage, and why is 100% coverage not necessarily a sufficient quality metric?
Coverage measures executed lines, not asserted behavior; it is a lower bound on risk
Coverage counts which lines, branches, or statements were executed during a test run. Line coverage tells you a line ran; branch coverage tells you both sides of a conditional were taken. Neither tells you whether the test asserted anything meaningful about the result. A test that calls a function and asserts nothing can achieve 100% line coverage and still miss every bug. Coverage is best used as a diagnostic for finding untested areas and as a ratchet to prevent regressions in coverage percentage, not as a target. High coverage with weak assertions gives false confidence, while lower coverage concentrated on critical paths with strong assertions is often better. Also be aware that coverage tools trace execution, which adds overhead and can miss behavior triggered only by threading, subprocesses, or error paths that never run in the test suite.
- 1
Line coverage: which statements executed. Branch coverage: which outcomes of conditionals were taken. Path coverage is rarely practical.
- 2
Mutation testing, such as mutmut or cosmic-ray, is a stronger signal because it checks whether tests fail when code is subtly changed.
- 3
Assertion-free tests inflate coverage. Review tests for meaningful assertions, not just execution.
- 4
Coverage cannot detect missing requirements. A feature can be fully uncovered by tests and still show 100% coverage of the code that exists.
- 5
Trade-off: chasing 100% coverage slows delivery and encourages gaming, while ignoring coverage leaves blind spots.
- 6
Common mistake: enforcing a global percentage threshold across all modules regardless of criticality. Thresholds should be higher on core logic and lower on glue code.
- 7
Version note: coverage.py supports branch coverage and subprocess coverage via COVERAGE_PROCESS_START. Combining with pytest-cov is standard practice.
Scenario Questions
0-2 years experience
- 1A module shows 100% line coverage but still ships a bug. How is that possible?
- 2What is the difference between line coverage and branch coverage?
2-5 years experience
- 1Your team enforces 100% coverage and developers write tests that assert nothing. How do you fix the incentive?
- 2You have 40% coverage concentrated on the payment module. Is that better or worse than 80% spread evenly?
5-8 years experience
- 1You want a stronger correctness signal than coverage. How do you introduce mutation testing and what score is meaningful?
- 2Some code only runs in subprocesses, so coverage reports 0% for it. How do you measure it correctly?
8+ years experience
- 1Design a quality strategy for a regulated codebase that combines coverage ratchets, mutation testing, property-based tests, and contract tests.
- 2Explain how coverage, mutation score, and flakiness metrics interact, and how you would present them to leadership without encouraging gaming.
Follow-up Questions
- How does branch coverage differ from line coverage and when is it worth enforcing?
- How would you introduce mutation testing into a team without slowing down every pull request?