We're writing lots of tests for a current client project that's going to be open sourced. I understand the mechanics of writing unit tests but always wondered if my tests could be simpler and more thorough, and worried about missing things. I was curious to learn if I could write better tests, so I did a quick literature review. It was a valuable exercise as I did get a deeper understanding of how to think about tests and picked up a few super useful tips.
The following are my notes from The Art of Readable Code, The Pragmatic Programmer, and notes I took from a session by Dan Vanderkam. I found Readable Code to have the most useful and concrete advice that immediately made my tests better. Pragmatic Programmer took a more high-level philosophical approach, and spent a good deal of its page space convincing us that testing is a good idea, like this pull quote: "Test your software, or your users will."
Why is it so important my test code is readable?
You want future developers to write their own tests, so you don't want them to be intimidated by the test code. You want to make it super easy for them to add more tests.
Tests serve as documentation. Unlike comments, they have to stay in sync with the code as it changes.
How many tests should I write?
There are tools that keep track of how many lines of code in your codebase have been executed when you run your test suite, and then calculate a percent coverage.
Take this number with a grain of salt—it may not be worth the time cost to cover every single line of code. Balance that cost against the project requirements and consequences of a bug.
"Write tests until fear turns to boredom"
Look at the contract (the description of inputs and outputs) for your function. Make sure unacceptable inputs are rejected, and then that acceptable inputs produce the right outputs.
If you're testing modules that depend on other modules, thoroughly test the submodules first. Then, test the modules that depend on the submodules, since you can rely on the submodules tests to make sure its contracts are met.
How can I make my individual tests better?
Use simple input data. If you want to test negative numbers, don't use -9999.87. Just use -1. If you want to test a long array, don't write out the literal, generate it programatically. Make it obvious to the reader what your concern is for each test.
Each test should probe one concern. E.G. don't test both sorting and handling of negative numbers in one test. First test negative number handling, then test sorting with only positive numbers. Separate your concerns.
Don't forget edge cases, like empty inputs.
Hide less important details—make the tests as simple as possible. One way to do this is to do some work up front setting up input data structures or helper functions, so that the actual test can be just one line of code. This is upfront work that can be shared across many tests, which in turn makes it easier for people to add more tests to the codebase!
How can I make it easier to fix tests when they fail?
Aim for helpful error messages. Some testing frameworks print both inputs and outputs of failed tests.
Long function names are ok. They won't be used around the codebase so there's no reason to keep them short. Also, if their names are descriptive, that will help you track down failing tests more easily from the error messages.
When should I add tests down the road?
Whatever you're doing in a REPL (to debug or test) should turn into a test.
Catch a bug? Write a test (called a 'regression test') first, then fix the bug. You might say "it won't happen again" but it will.
What helpful tools should I use?
Use git commit hooks to run your test suite with every commit.
Use a testing library. Think twice about coding your own. Many languages have testing frameworks already in place that provide powerful tools. For example, Python has unittest, doctests, and nose.
How can tests actually improve my code?
Unit tests are a good canary for well-written code. If tests are hard to write, you may want to refactor your code.
Taken to its logical extension, the philosophy of test-driven development (TDD) argues that you will construct better code if you write the tests first. Code that's easy to test can't be in huge interconnected pieces, so testing first forces you to design small, orthogonal components.
Brief update: I'm reading the tests over at scikit-learn, which tests similar things to my project and has a nice clean data set for testing its extraction functions.