Software changes over time. Your company’s web site this year does things you had no idea you needed two years ago. How can we keep the cost of change manageable?
Automated testing is one way. Especially for large software projects, the practice of automated testing can dramatically reduce the cost of adapting software to new business necessities.
As a developer, when you upgrade to a new version of software or add a new feature, automated tests tell you right away when you’ve broken a working feature. Automated tests find bugs instantly. Manual testing can also find bugs, but it’s slower and more expensive. So professional software development teams use automated tests extensively.
Sometimes we can get to a deadline sooner by leaving out automated tests, but it increases the cost of development from then on,at least until we take the time to write the automated tests. We call shortcuts like going without tests “technical debt” — taking those shortcuts in development can speed you up in the short term, but then you’re stuck “paying the interest” on the debt in the form of extra time and cost.
Predicting and Avoiding Unpredictability
One of the most difficult issues in software project management is the strong>inherent unpredictability of software project schedules. The only predictable task, in any field, is one you’ve already done before. But in software, if you’re writing code that’s been written before, you’re doing it wrong — you should just copy the existing files into the new project!
This means that almost everything we do as software developers is
something that has never been done before. Herman the Handyman,
who just installed a tile floor for me, has probably installed
hundreds of tile floors. He has to keep installing tile floors again
and again as long as new tile floors are needed. We in the software
industry would have long since written a Tile Floor Template Library
(TFTL) and generating new tile floors would be trivial.
… Everything you do is on the cutting edge by definition. So by
definition it is unpredictable.
In the face of this inherent unpredictability, we can at least remove much of the avoidable unpredictability. One of the biggest causes of avoidable unpredictability is tracking down bugs. An unexpected nasty bug can blow your development budget and your schedule. Because automated tests let you find bugs quickly, tests reduce the risks of software project budgets.
What Programmers Do, and Why Catching Bugs Faster is Such a Big Deal
A lot of people think that software development consists mostly of typing code into a computer, with the occasional error that needs to be fixed. A little arithmetic will show how wrong that is.
Nearly all programmers are adept typists, capable of speeding along at 90 words per minute. A line of code is typically about five words worth, so if a programmer is just typing in code from a printout, he or she can type about 18 lines of code per minute, or 1080 lines per hour, or 8640 lines per day.
But, on real-life software projects, programmers might produce between 3 and 30 lines of code per day,
or up to 100 lines in extreme circumstances on small projects. So they’re spending at least 99% of their time doing things other than typing in production code.
Where does that 99% of the time go? It’s spent on understanding what the existing code does (this alone is about 50%), on figuring out what the new code needs to do, on organizing the code, on testing to see what the code does, and on hunting down bugs in the code to make it work right. In particular, people often underestimate the enormous amount of time even the best programmers have to spend ferreting out their own mistakes.
How Automated Testing Works
You have a bug to fix. Load up the test runner and push the Run button, and a green progress bar showing the progress of the run of its test suite appears. A few seconds or minutes later, the test run is complete, and you see that the bar is still green, so the baseline system is in working order on your workstation. Now onto the bug…
Add a new test, which consists of writing a few lines of code that demonstrate the bug you’re going to fix; hit Save; then hit Run again. This time, the bar turns red, and shows that the test you’ve just written is failing. You fix the bug, push Run again, and the tests go green again — telling you not only that your fix worked, but also that you didn’t break anything else in the system in the process.
There are some things about this scenario that are crucial to getting the benefits of automated testing:
- First, you didn’t have to wait long to get the results of the automated test run. If it took ten minutes, or an hour, or overnight, to get the results, you wouldn’t run the test suite before you started developing, and there’s a possibility that the code or the test suite would already be buggy. When you finally ran the test suite, and the tests went red, you’d think you had introduced a bug in an unrelated part of the system, and you’d waste time trying to figure out how you could possibly have broken it. For some projects, it’s worthwhile having a separate suite of “slow tests”, which can be run less frequently, but it’s crucial to keep the normal test run snappy in order to make it useful.
- Second, you didn’t need to inspect the output of the test run in order to figure out whether the tests passed or failed. The test suite contains not just the script to exercise the system, but also the expected output. You get a simple red/green indicator that everything is okay, out of a suite that may be thousands or tens of thousands of individual tests.
- Third, you didn’t have to wait for a separate Testing Team to write the test. QA teams are very useful for many software projects, but some cases, it’s efficient for the developer to write the test, even if the QA team found the bug originally.
- Fourth, you tested the test. You verified that if the bug was present, the test failed.
In this case, you’re testing the test by writing the test before the code that’s supposed to make it pass. This approach is called Test-Driven Development (TDD) or Behavior-Driven Development (BDD), and it’s widely believed to improve code quality and speed up development, for a variety of reasons. But, while TDD/BDD is more convenient in many cases, you can still get benefits from automated tests written after the fact.
Of course, it’s always possible that you can introduce a bug that the existing test suite won’t catch. You still have to be careful and think about what you’re doing. But a good test suite will catch almost all of the potential slip-ups.
Conclusions
There are other benefits of automated tests besides just reducing the number of bugs. A kind of automated test called an “acceptance test” serves not only to catch bugs, but also to document customer requirements. There are special tools for maintaining automatic acceptance tests in a form that’s understandable and even editable by non-technical customers.
Automated “unit tests” also provide a similar value for reusable modules of code, documenting what those modules do, and don’t do, to future programmers — as well as documenting how to use them — enhancing the practical reusability of those modules, reducing future development effort. And test-driven development is widely believed to improve the internal design of software-something that’s invisible to users, but reduces the cost of later changes.
Automating the tests that can be automated means that human QA testers can focus their effort on finding trickier bugs,including problems with usability, which gives customers much more value per QA hour spent. Many consider these benefits more important than merely catching bugs.
Finally, it’s not enough to know that automated tests are a good idea; how do you decide what to test, and how much testing is too much? Hold that thought; we’ll consider these as potential topics for future discussion.
Editor’s note: This blog post was originally published by Beezwax in 2012, with contributions from several Beezwax team members. And yet, as we republish it, very few updates were made, simply because automated testing, and TDD/BDD still holds much of the same value as it did back then.