The Benefits of Automated Testing

Software changes over time. Your company’s web site this year does things you had no idea you needed two years ago. How can we keep the cost of change manageable?

Automated testing is one way. Especially for large software projects, the practice of automated testing can dramatically reduce the cost of adapting software to new business necessities.

As a developer, when I upgrade to a new version of software or add a new feature, automated tests tell me right away when I’ve broken a working feature. Automated tests find bugs instantly. Manual testing can also find bugs, but it’s slower and more expensive. So professional software development teams use automated tests extensively.

 

Sometimes we can get to a deadline sooner by leaving out automated tests, but it increases the cost of development from then on,at least until we take the time to write the automated tests. We call shortcuts like going without tests “technical debt” — taking those shortcuts in development can speed you up in the short term, but then you’re stuck “paying the interest” on the debt in the form of extra time and cost.

Predicting and Avoiding Unpredictability

One of the most difficult issues in software project management is the strong>inherent unpredictability of software project schedules. The only predictable task, in any field, is one you’ve already done before. But in software, if you’re writing code that’s been written before, you’re doing it wrong — you should just copy the existing files into the new project!

As Joel Spolsky said in 2001:

This means that almost everything we do as software developers is
something that has never been done before. Herman the Handyman,
who just installed a tile floor for me, has probably installed
hundreds of tile floors. He has to keep installing tile floors again
and again as long as new tile floors are needed. We in the software
industry would have long since written a Tile Floor Template Library
(TFTL) and generating new tile floors would be trivial.
… Everything you do is on the cutting edge by definition. So by
definition it is unpredictable.

In the face of this inherent unpredictability, we can at least remove much of the avoidable unpredictability. One of the biggest causes of avoidable unpredictability is tracking down bugs. An unexpected nasty bug can blow your development budget and your schedule.  Because automated tests let you find bugs quickly, tests reduce the risks of software project budgets.

What Programmers Do, and Why Catching Bugs Faster is Such a Big Deal

A lot of people think that software development consists mostly of typing code into a computer, with the occasional error that needs to be fixed. A little arithmetic will show how wrong that is.

Nearly all programmers are adept typists, capable of speeding along at 90 words per minute. A line of code is typically about five words worth, so if a programmer is just typing in code from a printout, he or she can type about 18 lines of code per minute, or 1080 lines per hour, or 8640 lines per day.

But, on real-life software projects, programmers might produce between 3 and 30 lines of code per day,
or up to 100 lines in extreme circumstances on small projects. So they’re spending at least 99% of their time doing things other than typing in production code.

Where does that 99% of the time go? It’s spent on understanding what the existing code does (this alone is about 50%), on figuring out what the new code needs to do, on organizing the code, on testing to see what the code does, and on hunting down bugs in the code to make it work right. In particular, people often underestimate the enormous amount of time even the best programmers have to spend ferreting out their own mistakes.

How Automated Testing Works

I have a bug to fix. I load up the test runner and push the Run button, and a green progress bar showing the progress of the run of its test suite appears. A few seconds or minutes later, the test run is complete, and I see that the bar is still green, so the system is in working order on my workstation. I add a new test, which consists of writing a few lines of code that demonstrate the bug I’m going to fix;  I hit Save; and I hit Run again. This time, the bar turns red, and shows him that the test I’ve just written is failing. I fix the bug, push Run again, and the tests go green again — telling me not only that my fix worked, but also that I didn’t break anything else in the system in the process.

There are some things about this scenario that are crucial to getting
the benefits of automated testing:

  • First, I didn’t have to wait long to get the results of the automated test run. If it took ten minutes, or an hour, or overnight, to get the results, I wouldn’t run the test suite before I started developing, and there’s a possibility that the code or the test suite would already be buggy. When I finally ran the test suite, and the tests went red, I’d think I had introduced a bug in an unrelated part of the system, and I’d waste time trying to figure out how I could possibly have broken it. For some projects, it’s worthwhile having a separate suite of “slow tests”, which can be run less frequently, but it’s crucial to keep the normal test run snappy in order to make it useful.
  • Second, I didn’t have to inspect the output of the test run in order to figure out whether the tests passed or failed. The test suite contains not just the script to exercise the system, but also the expected output, and I get a simple red/green indicator that everything is okay, out of a suite that may be thousands or tens of thousands of individual tests.
  • Third, I didn’t have to wait for a separate Testing Team to write the test. QA teams are very useful for many software projects, but some cases, it’s efficient for the developer to write the test, even if the QA team found the bug originally.
  • Fourth, I tested the test. I verified that if the bug was present, the test failed.

In this case, I’m testing the test by writing the test before the code that’s supposed to make it pass. This approach is called Test-Driven Development (TDD) or Behavior-Driven Development,and it’s widely believed to improve code quality and speed up development, for a variety of reasons. But, while TDD is more convenient in many cases, I still get a lot of benefit from automated tests written after the fact.

Of course, it’s still possible that I could introduce a bug that the existing test suite won’t catch. I still have to be careful and think about what I’m doing. But a good test suite will catch almost all of
my slip-ups.

Conclusions

There are other benefits of automated tests besides just reducing the number of bugs. A kind of automated test called an “acceptance test”serves not only to catch bugs, but also to document customer requirements. There are special tools for maintaining automatic acceptance tests in a form that’s understandable and even editable by non-technical customers:  FitNesse and Cucumber are two of the most popular.

Automated “unit tests” also provide a similar value for reusable modules of code, documenting what those modules do, and don’t do, to future programmers — as well as documenting how to use them — enhancing the practical reusability of those modules, reducing future development effort. And test-driven development is widely believed to improve the internal design of software-something that’s invisible to users, but reduces the cost of later changes.

Automating the tests that can be automated means that human QA testers can focus their effort on finding trickier bugs,including problems with usability, which gives customers much more value per QA hour spent. Many consider these benefits more important than merely catching bugs.

Finally, it’s not enough to know that automated tests are a good idea; how do you decide what to test, and how much testing is too much? Hold that thought; we’ll discuss these topics in later blog posts.

Leave a Reply