The Swamp of Regression Testing
Why? Because otherwise you’re bound to get bogged down by the swamp of regression testing.
Regression Testing in Incremental Development
Suppose we’re building a system using a traditional method; We design it, build it and then test it – in that order. For the sake of simplicity assume that the effort of acceptance testing the system is illustrated by the box below.
The amount of acceptance testing using a traditional approach
Now, suppose we’re creating the same system using an agile approach; Instead of building it all in one chunk we develop the system incrementally, for instance in three iterations.
An agile approach. The system is developed in increments.
New functionality is added with each iteration
Potential Releasability
Now, since we’re “agile”, acceptance testing the new functionality after each iteration is not enough. To ensure “potential releasability” for our system increments, we also need to check all previously added functionality so that we didn’t accidentally break anything we’ve already built. Test people call this regression testing.
Can you spot the problem? Incremental development requires more testing. A lot more testing. For our three iteration big example, twice as much testing.
The theoretical amount of acceptance testing using an agile approach and three iterations, is twice as much as in Waterfall
Unfortunately, it’s even worse than that. The amount of acceptance testing increases exponentially with the number of increments. Five iterations requires three times the amount of testing, nine requires five times, and so on.
The amount of acceptance testing increase rapidly with the number of increments.
This is the reason many agile teams lose velocity as they go, either by yielding to an increasing burden of test (if they do what they should) or by spending an increasing amount of time fixing bugs that cost more to fix since they’re discovered late. (Which is the symptom of an inadequately tested system.)
A Negative Force
Of course, this heavily simplified model of acceptance testing is full of flaws. For example:
- The nature of acceptance testing in a Waterfall method is very different from the acceptance testing in agile. (One might say though that the Waterfall hides behind a false assumption (that all will go well) which looks good in plans but becomes ugly in reality).
- One can’t compare initial acceptance testing of newly implemented features, with regression testing of the same features. The former is real testing, while the second is more like checking. (By the way, read this interesting discussion on the difference of testing and checking: Which end of the horse.)
Still, I think it’s safe to say that there is a force working against us, a force that gets greater with time. The burden of regression testing challenges any team doing incremental development.
So, should we drop the whole agile thing and run for the nearest waterfall? Of course not! We still want to build the right system. And finding that out last is not an option.
The system we want is never the system we thought we wanted.
OK, but can’t we just skip the regression testing? In practice, this is what many teams do. But it’s not a good idea. Because around The Swamp of Regression Testing lies The Forest of Poor Quality. We don’t want to go there. It’s an unpleasant and unpredictable place to be.
We really need to face the swamp. Otherwise, how can we be potentially releasable?
The only real solution is to automate testing. And I’m not talking about unit testing here. Unit testing is great but has a different purpose: Unit-testing verifies that the code does what you designed it to do. Acceptance testing on the other hand verifies that our code does what the customer (or product owner, please use your favorite word for the one that pays for the stuff you build) asked for. That’s what we need to test to achieve potential releasability.
So, get out there and find a way to automate your acceptance testing. Because if you don’t, you can’t be agile.
Cheers!
Yes, I’ve seen this again and again over the years. You need to do a lot of integration testing, and if you invest enough time in automating it, you can save massive amounts of time as the project progresses (and keep development from getting bogged down). The first time I saw this in practice was the symbolic computation engine, Maple. They had fully automated all of the language testing, which allowed us to port Maple to a huge number of platforms in a very short time, and we had the confidence that the ports were correct.
Later I implemented the same pattern for a very complex caching system, which eventually ran for years with only one bug slipping through. A while ago I used Apache jmeter to pound an interface and caught one big bug within the first day of work. Lately I’ve been building another test environment to control semi-automated hardware testing and allow data-mining. I’ve found that automated system testing takes a bit of work, but is hugely beneficial.
Knowing that, I’ve often felt that the focus on unit testing is too much. It’s great for some isolated pieces, but to have real confidence (and save time) the tests have to occur at a higher level. Knowing that the unit works is not as important as knowing that it works in a specific context (the system). If programmers burn all their energy on the former, then don’t have any resources left for the later.
Paul.
Hi Paul!
I agree that the balance between the amount of work put into automatic unit testing and acceptance testing is a bit out of proportion. I guess one reason could be that unit testing is a lot easier.
The two goes together though, but if I had to choose between automating one or the other I’d go with acceptance tests. For the reasons given in the above post.
Cheers!
I’m with you 🙂
I was thinking that in most programs, the heart of the system — the stuff that drives it and makes it complex — is usually about 20% or less of the code. I think putting unit tests on that code makes a lot of sense, but the other 80% is usually just stock-standard stuff that moves the data back and forth between the user and the database, much of it for simple administration screens. I’ve seen people call for 100% coverage of unit tests, but with the above in mind, I’d think that exceeding 20% is starting to burn calories without enough justification. Just a thought 🙂
I agree!
One thing I’ve tried with really good result is to skip regression test in every sprint that doesn’t end up in production. Yes, perhaps that is cheating according to Scrum but it worked for us.
Every time the team was about to deliver the system to production we wrote a story for all the stuff like release notes, regression test and so on. This was really a success and we saved a lot of time.
Sorry, perhaps I missed something. So, how can we automate acceptance testing?
@Paul W. Homer
I generally agree with you, although I’d put my “sweet-spot” a bit higher. In my experience close to 80% of the code justifies unit-testing. Then of course I assume that the tests are written first, so that they’re used as a design tool and not only there for safety net reasons.
@Linda Hoff
Hi Linda, thank’s for your comment!
I have used the exact same strategy with my teams, whose automation effort I used to consider being too high (I now do a different evaluation, as my post points out).
It seemed to be a good compromise to wait with acceptance testing until production release, but it got the exact same inherent problem of an exponentially increasing test burden; we have only mitigated the problem with bigger increments (releases), which flattens the curve a bit. But the fact is this: the bigger the system gets, the more expensive each release gets due to regression testing.
Also, the strategy results in some bugs being discovered late and are therefore more expensive to fix.
So, the only thing we’ve achieved by skipping the acceptance testing, is to wrap a traditional development method in agile clothes. But it really isn’t, and that’s what I’ve come to realize recently: that we have to find a way to automate our acceptance tests. If we want to be agile that is, and effective.
Cheers!
@dani
Hi Dani, good question.
In fact there are several ways to automate acceptance tests. You can use scripts for batch-like systems, recording tools that invoke your GUI components, tools like FitNesse or xUnit to write acceptance tests that skip the GUI layer and focus on the underlying business logic layers. A good thing with the last type of tools is that they can be written before the features are implemented. That is a huge advantage and provide the best specification a developer can get.
A thing to watch out for, though, is that many of these tools, especially the recording based ones, usually results in brittle tests; even with small changes a lot of the tests break and so they tend to be costly to maintain.
Cheers!
Edit growth is quadratic not exponential. Excellent post and a good method of argument super linear employment costs are likely to attract attention from managers.
So who’s going to maintain the ever growing mountain of unit tests, especially when you make a bigger change?
I would expect the full regression approach to quickly fossilize an application. Rather the opposite of Agile.
I’d love to see a copy of this article without “if you don’t do X, you won’t be agile”, because where I work, agile isn’t a goal, but quality and quick turnaround is. If I passed this around, the word ‘agile’ would get in the way of an otherwise great message.
@noah healy thank you for setting me straight on the math. That is clearly not my strongest subject. 🙂
@Andomar good point and a valid question (although I didn’t refer to unit tests in my post, but rather automated acceptance tests, still the same problems apply)
The automatic test suites have their own set of problems and if we don’t pay close attention to them they could be the next thing that brings us down. We need to maintain our tests in the same way that we maintain our code. We need to constantly clean the small messes we make and fix broken tests.
So, we spend some of the time we gain by automating the tests maintaining the tests.
@Ricky Clarkson thank you for your comment. I see and agree with your point. Regression testing is not an agile problem alone, but a problem with incremental development in combination with frequent releases. I might revise the text and try to throw some of the agile references out.