How to automate acceptance tests
In a comment to my previous post AC wonders how I automate acceptance testing. He considers that as being done by real testers. Well, he’s absolutely right. I expressed myself a bit sloppy, so let me use this post to explain what I meant to say.
Acceptance testing is done by the customer to make sure she got what she ordered, to make sure you delivered the right system. This process cannot and should not be automated. What you can do – and this was what I meant to say – is to automate acceptance tests, and use them during construction.
The customer defines the acceptance tests. These are valuable to the developer since they can be used to validate the system as it develops. The sooner you get a hold on these test cases the better, so make sure you press the customer to produce them early. Better yet, help the customer in the process. That way you can help making the test cases automatable.
So, how do you design a test so that it can be automated? First of all you need to stop thinking in terms of user interface. Instead you should describe the test, in plain text, and in terms of action, function and result. Look for possible test data, edge cases, exceptional uses and describe those as well.
The second thing you need to do is to make the system in such a way that it can be automated. Separating the GUI from the business logic is often all you need to do to achieve that. You then automate the acceptance tests in the same way as you automate integration tests. In fact, they could even become a part of your integration testing harness. The only difference being the fact that they are defined by the customer.
I’m not saying this is easy. (Here’s a nice post which discusses when to automate.) I’m not saying it can be done for all of your acceptance tests. What I say is: it can be done for far more tests than you might think. For example, we are customizing a GIS system for an internal customer’s needs. The application is user controlled and involve plenty of complex actions, modifying graphical data and properties. Still we were able to automate most of the acceptance tests.
We spent a lot of time writing code to set up fixtures, initiate actions and check the results. It was worth every second though. You see, manually running one of our acceptance test cases usually takes several minutes. By having the computer do all the work, time is reduced significantly and free up developer time. We use the automated test cases individually, almost like an extended compiler, to verify features as we implement them. And at night we run all of our acceptance tests to get feedback on how far we’ve come and to spot unexpected problems.
But the real acceptance testing is still going to be done manually, by the customer, at the end of the project. Just like AC pointed out.
Cheers!