Archive for the ‘testing’ Category

New Tutorial: Getting Started With UISpec on XCode 4

July 8th, 2011 No comments

One of the most popular pages on this website is the tutorial that helps you getting started with UISpec; i guess there’s a great demand for acceptance testing frameworks on Objective-C.

Anyway, I have just made a switch from XCode 3 to XCode 4, a huge improvement. A lot has changed in the user interface so I decided to upgrade the UISpec tutorial, but kept the old one in case someone prefer the old IDE:


The Swamp of Regression Testing

January 17th, 2011 14 comments
Have you ever read a call for automation of acceptance testing? Did you find the idea really compelling but then came to the conclusion that it was too steep a mountain for your team to climb, that you weren’t in a position to be able to make it happen? Maybe you gave it a half-hearted try that you were forced to abandon when the team was pressured to deliver “business value”? Then chances are, like me, you heard but didn’t really listen.
You must automate your acceptance testing!

Why? Because otherwise you’re bound to get bogged down by the swamp of regression testing.

Regression Testing in Incremental Development

Suppose we’re building a system using a traditional method; We design it, build it and then test it – in that order. For the sake of simplicity assume that the effort of acceptance testing the system is illustrated by the box below.

Box depicting a specified system

The amount of acceptance testing using a traditional approach

Now, suppose we’re creating the same system using an agile approach; Instead of building it all in one chunk we develop the system incrementally, for instance in three iterations.

The system is divided into parts (in this example three parts: A, B and C).

An agile approach. The system is developed in increments.

New functionality is added with each iteration

Potential Releasability

Now, since we’re “agile”, acceptance testing the new functionality after each iteration is not enough. To ensure “potential releasability” for our system increments, we also need to check all previously added functionality so that we didn’t accidentally break anything we’ve already built. Test people call this regression testing.

Can you spot the problem? Incremental development requires more testing. A lot more testing. For our three iteration big  example, twice as much testing.

The theoretical amount of acceptance testing using an agile approach and three iterations, is twice as much as in Waterfall

Unfortunately, it’s even worse than that. The amount of acceptance testing increases exponentially with the number of increments. Five iterations requires three times the amount of testing, nine requires five times, and so on.

The amount of acceptance testing increase rapidly with the number of increments.

This is the reason many agile teams lose velocity as they go, either by yielding to an increasing burden of test (if they do what they should) or by spending an increasing amount of time fixing bugs that cost more to fix since they’re discovered late. (Which is the symptom of an inadequately tested system.)

A Negative Force

Of course, this heavily simplified model of acceptance testing is full of flaws. For example:

  • The nature of acceptance testing in a Waterfall method is very different from the acceptance testing in agile. (One might say though that the Waterfall hides behind a false assumption (that all will go well) which looks good in plans but becomes ugly in reality).
  • One can’t compare initial acceptance testing of newly implemented features, with regression testing of the same features. The former is real testing, while the second is more like checking. (By the way, read this interesting discussion on the difference of testing and checking: Which end of the horse.)

Still, I think it’s safe to say that there is a force working against us, a force that gets greater with time. The burden of regression testing challenges any team doing incremental development.

So, should we drop the whole agile thing and run for the nearest waterfall? Of course not! We still want to build the right system. And finding that out last is not an option.

The system we want is never the system we thought we wanted.

OK, but can’t we just skip the regression testing? In practice, this is what many teams do. But it’s not a good idea. Because around The Swamp of Regression Testing lies The Forest of Poor Quality. We don’t want to go there. It’s an unpleasant and unpredictable place to be.

We really need to face the swamp. Otherwise, how can we be potentially releasable?

The only real solution is to automate testing. And I’m not talking about unit testing here. Unit testing is great but has a different purpose: Unit-testing verifies that the code does what you designed it to do. Acceptance testing on the other hand verifies that our code does what the customer (or product owner, please use your favorite word for the one that pays for the stuff you build) asked for. That’s what we need to test to achieve potential releasability.

So, get out there and find a way to automate your acceptance testing. Because if you don’t, you can’t be agile.


Benefits From a Software Testing Job

January 12th, 2011 No comments

This is a guest post by Janet J. Fleming.

Are You Going to Get Benefits from a Software Program Testing Job?
This is a good question….initially you must see if you have a certain type of personality to accomplish software testing. You ought to be organized, logical and thorough. You will be writing test cases depending on business and functional requirements – or in other words you should do. 
Then you’ve to implement those tests – often repeatedly. Your primary goal is always to be sure that no software goes out to your customer without all the bugs found. It’s rarely achievable, but should be your main goal. I always prefer to believe that your 2nd goal must be to have every developer hate you because you keep finding bugs inside their code 🙂
The answer if software testing is an excellent career option is dependent upon who’s asking the question. I’ll answer it as if my audience is definitely an engineer.
I’m flip, but sincere – my working practical experience has proven to me that the theory of software development never comes about in reality.
Theoretically, software testing is:
– Validating and recording that software program performs the functions it’s supposed to.
– Making sure and recording it doesn’t do anything whatsoever it is not designed to
This presupposes you have been told what it is supposed and not supposed to do. People you’re working for don’t always do this – they might not necessarily rely on you not to run away with their secrets.
Because software program is a business (except when you are doing work for the military) business guidelines apply a lot more strongly than engineering principles. Software testing is expensive, and so the decisions about goals and how much to do can be extremely based on ROI considerations.
Inside the end-user relationship, the user’s perception just isn’t necessarily directly related to the physical world, and it’s also the user’s perception of whether your system works that finally rules inside the minds of management, whose job is purely to ensure no one is complaining concerning the software.
Therefore, the truly practical explanation of software testing could be summarized as 3 goals:
1° Verify the people that use software believes it’s doing whatever they require it to accomplish
2° Verify that this software doesn’t do anything immediately detectable that is not desirable for the user.
3° Verify that any undesirable action has a sufficient length period that the software will look to perform properly long enough for you to make it to another round of VC investment or sell the business 🙂 
And you? Do you consider Software Testing could be the right career path?

Who am I ?: Janet J. Fleming is writing for the software testing course online blog, her personal and non-commercial in nature hobby web site to offer free strategies for software testing novices/specialists to help them get a new work.

Categories: guest posts, testing Tags:

Great Bug Reports

May 25th, 2010 No comments

Is your team maintaining a bug report dump? When new issues get reported at a pace the team cannot keep up with it’s in trouble. At some point the team must yield and bug reports will start to pile up. The unattended bug reports lose their value over time and become a terrible waste.

Shallow Testing

There are many possible reasons for a bug tracking system to turn into a dump of wasted effort. One reason is when the testing team is practicing shallow testing. That is when the testers file bug reports right after discovering bugs, leaving any further investigation to the developing team. This is convenient for the testers but bad for the overall process of finding and fixing bugs.

The shallow testing technique results in many bug reports, most of them with poor quality. But the measure of effective testing is not how many bug reports it produces. What counts is the amount of bug reports that lead to bug fixes. To be effective we need great bug reports.

Great Bug Reports

So, what does a great bug report look like? We need to ask another question to answer that. What does the developer need to fix a bug? The first step of a successful bug elimination is to reproduce it in a controlled development environment.
In order to do that he needs to:

  1. Put the system in a state from which the error can be produced.
  2. Perform the actions that produce the error.
  3. Know how to identify the error.

Most bug reports contain enough information to make step 2 and 3 clear. Clues toward step 1, on the other hand, are usually missing. As a result the developers end up doing additional testing to pin down the problems.

If we do a better job with our bug reports we can get two really important effects. We help developers spend less time on each bug and instead fix more of them, and we increase the chance that the report will be acted upon. This is why thorough testing on behalf of the testing team is so important.

Focus On the Right Things

The basic formula to produce great bug reports is as follows.

  1. A descriptive name.
    The most important details of the bug presented in a sentence.
  2. Steps to reproduce.
    This is the part that distinguishes great bug reports from lesser ones. A well-described sequence of action steps that describes how to reproduce the bug removes uncertainties and signals thorough research. Developers are therefore more inclined to act upon such a bug report than a fuzzy one.
    It’s also important that the reproduction steps start at a known state, preferably at a clean program start.
  3. Expected results.
  4. Actual results.
  5. Any possibly relevant attachments, for example screen-shots and data.

Depending on the system you’re using, a multitude of additional parameters may be set to categorize the bug further. Provide the extra information if applicable, but it’s not essential so don’t let it steal focus from the basic formula.

Bug Mining

The foundation of a great bug report is built with great testing. Great testing is hard work and requires a special mindset: the love of bugs and the willingness to spend a lot of time examining each new specimen.

When a great tester discovers a bug the first thing she does is not to fire up the error reporting tool and start describing it. She leaves that for later. Instead she goes about to find the answers to these two questions:

  1. Can I reproduce this bug?
  2. If so, what is the smallest number of steps I need to do so?

This usually requires several program resets, attacking the bug from different angles. I like to call this process Bug Mining and it’s the tester’s most important task.

Why all the hard work?

Why should the tester have to do all this work, you might ask. After all, the developers introduced the bugs in the first place so they should clean up their own mess, right? Besides, isn’t it better if the testers spend their time testing and finding more bugs?

The problem is this: If the testers prioritize quantity over quality we’ll end up with a very ineffective test-correction-test cycle, and an ever growing mountain of unresolved bugs.

The more time developers spend on each bug the fewer bugs get corrected. And if fewer bugs get fixed, more time will pass between bug discovery and bug correction, making it even more expensive to fix.
Since the developers are the only ones that can make the bugs go away you really need to think twice before making them do work someone else can do just as well. This is especially true for Bug mining.

Guiding Principles

As a summary, here are some guiding principles to produce great bug reports:

  • Focus your effort on the steps to reproduce the bug. They should be clear, complete and as short as possible.
  • Only one bug per report.
    Handling several bugs as one is problematic for several reasons. For example, what should you do if the bug report contains bugs with different severity? Also, it’s difficult to describe several bugs in the name sentence so some might be overlooked. A third reason is the communication problem that happens when only some of the reported bugs get corrected.
  • A few really good bug reports is better than plenty of not so good ones. Quality is better than quantity when it comes to testing.


CodeGear, Please Fix the Anonymous Method Assymetry

June 3rd, 2009 4 comments

As I noted in my previous post, anonymous methods is a big new feature in Delphi 2009 for the Win32 platform. While “closures” is natural and much appreciated in other languages, most notably Ruby, the Delphi community is still a bit reluctant and hesitant. Segunda Feira put words on it in his post on the subject.

I am still not convinced that this convenience is worth getting all that excited about – It has not enabled anything that was previously impossible, nor even especially difficult either to implement or to understand


anonymous methods should be used only when you absolutely have to, and so far I have not yet seen an example where anonymous methods are the only way to achieve anything

I agree that more often than not a problem can be solved with equal elegance using a pure object orientated approach, but there are situations where anonymous methods may actually be the better alternative. One situation that comes to mind is the setting up of test fixtures.
Say, for instance, that we want to test the event firing mechanism of a component. An easy way to set this up could be like the following.

procedure TestSomeComponent.TestOnChange;
  SUT: TSomeComponent;
  OnChangeFired: Boolean;
  OnChangeFired := False;
  // Set up the fixture
  SUT := CreateTheComponentToTest;
  SUT.OnChange :=
    procedure(ASender: TObject)
      OnChangeFired := True;
      CheckSame(SUT, ASender, 'The Sender event argument should be set');
  // Perform operation
  SUT.Text := 'Some text';
  // Check the result
  CheckTrue(OnChangeFired, 'Setting the Text property should fire the OnChange event');

The above code checks that the component’s OnChange event is fired when the Text property is set, and that the component is set as the Sender reference to the event handler. Except for being more compact, the biggest advantage to using anonymous methods in this scenario is that we avoid the obscure test smell and that we don’t have to clutter our test class with an instance variable (like FOnChangeFired) to identify if the event was really fired or not.

The only problem is: it doesn’t work. Why? Well, because the OnChange property is most likely of an instance method reference type (i.e. TNotifyEvent), meaning it doesn’t accept a references to an anonymous method even if it has the same signature.

type TNotifyEvent = procedure(ASender: TObject) <strong>of object</strong>;

For our code to compile we need to redeclare TNotifyEvent and remove the “of object” keywords and instead use the method reference type.

type TNotifyEvent = <strong>reference to</strong> procedure(ASender: TObject);

But of course, that’s not an option. It would mean that the traditional way of setting up an event handler (by using an instance method) would not work.

I see a definite problem with how the Delphi language explicitly forces you to distinguish between instance methods (and class methods for that matter) and anonymous method references, even though they share the same signature.
This is most unfortunate since I feel that the situations where we’d like to support both kinds of event handlers are quite common. And with the current semantics we have to use code duplication in order to achieve that. Like the two Synchronize methods of the built in TThread class.

In Delphi 2009 an overloaded variant of the TThread.Synchronize method was introduced, one that make use of anonymous methods. Here are snippets from that code:

  TThreadMethod = procedure of object;
  TThreadProcedure = reference to procedure;
  class TThread = class
    procedure Synchronize(AMethod: TThreadMethod); overload;
    procedure Synchronize(AThreadProc: TThreadProcedure); overload;

The two methods have almost identical implementations. The only real difference is the type of the argument.

procedure TThread.Synchronize(AMethod: TThreadMethod);
  FSynchronize.FThread := Self;
  FSynchronize.FSynchronizeException := nil;
  FSynchronize.FMethod := AMethod;
  FSynchronize.FProcedure := nil;

procedure TThread.Synchronize(AThreadProc: TThreadProcedure);
  FSynchronize.FThread := Self;
  FSynchronize.FSynchronizeException := nil;
  FSynchronize.FMethod := nil;
  FSynchronize.FProcedure := AThreadProc;

I may be overly sensitive, but code like that really disturbs me. Unfortunately it gets worse. If we follow the execution chain down into the Synchronize class method that is invoked by the two, we find this.

class procedure TThread.Synchronize(ASyncRec: PSynchronizeRecord; QueueEvent: Boolean = False);
  SyncProc: TSyncProc;
  SyncProcPtr: PSyncProc;
  if GetCurrentThreadID = MainThreadID then
    if Assigned(ASyncRec.FMethod) then
    else if Assigned(ASyncRec.FProcedure) then
    end else

It would be a lot nicer if the two reference types were joined under a common reference type. And I can’t see why it couldn’t be done. When I look at the “of object” keywords I get a feeling that the language is leaking compiler implementation through the language interface; information that is indifferent to the developer. What matters from a callers’ perspective is the method signature, not whether the method has a self pointer or not.

I hope CodeGear recognizes this problem and find a way to clean this assymetry from the language. Anonymous methods would be so much more useful if they do.


Did I say don’t unit-test GUIs?

January 3rd, 2008 1 comment

Isn’t life funny? Two weeks ago I stated my opinion that unit-testing graphical user interfaces isn’t worth the trouble. Now I find myself doing it, writing unit-tests for GUI components.

What happened, did I come to my senses or go crazy (pick the expression that fits your point of view) during Christmas holidays? No, I still think that unit-testing user interfaces is most likely a waste of time, but for some special occasions, like this one, they will pay off.

My task is to make changes to a tool box control in a legacy application. The control is full of application logic and it has a strong and complicated relationship to an edit area control. There are two things I need to do:
First I have to pull out the application logic of the tool box and break the tight relationship to the edit area. Then I need to push that application logic deeper into the application core, so that the tool box could be used via the plug-in framework.

I figured I had two options. I could either refactor the tool box and make the changes in small steps, or I could rebuild the complete tool box logic and then change the controls to use the new programming interface.
I found the rebuild alternative too risky. The code base is full of booby traps and I have to be very careful not to introduce bugs. Therefore I decided to go with refactoring.

But refactoring requires a unit-testing harness, which of course this application doesn’t have. Trust me; you don’t want to refactor without extensive unit-testing, so here I am setting up the tests around the involved GUI controls. It’s a lot of work, especially since I don’t have a decent framework for creating mock objects, but it’ll be worth the effort once I start messing around with the code.

As a conclusion I’d like to refine the “Don’t unit-test GUI” statement to read “Don’t unit-test GUI unless you’re refactoring it, and there’s no easier way to make sure your changes doesn’t break anything.”


Don’t unit-test GUI

December 20th, 2007 21 comments

I’m currently rereading parts of the book Test-Driven Development: A Practical Guide, by David Astels. It’s a great book in many ways, well worth reading, but I have objections to one particular section in the book.
The author tries to convince me that developing my user interfaces using a test-driven approach is a good thing to do. I disagree.

I love TDD, it’s one of the most powerful development techniques I know, but it’s not without limitations. For one thing, code with unit-tests attached is more difficult to change. This is often an acceptable price to pay since the benefit of producing and having the unit-tests is usually greater.

But the return of the investment isn’t always bigger than the price, and sometimes the cost of change exceeds the benefit of protection. That’s why most developers won’t use TDD for experimental code, and that’s why I’m not using it to test my user interfaces.

I prefer to develop GUIs in a RAD environment, visually dragging and dropping components, moving them around, exchanging them for others if better ones are to be found. In that process unit-testing just gets in my way. After all, the GUI is supposed to be a thin layer, without business logic, so there is only so much to test.

One could theoretically test that the form contains certain key components, that they are visible, have a certain position or layout, stuff like that – but I find that kind of testing too specific for my taste.

In my opinion, unit-testing should test functionality, not usability. It shouldn’t dictate whether I decide to show a set of items in a plain list or in a combo-box. What it should do, is test that the business logic of my code produce the correct set of items, and leave the graphical worries to the testers.

This brings us to something that is sometimes misunderstood: Unit-testing can never replace conventional testing. Some things are just better left to humans. Like testing user interfaces.


Does unit-testing deserve its own DSL?

December 10th, 2007 7 comments

We’ve done a lot with testing frameworks over the years, but does the testing concern deserve its own standalone DSL?

This intriguing question was asked by Michael Feathers in his Mock Objects: Leaping out of the Language post. My spontaneous answer is: Absolutely!

I’m a big fan of xUnit frameworks, but when I imagine an alternative unit-testing specific language one special property comes to mind, a feature that would really make a difference. I’d call it unconditional mocking. With a DSL based unit-testing framework one could test really complex objects, even legacy code, since mocking internal objects would require no change to the original programming interface.

For example, this (nonsense) code

class A {
  private B _b;

  // constructor
  this A() {
    _b = new B()

unittest {
  // B can not be mocked
  A a = new A();

would require refactoring in order for _b to be mockable.

class A {
  private B _b;

  // constructor
  this A(B b) {
    _b = b;

unittest {
  // B could be mocked
  B b = new BMock(...);
  A a = new A(b);

But in a unit-testing DSL, one should be able to mock any object, in this case B, without changing the source code first. This is handy for dealing with the unit-testing paradox: Refactoring requires a unit-testing harness to make sure no functionality gets broken, but unit-testing requires testable code; So what to do when the code isn’t testable? A unit-testing DSL would make it easier to put up the initial testing harness.

Also, as Michael points out, a unit-testing DSL could be used to mock any kind of construction, not just objects: Functions and methods for instance. Oh man, could I have use for such a feature?

To give us an image of a DSL for unit-testing in a non-object-oriented language like C, Michael provides this example:

function send_port_command with 90, “CMD: 12”
            calls io_mode which returns M_READ
            calls set_mode with M_WRITE
            calls write_byte with 90
            calls write_bytes with “12”
            returns E_OKAY

That would be testing a function like this:

status send_port_command(byte port, const char *message)
  if(io_mode() == M_READ)
  return E_OKAY;

I have a problem with his example though. In my opinion the test-code resembles the target code a little too much, like a bad manager performing low-level supervision. Too detailed testing beats the purpose since it makes changes more difficult. My philosophy is that test-code should test WHAT the code does, and not bother too much on the HOW.

So, my not so thought through proposal, using Michaels example, would be something like this:

TEST send_port_command

MOCK write_byte(port)
EXPECT port == 90

MOCK write_bytes(bytes)
EXPECT bytes == "12"

CALL send_port_command with 90, "CMD: 12"

Of course there should be support for more advanced mock features like call counting:

MOCK  write_bytes(bytes)
EXPECT   "12", "13"

CALL send_port_command with 90, "CMD: 12"
CALL send_port_command with 90, "CMD: 13"


MOCK  write_bytes(bytes)

or sequential values

MOCK  io_mode

Implementing the DSL would be a hefty task though. But, the problems aside, how would your unit-testing DSL be like? I’d be very interested to hear your opinions.


Categories: DSLs, programming, testing Tags: