Testing with production data

When you're faced with testing a system in a complex environment the intuitive solution is to create an environment which is as similar to the production environment as possible and use production data for testing. Despite the risk of compromising sensitive information this is the solution many organisations choose to invest in.

But what do they get for their money?

Inconsistent and unrepeatable tests – probably. Slow running test – almost certainly. Conflicts over access to a test environment – check.

And it's not cheap either – the cost of developing and a maintaining routines for copying and scrubbing large quanitites of data from a wide variety of sources is not to be sneezed at.

This type of testing is also almost inevitably done at the end of the development process which means that you're in effect trying to test quality into your product. Which as Harold S. Dodge said many years ago, you can't do.

Suddenly the intuitive solution isn't quite attractive as it first seems.

Why do we keep doing things which we know don't work?

In any organisation, some working practices simply don't work. Unfortunately the knee jerk reaction to this seems to do more of what doesn't work.

We've tried document driven system development - it doesn't work. The solution is not more documents.

We've tried up front analysis and design - it doesn't work. The solution isn't more up front analysis and design.

We've tried enforcing enterprise wide architecture - it doesn't work. The solution isn't more enforced architecture.

We've tried detailed project plans - it doesn't work. The solution isn't even more detailed plans and estimates.

We've tried enforcing pre-defined processes to systems development - it doesn't work. The solution isn't more process.

We've tried acceptance testing as a final step before production setting - it doesn't work. The solution isn't more late acceptance testing.

There are so many things that "Agile" has proven do work. Why is it so difficult to get organisations to do more of the things that work rather than more of the things we know don't work?

Open Space Technology

We've used Open Space Technology several times in our project and it's worked well in a variety of situations from retrospectives to architecture and design discussions. Up until last week the number participants has always been less than 20 so it was with both anticipation and trepidation that I found myself facilitating an Open Space for around 200 people.

After a brief introduction it was a relief to see the agenda appear almost by magic and I found myself having to hold back the enthusiastic crowds as they champed at the bit to start their discussions.

The whole day was a credit to Open Space as a format for meetings and conferences, and also a credit to the CIO for having the courage to agree to this unconventional approach.

Monoculture misery

Most large organisations struggle to keep control of their software maintenance costs. Many try to do this by trying to restrict which tools and techniques are used. This seems a reasonable strategy – the more homogeneous environment you can achieve the easier it ought to be to maintain. To a certain extent this may be true, but taken too far the result is almost inevitably the opposite – an ever increasing burden of maintenance costs.

Whatever you're buying, a good rule of thumb if you're concerned about maintenance, is to go for quality. Software is no exception. In fact with some estimating that maintenance costs account for over 90 % of software costs, it's a textbook example of this. There's nothing that dents your IT budget more than buggy applications.

What happens when you try to achieve a monoculture of techniques and tools? The people building the product are forced to use tools and techniques which aren't quite up to the job. And since any standardization effort inevitably lags advancements in technology the product will be dated before it's released. Is there anybody who thinks this is a good strategy for improving quality?

I have a theory. This theory is that the three most important factors for reducing the cost of software maintenance are:

• The quality of the product – fewer bugs mean less maintenance.

• Building the right product – often up to 80 % of software “requirements” aren't really required but add to the maintenance burden

• Employing the right people – the wrong people will build the wrong product whether or not they use the mandated tools.

So what do you think? What's the most effective way of employing your brightest staff?

a) Set them to work mandating which tools and techniques are to be used and enforcing these policies.

b) Getting them to act as a motor for innovation by helping development teams to take advantage of modern advancements in software development.

I know which one I'd choose. And I know which organisation I'd choose to work for.

Agile Testing - Test early

A common feature on many scrum boards is a column with the title “Test” or “Verify” or perhaps “Acceptance”. This column is almost invariably towards the right hand end of the board. Implement first, test afterwards. Does that sound agile or are we just making the same old mistakes?

Once you've got your testers in the team, and teaming up with the programmers, it's time to get serious about testing early. Not just early programmer testing but early customer testing too.

Having a separate column for verification on your scrum board merely reinforces the traditional idea that testing is something you start doing when a feature is implemented.

You don't have to wait until features are complete before starting work on customer tests. And if you can successfully encourage testers and programmers to work in harmony you might even find that a feature is tested and accepted at the same time that the implementation is completed.

Agile Testing – Teaming up

Interesting things happen when you start using Test Driven Development. One revelation is that TDD isn't just about discovering bugs:

"The act of writing a unit test is more an act of design than of verification." (Bob Martin)

As I previously wrote, a first step towards agile customer testing is bringing your testers into the team. A second step is to encourage your testers to team up with programmers and work together on the development of features.

As we have brought testers into our team and encouraged them to team up with programmers we have also seen benefits that aren't just limited to finding bugs.

To paraphrase Uncle Bob:

The act of writing a customer test is more an act of process improvement than of verification.

So what are these process improvements? Here are a handful we have experienced:

• Stories get broken down into smaller stories that allow the testers to get testing early

• Customer tests define the features of the product, eliminating the need for many of the documents traditionally associated with testing

• There is no need for a separate acceptance activity – when the programmers have completed the feature the customer tester has completed their testing

• Programmers get involved in, and a better understanding of customer testing (and vice versa).

I'm sure we will discover more benefits as we get better at customer testing.

Agile Testing – The embedded tester

It's the obvious first step towards agile customer testing – bring the testers into the team.

There is certainly a risk that a tester will lose their objectivity when they become part of the development team. But would a good tester allow this to happen? Would an alert agile coach allow this to happen? Surely not.

Being an embedded tester means changing how you work. Instead of merely diagnosing bugs after they've become chronic, you can help to prevent bugs from infecting the product in the first place.

An agile tester doesn't have to be the person who just ticks off features against a list of requirements. They can, and should be much more than this. Working in the team the tester can take a much more active role in improving the product and influencing the development process. They can help build quality into the product rather than trying to test quality into it.

Not all testers find it easy, or are prepared to adapt to this change in approach. But it's not unreasonable to demand a degree of personal agility of your agile testers.

Success has it's foundation in people. So if you're serious about agile customer testing, make sure you have the right testers in the right place. In the team.