Testing with production data

When you're faced with testing a system in a complex environment the intuitive solution is to create an environment which is as similar to the production environment as possible and use production data for testing. Despite the risk of compromising sensitive information this is the solution many organisations choose to invest in.

But what do they get for their money?

Inconsistent and unrepeatable tests – probably. Slow running test – almost certainly. Conflicts over access to a test environment – check.

And it's not cheap either – the cost of developing and a maintaining routines for copying and scrubbing large quanitites of data from a wide variety of sources is not to be sneezed at.

This type of testing is also almost inevitably done at the end of the development process which means that you're in effect trying to test quality into your product. Which as Harold S. Dodge said many years ago, you can't do.

Suddenly the intuitive solution isn't quite attractive as it first seems.