Testing with production data

When you're faced with testing a system in a complex environment the intuitive solution is to create an environment which is as similar to the production environment as possible and use production data for testing. Despite the risk of compromising sensitive information this is the solution many organisations choose to invest in.

But what do they get for their money?

Inconsistent and unrepeatable tests – probably. Slow running test – almost certainly. Conflicts over access to a test environment – check.

And it's not cheap either – the cost of developing and a maintaining routines for copying and scrubbing large quanitites of data from a wide variety of sources is not to be sneezed at.

This type of testing is also almost inevitably done at the end of the development process which means that you're in effect trying to test quality into your product. Which as Harold S. Dodge said many years ago, you can't do.

Suddenly the intuitive solution isn't quite attractive as it first seems.

Why do we keep doing things which we know don't work?

In any organisation, some working practices simply don't work. Unfortunately the knee jerk reaction to this seems to do more of what doesn't work.

We've tried document driven system development - it doesn't work. The solution is not more documents.

We've tried up front analysis and design - it doesn't work. The solution isn't more up front analysis and design.

We've tried enforcing enterprise wide architecture - it doesn't work. The solution isn't more enforced architecture.

We've tried detailed project plans - it doesn't work. The solution isn't even more detailed plans and estimates.

We've tried enforcing pre-defined processes to systems development - it doesn't work. The solution isn't more process.

We've tried acceptance testing as a final step before production setting - it doesn't work. The solution isn't more late acceptance testing.

There are so many things that "Agile" has proven do work. Why is it so difficult to get organisations to do more of the things that work rather than more of the things we know don't work?

Open Space Technology

We've used Open Space Technology several times in our project and it's worked well in a variety of situations from retrospectives to architecture and design discussions. Up until last week the number participants has always been less than 20 so it was with both anticipation and trepidation that I found myself facilitating an Open Space for around 200 people.

After a brief introduction it was a relief to see the agenda appear almost by magic and I found myself having to hold back the enthusiastic crowds as they champed at the bit to start their discussions.

The whole day was a credit to Open Space as a format for meetings and conferences, and also a credit to the CIO for having the courage to agree to this unconventional approach.

Monoculture misery

Most large organisations struggle to keep control of their software maintenance costs. Many try to do this by trying to restrict which tools and techniques are used. This seems a reasonable strategy – the more homogeneous environment you can achieve the easier it ought to be to maintain. To a certain extent this may be true, but taken too far the result is almost inevitably the opposite – an ever increasing burden of maintenance costs.

Whatever you're buying, a good rule of thumb if you're concerned about maintenance, is to go for quality. Software is no exception. In fact with some estimating that maintenance costs account for over 90 % of software costs, it's a textbook example of this. There's nothing that dents your IT budget more than buggy applications.

What happens when you try to achieve a monoculture of techniques and tools? The people building the product are forced to use tools and techniques which aren't quite up to the job. And since any standardization effort inevitably lags advancements in technology the product will be dated before it's released. Is there anybody who thinks this is a good strategy for improving quality?

I have a theory. This theory is that the three most important factors for reducing the cost of software maintenance are:

• The quality of the product – fewer bugs mean less maintenance.

• Building the right product – often up to 80 % of software “requirements” aren't really required but add to the maintenance burden

• Employing the right people – the wrong people will build the wrong product whether or not they use the mandated tools.

So what do you think? What's the most effective way of employing your brightest staff?

a) Set them to work mandating which tools and techniques are to be used and enforcing these policies.

b) Getting them to act as a motor for innovation by helping development teams to take advantage of modern advancements in software development.

I know which one I'd choose. And I know which organisation I'd choose to work for.

Agile Testing - Test early

A common feature on many scrum boards is a column with the title “Test” or “Verify” or perhaps “Acceptance”. This column is almost invariably towards the right hand end of the board. Implement first, test afterwards. Does that sound agile or are we just making the same old mistakes?

Once you've got your testers in the team, and teaming up with the programmers, it's time to get serious about testing early. Not just early programmer testing but early customer testing too.

Having a separate column for verification on your scrum board merely reinforces the traditional idea that testing is something you start doing when a feature is implemented.

You don't have to wait until features are complete before starting work on customer tests. And if you can successfully encourage testers and programmers to work in harmony you might even find that a feature is tested and accepted at the same time that the implementation is completed.

Agile Testing – Teaming up

Interesting things happen when you start using Test Driven Development. One revelation is that TDD isn't just about discovering bugs:

"The act of writing a unit test is more an act of design than of verification." (Bob Martin)

As I previously wrote, a first step towards agile customer testing is bringing your testers into the team. A second step is to encourage your testers to team up with programmers and work together on the development of features.

As we have brought testers into our team and encouraged them to team up with programmers we have also seen benefits that aren't just limited to finding bugs.

To paraphrase Uncle Bob:

The act of writing a customer test is more an act of process improvement than of verification.

So what are these process improvements? Here are a handful we have experienced:

• Stories get broken down into smaller stories that allow the testers to get testing early

• Customer tests define the features of the product, eliminating the need for many of the documents traditionally associated with testing

• There is no need for a separate acceptance activity – when the programmers have completed the feature the customer tester has completed their testing

• Programmers get involved in, and a better understanding of customer testing (and vice versa).

I'm sure we will discover more benefits as we get better at customer testing.

Agile Testing – The embedded tester

It's the obvious first step towards agile customer testing – bring the testers into the team.

There is certainly a risk that a tester will lose their objectivity when they become part of the development team. But would a good tester allow this to happen? Would an alert agile coach allow this to happen? Surely not.

Being an embedded tester means changing how you work. Instead of merely diagnosing bugs after they've become chronic, you can help to prevent bugs from infecting the product in the first place.

An agile tester doesn't have to be the person who just ticks off features against a list of requirements. They can, and should be much more than this. Working in the team the tester can take a much more active role in improving the product and influencing the development process. They can help build quality into the product rather than trying to test quality into it.

Not all testers find it easy, or are prepared to adapt to this change in approach. But it's not unreasonable to demand a degree of personal agility of your agile testers.

Success has it's foundation in people. So if you're serious about agile customer testing, make sure you have the right testers in the right place. In the team.

Agile Testing – Simple in practice

Agile developers know that a few basic practices can be used to drive simplicity into programming. Working in collaboration with the customer simplifies communication. Pair programming reduces the need for documentation and controls. Programmer testing drives you towards a simpler design.

So what are the practices which can be used to drive simplicity into agile customer testing?

Face to face communication is the simplest and most effective way for us to communicate, by a long chalk. So bringing the tester into the team is an obvious way of bringing programmer and tester face to face.

Another practice to simplify testing is encouraging testers to team up with the programmers - it's not just programmers that can make a pair. For the more open minded, why not try a threesome – programmer/programmer/tester? You'll find you simply don't need a lot of the documents that are traditionally required when testing.

Finally you can start customer testing early. Whether it's automated testing or exploratory testing, you shouldn't need to wait until the end of the sprint to perform them. And by encouraging the programmers to break up large features into smaller ones the tester can drive the team to deliver features that are simpler to test.

So there you have it – three basic practices to simplify agile customer testing:

• tester in the team

• teaming up with programmers

• test early customer testing

I'll stick my chin out and suggest that these are defining practices for agile customer testing. There are several other good practices that agile testers can adopt, but if you're not doing these three then you're not agile testing. It's as simple as that.

Agile Testing - Keep it simple

One of the things which always seems to make discussing testing difficult is the plethora of different types of testing: System Testing, Acceptance Testing, Functional Testing, User Testing, Integration Testing etc. Worse still, these terms can be combined to come up with almost endless variations: User Acceptance Testing, System Integration Testing, Developer System Testing, ...

And everybody seems to have their own individual interpretation of what these terms mean. No wonder it's so difficult to have a reasonable discussion about agile testing.

Kent Beck explained back in 1999 that simplicity is one of four values for extreme programming. Simplicity also appears as one of the principles behind the Agile Manifesto. Kent kept it simple even when talking about testing. He talked about just two types of tests; programmer tests and customer tests. Programmers write tests which must run flawlessly for development to continue. Customers write tests demonstrating that features are finished.

So let's see how far we can get by using just these terms. Hopefully it will make discussing agile testing simpler. Who knows, maybe even performing agile testing will be more effective if we can also keep it simple in practice.

In my next post I'll identify three practices which can simplify agile customer testing.

Testing, verification and validation . What's the difference?

Software testing, I've read, is the process of validating and verifying a software program/application/product. But what's the difference between verification and validation? Here's the clearest definition I've managed to find.

• Verification shows the conformance with specification
• Validation shows that the program meets the customers requirements

But wait a moment. Aren't the specifications just a way of describing some of the customers' requirements? At least they should be. Otherwise the customers are paying for something they don't need, or not getting what they're paying for.

In other words, verification and validation should be the same thing, at least when it comes to software testing.

Another thing that makes me deeply suspicious of the terms ”verification and validation” is the worrying frequency which the V model is used to explain them.

Why not how nor what

Here's a quick summary of my career as a programmer.

1. write working code
2. don't understand my own code when I return to it later
3. write lots of comments in my code
4. still don't understand the code since the comments didn't get updated when code changed
5. write clearer code and fewer comments

Interestingly one of the 97 things every programmer should know is write plenty of comments while another is write fewer.

Confused? No need to be. Just comment why you've done something a particular way. If you find yourself commenting on what your code does or how then make the code clearer instead.

Agile cyclist

Cyclists seem to be universally hated by motorists. A hate that is hardly in proportion to the inconvenience the cyclist occasionally causes.

Or is it perhaps envy? The cyclist can cruise past grid locked drivers, ignoring the normal rules of the road on a machine that costs a fraction of what the motorists paid for his vehicle. And has more fun too.

A bit like agile development really. Cheap, gets you there quickly and you have a lot more fun on the way.

Stay loose

Loose coupling in complex systems is almost universally regarded as a good thing. So I've been surprised how often and how vigorously we have had to defend the system we're developing from other systems wanting to access our private parts.

Problems caused by tight coupling are not restricted to IT systems. Two recent incidents reminded me of just how important it is to think very carefully before introducing unnecessary coupling in any complex system.

A colleague, lets call him Pontus, was travelling to work in his new car when it came to an unexpected standstill. Now this particular model has a feature which reduces the engines effect as it starts running low on fuel. But Pontus hadn't run out of fuel; the petrol gauge had malfunctioned. Normally not a critical problem, unless the petrol gauge is coupled to the engine ...

This type of problem doesn't only affect cars; trains are also susceptible.

Last week I was travelling to work by train when it came to an unexpected standstill. A toilet malfunction was immediately suspected. Now a dodgy toilet would not normally be a critical problem, unless the flush uses the same vacuum system that powers the train's brakes ...

So beware of creating unnecessary coupling. You will regret it. Maybe not today, maybe not tomorrow, but soon and for the rest of your life.

JFokus sound bites

Yesterday's journey to JFokus was fraught with difficulties (signal failure, accident, snow storm) but it was worth the journey.

Here are some sound bites I jotted down (freely translated where necessary):

That which is normal isn't always best (think royal family).

On certification: Outsourced competence judgement

On developing systems:

Everything we do, we do for the first time. Otherwise we're doing it wrongly and should be using existing components.

... which is why Gantt charts never work. You can't predict the unpredictable.

best practices proven practices

Certification again: sheep dip training

Scrumitis - done doesn't mean done done

white water rafting (doing scrum in a waterfall way)

come back when you're doing one week sprints (on scrum coaching)

On average all stories are average size. (track stories not hours)

Get good at stories (key to succeeding in scrum)

Write small stories - all stories should be size 1.

You get what you measure (dangers of focusing on metrics)

Let the architects work directly with the teams as consultants.

Hundreds of bug fixes (Maven 3)

Give positive feedback in a 5:1 ratio to negative feedback.

Create an environment for creating something new (on leadership)

Freedom leads to creativity.

Leadership cannot be separated from the situation. Nobody can be a great leader in all situations.

Information which has value is knowledge.

FitNesse revisited

I recently wrote about FitNesse saying that our goal was to allow domain experts to get on with testing while the developers could concentrate on building new features.

Wrong. Wrong. Wrong.

What I should have said was that the key to acceptance testing with FitNesse is collaboration. Testers and developers should work together when writing tests and developing features. This is what FitNesse makes possible.

Testing will always come at a cost. The trick is to make sure the returns exceed this cost. Junit has made this possible for unit testing.

When it comes to acceptance testing, the cost-benefit relationship is not so clear. FitNesse can swing the balance in favour of automation. So here's my revised list of tips for how to do this:

1. Collaborate when creating acceptance tests. Developers and testers should be encouraged to work together.
2. Create separate FitNesse fixtures for setting up test data, exercising the SUT, and making assertions on the result.
3. Names are important. The readability of your test cases will improve.
4. Write unit tests for your fixtures. Their design will benefit.
5. Keep your tests independent. Intra test dependencies will lead to tears.
6. Use dependency injection. Decoupled fixtures can be reused more easily.
7. Document your fixtures. This will help you write clear test cases.
8. Think carefully about logging. Debugging will be less painful.

Pythagoras moment

Monday evening was spent discussing retrospectives with fellow members of the Swedish Scrum User Group. I came away from the evening with a few new ideas but it wasn't until a couple of days later that I had a pythagoras-in-the-bath moment.

But first more about Monday. Once again we used the Open Space format for this get together. As Harrison Owen puts it:

Open Space Technology is effective when real learning, innovation, and departure from the norm are required. When you aren't quite sure where you are, and less than clear about where you are headed, and require the best thinking and support from all those who wish to be involved, Open Space Technology will provide the means.

Hold your horses! Isn't that also a perfect description of a retro. A bunch of enthusiasts getting together to come up with imaginative solutions to tricky problems. Why use open space just to discuss retros when open space can be used to do a retro? Eureka! Now I know how I'm going to do our next retro.

Retrospective serendipity

If we hadn’t run out of time at our latest retrospective then I wouldn’t have discovered SurveyMonkey.com.

We usually end our retros by dot voting on a list of suggested improvements. This time the retro had been particularly fruitful – lots of interesting discussions and a dozen improvement suggestions. And then our time ran out.

With various team members already heading off to their next meetings I rashly promised to send out an internet based survey so that we could at least prioritize our improvements list before the afternoon’s planning session.

And then I started to wonder if I’d promised too much. With no time to do an in depth market analysis and after a quick google I selected SurveyMonkey. A choice I wasn't going to regret. 15 minutes later the survey was sent out and in the following planning meeting we had a nicely prioritized list of suggested improvements.

It worked so well that I think we’ll do it this way again – even if we don’t run out of time in our retrospective.

Scrumming for Britain

As a scrum master I sometimes find it useful to compare what I do with coaching a sports team. Recently I was given the opportunity to put this theory into practice when I was invited to coach a national team at the World Orienteering Championships.

One of the most important jobs of a scrum master is to ensure that the day to day work of the team runs smoothly. Above all a happy team is a productive team. Likewise a happy team of athletes will get better results. So this was one of my main targets when going into the World Championships – shield the runner’s from any external problems and create a positive atmosphere.

You might think that a coach would spend most of his time telling the athletes what to do. But for a team of experienced international athletes there’s not an awful lot new to tell them. In fact I found myself working more like a facilitator – stimulating discussions, and encouraging the athletes to come up with their own solutions for handling the race situations. Very scrum masterish!

Of course there are differences too. Running a major championship is more comparable to releasing a system rather than working in scrum sprints. Though this is just what the athletes do in the run up to a major race. Using short, medium and long term goals they work incrementally towards the big release date.

So how did it go? No medals this time but some successes as well as a few disappointments. Now I hope there will be a chance to do a retrospective with the team – I’m sure that would help them get better results next time. And help me to do a better job, if I’m asked to help again.

Fitnesse refactored

We have put a lot of effort into automated testing. Not just unit testing but also system testing with Fitnesse. But it has come at a cost. In particular creating and maintaining our Fitnesse fixtures has become a burden.

In theory, Fitnesse should enable our domain specialists to write tests while the developers can get on with developing the features.

So far we haven't reached this nirvana.

For each major new feature we find ourselves either having to change our current fixtures or develop new ones. This taxes our development resources and our system tests lag behind development instead of leading it.

We must be able to do better than this.

Recently I've been wrestling with our fixtures - refactoring them to reduce the development debt. Finally they're starting to smell a little sweeter.

Here are some principles which have guided me ("best practices" if you like)

Principle 1. Structure fitnesse tests like unit tests.

They should have three distinct sections:

• setup the test data
• exercise the code to be tested
• perform assertions on the result

Principle 2 (corollory to Principle 1) Create separate fixtures for each of these phases.

One fixture say for priming the database with data. Another fixture for performing a query. And a third fixture for checking the result of the query.

Principle 3. Names are important.

By taking advantage of "graceful names" and choosing method names carefully our tests become more readable. They now look lke this:

|object id | is in query result? |

instead of

|objectId | checkResult?|

Principle 4. Fixtures should be tested too.

Writing unit tests for fixture code speeds up development and improves the quality and design of the fixtures.

Principle 5. Use dependency injection to decouple fixtures.

We use Spring to inject database connections, JMS queues and domain specific beans into our fixtures.

Nothing too radical there of course. But hopefully by concentrating on these key practices the returns from our investment in Fitnesse will improve.

Service Registry Manifesto

How should you go about creating a service registry? I've been working on this recently and the more I dig into it, the more I agree with the ideas expressed by Martin Fowler when he wrote about a Humane Registry.

Inspired by this article and also the Agile Manifesto I came up with a service registry manifesto. Namely, we should value:

• clarity and conciseness over detail and completeness
• flexibility and expandability over specification and planned structure
• integration with existing sources over creation of new content
• encouragement of use over contractual restrictions

That is, while there is value in the items on the right, we value the items on the left more.

Coincidentaly, after I created my manifesto, MF himself gave his seal of approval for using this type of list when he blogged on Comparative Values.