Archive

Archive for May 30, 2010

Data driven testing: the last mile in the journey towards automated testing

May 30, 2010 1 comment

The promise of automated testing benefits remains just a promise unless there is commitment to implement data driven testing.

Instead of defining data driven testing (there are many good books and articles written on the subject), let’s simply take a look at the problem from two perspectives:  “I am a customer and I have a problem” and “I am a software vendor and I can’t believe this customer keeps encountering the same problem”.

“I am a customer and I have a problem”; in every instance – the only perspective that matters:

– The new software release has done nothing to solve the problem that has been open for many months
– This error happens frequently enough.  Why can’t someone fix it?
– Other customers we spoke to do not seem to have the same problem (one of the indicators that the customer’s unique data may be a contributing factor)

“I am a software vendor and I can’t believe this customer keeps encountering the same problem”:

– We tested everything and this problem still affects only this specific customer

It’s important to recognize that companies in the same industry will use the software differently.   To illustrate, companies in the retail industry (customers of your software products) can sell …

– Apparel
– Consumer electronics
– Office supplies
– Pharmaceuticals
– Footwear
– Home improvement supplies
– Jewelry
– Luxury leather goods

In addition to performing the usual testing activities, each software release should be tested with one or more test data environments (I call these environments TDEs), where each TDE represents a certain data bias for very good reasons:

– Customer segment:  apparel, office supplies, footwear
– Customer size:  large, medium, small
– Customer satisfaction index:  ideally – actual customers with the most defects or unusual problems can be encouraged to provide a copy of their data (scrubbed and cleansed of course).    In the past, I was responsible for a software product which could not be shipped until the ‘gold image’ was tested with real data from one specific customer.  This approach can save a lot of time and money, especially for the customer waiting for the new software release.

So – how does all of this work?

Part 1 – Create unique QA environments (usually one for each TDE, or test data environment with a certain bias)

– Create automated procedures to deploy software release candidate to each TDE
– Deploy on demand or – even better – automatically after each nightly software build
– Ask your customers to provide copies of data required for testing (one customer – for each data bias)
– Scrub the data / mask all confidential or personally identifiable information
– Enjoy the comfort of pressing a button and automatically generating a test data environment which has required bias:  home improvement retailer, 2 years of sales data, 40 stores, etc.

Part 2 – Create a portfolio of end-to-end scenarios which are based on real experience working with customers

– Example A:  footwear customers may experience more problems processing merchandise returns.  Develop automated test scenarios which will exercise anticipated trouble spots
– Example B:  luxury leather goods customers may experience more problems while running reports.   Again – develop automated  test scenarios to target this area of product functionality

Part 3 – Make it work

– After the nightly build, deploy the release candidate to one or more test data environments
– Trigger automated data generation programs and generate TDEs with desired data bias profiles
– Run test scenarios as needed against target test data environments
– Automate collection of test results / email test results to everyone that needs to see them

Part 4 – Arrive the morning and enjoy the benefits of automated testing implemented to the fullest extent

– Inbox should contain messages with test results from functional, regression, and now data driven tests

Some software products are not as sensitive to variability in customer data.   But most software products are.  Data-driven testing is the last mile in the journey towards getting the maximum value from automated testing.

Categories: Uncategorized