Archive for May, 2010

Data driven testing: the last mile in the journey towards automated testing

May 30, 2010 1 comment

The promise of automated testing benefits remains just a promise unless there is commitment to implement data driven testing.

Instead of defining data driven testing (there are many good books and articles written on the subject), let’s simply take a look at the problem from two perspectives:  “I am a customer and I have a problem” and “I am a software vendor and I can’t believe this customer keeps encountering the same problem”.

“I am a customer and I have a problem”; in every instance – the only perspective that matters:

– The new software release has done nothing to solve the problem that has been open for many months
– This error happens frequently enough.  Why can’t someone fix it?
– Other customers we spoke to do not seem to have the same problem (one of the indicators that the customer’s unique data may be a contributing factor)

“I am a software vendor and I can’t believe this customer keeps encountering the same problem”:

– We tested everything and this problem still affects only this specific customer

It’s important to recognize that companies in the same industry will use the software differently.   To illustrate, companies in the retail industry (customers of your software products) can sell …

– Apparel
– Consumer electronics
– Office supplies
– Pharmaceuticals
– Footwear
– Home improvement supplies
– Jewelry
– Luxury leather goods

In addition to performing the usual testing activities, each software release should be tested with one or more test data environments (I call these environments TDEs), where each TDE represents a certain data bias for very good reasons:

– Customer segment:  apparel, office supplies, footwear
– Customer size:  large, medium, small
– Customer satisfaction index:  ideally – actual customers with the most defects or unusual problems can be encouraged to provide a copy of their data (scrubbed and cleansed of course).    In the past, I was responsible for a software product which could not be shipped until the ‘gold image’ was tested with real data from one specific customer.  This approach can save a lot of time and money, especially for the customer waiting for the new software release.

So – how does all of this work?

Part 1 – Create unique QA environments (usually one for each TDE, or test data environment with a certain bias)

– Create automated procedures to deploy software release candidate to each TDE
– Deploy on demand or – even better – automatically after each nightly software build
– Ask your customers to provide copies of data required for testing (one customer – for each data bias)
– Scrub the data / mask all confidential or personally identifiable information
– Enjoy the comfort of pressing a button and automatically generating a test data environment which has required bias:  home improvement retailer, 2 years of sales data, 40 stores, etc.

Part 2 – Create a portfolio of end-to-end scenarios which are based on real experience working with customers

– Example A:  footwear customers may experience more problems processing merchandise returns.  Develop automated test scenarios which will exercise anticipated trouble spots
– Example B:  luxury leather goods customers may experience more problems while running reports.   Again – develop automated  test scenarios to target this area of product functionality

Part 3 – Make it work

– After the nightly build, deploy the release candidate to one or more test data environments
– Trigger automated data generation programs and generate TDEs with desired data bias profiles
– Run test scenarios as needed against target test data environments
– Automate collection of test results / email test results to everyone that needs to see them

Part 4 – Arrive the morning and enjoy the benefits of automated testing implemented to the fullest extent

– Inbox should contain messages with test results from functional, regression, and now data driven tests

Some software products are not as sensitive to variability in customer data.   But most software products are.  Data-driven testing is the last mile in the journey towards getting the maximum value from automated testing.

Categories: Uncategorized

Put yourself first by taking the last place (how to sleep well at night after being acquired)

May 19, 2010 2 comments

Software is a highly rewarding, yet an intensely competitive business.   The rules are simple:  grow, become extinct, or be acquired.   There are very few software companies that manage to idle for a prolonged period of time.

When one software company acquired another, every employee wants to know, “Will I have a job in the new organization?”

One of my clients (a growing software company) purchased one of their competitors.  It wasn’t difficult to see why the company being acquired could not compete:

– There was evidence that some members of the sales team were more interested in making commissions than solving customer problems

– There was a lot of tension between Product Management and Engineering executives.  The product development team was busy justifying why it could not deliver certain functionality instead of understanding what was required to deliver a world-class product and then proposing the right budget

But the product was very well-engineered by some very talented and passionate software engineers.    All of these engineers – despite the post acquisition uncertainty in the air – continued to operate without being concerned.    All of them exhibited what I call the Golden Rules:

1.  Customers come first;  their pain is our personal pain

2.  Product comes second;  our business is to make an incredible product which makes the customer pain go away in a sustainable manner

3.  Then comes the team:  no one can succeed alone;  the enemy is outside – not inside

4.  Then come my own priorities

It became very clear why this acquisition was a bargain.  In addition to purchasing a well engineering product, the company also acquired an engineering team which lived and breathed the Golden Rules.

This team and its culture became the foundation for a new product strategy that will comfortably leave the competition behind.   To answer the question raised earlier:  every engineer in the acquired company had a job and never looked back.

Always practice the Golden Rules … and sleep well at night.

Why deployment of software is just like first impressions (very important)

About four weeks ago, the team was busy working on Release 6.5 (no need for actual product names …).

Key metrics about Release 6.5:

– 200: critical defects addressed;  these were eagerly awaited by customers
– 10:  components refactored, which in turned helped achieve …
– 70:  percent of critical functionality covered by automated tests
– 5:  late nights addressing “fixed one defect, created several new ones” symptoms
– 1:  weekend spent polishing release candidate into gold image

The team was very tired yet happy.  Quality assurance engineers ran all certification tests with no errors.

It was finally time to send the new product release to customers.

This particular product is installed by the customer in their own environment.    Within 2 days, the customer support team began to receive telephone calls.  Within 4 days, every customer that received and tried to install the new product release had some problems, preventing the product from being used.

Actual customer complaints:

– “Installation failed.  Unable to determine the root cause”
– “Tried to start the old product release.  No longer starts”
– “Finally got the old release to work.   Lost 3 days – very unhappy”

But – the customer complaint which I thought carried the most valuable message was …

“If you can’t get the installation procedure to work correctly, how can I have faith that you fixed 200 defects in this release?”

That’s why customer first experience – or first impression – with a software product starts with the installation.

Even before the customer has a chance to appreciate how the new release addressed 200 defects, the installation has to work.  Yet very often, installation procedures do not receive the same attention as the actual software being installed.

Release 6.5.1 was quickly delivered with significantly better installation instructions as well as installation software:

– Ability to verify operating environment:  OS releases, required frameworks, directory structures.  Missing components were identified

– Ability to verify the installation of the prior software release

– Ability to backup the prior software release and create a restore package

– Ability to restart installation process in case of a failure

– Ability to configure all parameters using a configuration editor

– Ability to detect if the customer changed critical files, including databases

– Ability to verify the new installation

Software installation is indeed like a first impression.  Do not underestimate how important it is for the customer to expect and experience a flawless installation process.