Plan for negative testing (or your customers will make the plan for you)

How much negative testing does your testing plan include?  Good question.

Negative testing is a broad topic with plenty of coverage (so this is not a tutorial).  I will cover the reasons why negative testing is important, impact of insufficient negative testing, and several practical recommendations.

Why negative testing is important

Negative testing – or actively seeking what does not work – allows all of us to witness how our software handles small or potentially serious problems without the customer seeing these problems or – worse – actually experiencing these problems.

Negative testing is much more than determining if the user entered a blank password or a password not meeting minimum security standards, i.e. length, character mix, and previous passwords used.

Since the topic of negative testing has been covered in detail by others, I will limit the discussion to 2 very important targets of negative testing (see Recommendations):  state transitions and error handling.

Impact of insufficient negative testing

The biggest impact:  customer’s confidence in your software.  “If this simple functionality fails, what else may broken”.

Some of the most difficult problems (that negative testing should include) are state transition failures.  Imagine a new customer placing an order and then canceling it.  The software fails to recognize order cancellation request, still accepts the order, ships a large item to the wrong address (because new customer registration process fails to detect a duplicate name), and charges customer’s credit card.   In addition to software problems, there are now real business problems:  shipment that should have never occurred, credit card charges that need to be reversed, scheduling pickup / return of a large item (could be very expensive) and incorrect customer information.

Recommendations

First – always include individuals responsible for creating and executing test scenarios as early as possible, especially during design activities.  I suggest to identify applicable negative testing scenarios during the normal course of design discussions.   Even the most elegant design will no longer seem elegant when a critical problem is discovered by a customer – and becomes critical enough to be solved yesterday.   This customer will be very disappointed by the delays caused by complex testing procedures.

Rule 1:  think of negative testing an another, essential element of the design process.  Any design improved by testing considerations will save a lot of time later.

It’s important to note that planning negative tests usually requires more experienced testing engineers, especially if the target of negative testing is a series of dependent steps.  For example, a batch interface between 2 systems can be seen as a collection of distinct functional components:

– Schedule / initiate data extraction from source system
– Process data extraction
– Apply transformation rules / create a new entity
– Stage
– Transmit
– Stage (again)
– Schedule / initiate update of the target system
– Verify & update logs

Every step of the above process will require some thought how to properly structure a negative testing scenario.

What happens when an empty file is generated?  Does it mean that no transactions were found?  Or – empty file means something unexpected took place?

What happens when the file is 8 times larger than previously anticipated?  Will the update of the target system take an extra 8 hours?  Is this acceptable?

Ensuring that errors receive proper attention (detection, logging) is also a part of negative testing.  Every error is an opportunity to show to the customer how much we care if an error takes place.   There is nothing more disappointing than a misleading error message.  But – customers will appreciate accurate and descriptive error messages, even if the error occurred.

Rule 2:  prioritize and plan specific negative testing scenarios based on the functional path of what’s being tested.  Do not forget to include error management and error messages in negative testing scenarios.

Finally, always include additional time in design and testing activities for negative testing.   I suggest to define specific negative user stores or use cases and include them in the project plan of choice.  Plan, plan, and plan again … which brings us to the final Rule 3.

Rule 3:  plan for negative testing or your customers will plan it for you.  However in the latter instance you will no longer control the schedule.