Home > Error Management, Software Engineering > Plan for negative testing (or your customers will make the plan for you)

Plan for negative testing (or your customers will make the plan for you)

How much negative testing does your testing plan include?  Good question.

Negative testing is a broad topic with plenty of coverage (so this is not a tutorial).  I will cover the reasons why negative testing is important, impact of insufficient negative testing, and several practical recommendations.

Why negative testing is important

Negative testing – or actively seeking what does not work – allows all of us to witness how our software handles small or potentially serious problems without the customer seeing these problems or – worse – actually experiencing these problems.

Negative testing is much more than determining if the user entered a blank password or a password not meeting minimum security standards, i.e. length, character mix, and previous passwords used.

Since the topic of negative testing has been covered in detail by others, I will limit the discussion to 2 very important targets of negative testing (see Recommendations):  state transitions and error handling.

Impact of insufficient negative testing

The biggest impact:  customer’s confidence in your software.  “If this simple functionality fails, what else may broken”.

Some of the most difficult problems (that negative testing should include) are state transition failures.  Imagine a new customer placing an order and then canceling it.  The software fails to recognize order cancellation request, still accepts the order, ships a large item to the wrong address (because new customer registration process fails to detect a duplicate name), and charges customer’s credit card.   In addition to software problems, there are now real business problems:  shipment that should have never occurred, credit card charges that need to be reversed, scheduling pickup / return of a large item (could be very expensive) and incorrect customer information.

Recommendations

First – always include individuals responsible for creating and executing test scenarios as early as possible, especially during design activities.  I suggest to identify applicable negative testing scenarios during the normal course of design discussions.   Even the most elegant design will no longer seem elegant when a critical problem is discovered by a customer – and becomes critical enough to be solved yesterday.   This customer will be very disappointed by the delays caused by complex testing procedures.

Rule 1:  think of negative testing an another, essential element of the design process.  Any design improved by testing considerations will save a lot of time later.

It’s important to note that planning negative tests usually requires more experienced testing engineers, especially if the target of negative testing is a series of dependent steps.  For example, a batch interface between 2 systems can be seen as a collection of distinct functional components:

– Schedule / initiate data extraction from source system
– Process data extraction
– Apply transformation rules / create a new entity
– Stage
– Transmit
– Stage (again)
– Schedule / initiate update of the target system
– Verify & update logs

Every step of the above process will require some thought how to properly structure a negative testing scenario.

What happens when an empty file is generated?  Does it mean that no transactions were found?  Or – empty file means something unexpected took place?

What happens when the file is 8 times larger than previously anticipated?  Will the update of the target system take an extra 8 hours?  Is this acceptable?

Ensuring that errors receive proper attention (detection, logging) is also a part of negative testing.  Every error is an opportunity to show to the customer how much we care if an error takes place.   There is nothing more disappointing than a misleading error message.  But – customers will appreciate accurate and descriptive error messages, even if the error occurred.

Rule 2:  prioritize and plan specific negative testing scenarios based on the functional path of what’s being tested.  Do not forget to include error management and error messages in negative testing scenarios.

Finally, always include additional time in design and testing activities for negative testing.   I suggest to define specific negative user stores or use cases and include them in the project plan of choice.  Plan, plan, and plan again … which brings us to the final Rule 3.

Rule 3:  plan for negative testing or your customers will plan it for you.  However in the latter instance you will no longer control the schedule.

Advertisements
  1. Delcuvellerie
    February 8, 2010 at 8:11 am

    150% agree with your statement about negative testing. It has to be planned in the cycle of development and include third parties if applicable. Questions : how do you capitalize on key learnings of negative testings from a project to another one ? Is it an internal activity or could it be outsourced ? Why ? Do you have examples of failure in the market due to unsufficient negative testings ? Isabelle

  2. Bob
    February 22, 2010 at 3:37 pm

    I do have to agree with the statement that you need senior QA people to create the negative testing stories/use cases during the design phase. Overall I have had limited success with QA people creating negative test scenarios during the design phase. The team has a tough time translating functional requirements into negative test cases without the software. (Some of this is due to my product being a very complicated database product with many data interrelationships.) What I have been doing is getting early releases while still in development phase and then having QA staff make their negative test cases at that time with a semi-functional product. I just plan for extra testing cycles upfront to handle bugs due to negative testing which end up being the majority of our bugs. I generally use history combined with the complexity of the feature to estimate how long QA process will take.

  1. November 5, 2012 at 7:03 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: