Archive for February, 2010

Agile can feel ungainly when migrating complex systems

February 28, 2010 2 comments

At some point, almost everyone in the software business either dealt or will have to deal with a complex system migration.

System migrations carry a much higher risk of being over budget and requiring more time to complete.   Can Agile development process help mitigate these risks?   By definition, system migrations require a significant amount of planning (I will talk about these elements shortly).   In some parts of the plan, Agile development process will help.  In others, it will feel very ungainly and may create more risks than benefits.

First – even before discussing planning elements, let’s talk about factors which will influence how the system migration will be planned.

– Time:  some systems have to be migrated before a busy shopping season or a critical financial milestone.  Time compression will almost always lead to a decision to split the project into parallel work streams, requiring more coordination and advance planning.

– Budget:   self-explanatory.

– Technology platform shift:   migrating a system within the technology family or between different technology families (for example, from .NET to Java or vice versa) will introduce separate considerations and additional risks.

– Technology paradigm shift:  migrating a system from a custom-built order management system to an enterprise ERP suite will introduce – again – a set of new considerations and risks.   In 1998, FoxMeyer Drug – a drug distributor – failed as a company after an implementation of a well-known ERP software also failed.    Quick Google search will produce many articles about this case.

– Data migration:  incremental, one time?  Data migration planning requires more rigor than any other stream of work in a complex system migration project.

– Operational performance:   will the new system be able to accept a 300% increase in orders?  In 5 new countries?  These – and other metrics – have to be clearly identified because the architecture required to support a certain level of business activity may be very different.

It’s important to recognize that the support from business owners (marketing, order processing, financial operations) is essential for success of any complex system migration project.   I won’t discuss business transformation elements in detail.  However, your plan should have a separate work stream with very clear milestones:  training instance ready, training program for Merchandisers developed, training scheduled, lessons learned from training processed, training delivered to 5 locations.  Anything less – and the project will fail even if all technology goals have been met.

It’s also important to recognize that the above factors make the planning phase by far the most important first step when considering how to migrate a system.  “Measure seven times, cut once”.

Applying Agile development principles can be very helpful in several areas of a complex system migration effort:

– Developing data extraction and conversion processes:  iterative data discovery approach can work very well when dealing with systems that are poorly documented

– Developing interfaces:  only after all architectural considerations have been addressed (especially performance and scalability), developing interfaces in an iterative manner can be also very effective.   I found that ‘design by contract’ approach also works very well.   Complex interfaces usually have stages of evolution, i.e. first – develop an ability to obtain real-time pricing, then – develop an ability to obtain real-time availability, etc.  Each stage of interface evolution can be designed and built in parallel with other efforts.  Again – the caveat is the same.  The design should have been already completed, including understanding of integration milestones with old and new systems.

– Developing functionality which – as a result of system migration – is new or has significant changes, for example new order approval workflow.

Can Agile feel very ungainly during complex system migration efforts?  Yes, it can.

– Any effort which has a high degree of architectural uncertainty, for example a critical interface that has significant performance implications on the rest of the system.    Once designed, the development approach to build the interface can be structured as a series of multiple iterations.

– Where there is a need to integrate with legacy systems that are not well understood or documented.

– Where there is a need to integrate with 3rd party external systems:  complying with their interfaces and requirements becomes the most important goal.

“The more complex the problem, the greater the need to consider a multitude of solutions techniques”.

Complex system migrations are a perfect example where the above holds true.

Categories: Software Engineering

Plan for negative testing (or your customers will make the plan for you)

February 7, 2010 3 comments

How much negative testing does your testing plan include?  Good question.

Negative testing is a broad topic with plenty of coverage (so this is not a tutorial).  I will cover the reasons why negative testing is important, impact of insufficient negative testing, and several practical recommendations.

Why negative testing is important

Negative testing – or actively seeking what does not work – allows all of us to witness how our software handles small or potentially serious problems without the customer seeing these problems or – worse – actually experiencing these problems.

Negative testing is much more than determining if the user entered a blank password or a password not meeting minimum security standards, i.e. length, character mix, and previous passwords used.

Since the topic of negative testing has been covered in detail by others, I will limit the discussion to 2 very important targets of negative testing (see Recommendations):  state transitions and error handling.

Impact of insufficient negative testing

The biggest impact:  customer’s confidence in your software.  “If this simple functionality fails, what else may broken”.

Some of the most difficult problems (that negative testing should include) are state transition failures.  Imagine a new customer placing an order and then canceling it.  The software fails to recognize order cancellation request, still accepts the order, ships a large item to the wrong address (because new customer registration process fails to detect a duplicate name), and charges customer’s credit card.   In addition to software problems, there are now real business problems:  shipment that should have never occurred, credit card charges that need to be reversed, scheduling pickup / return of a large item (could be very expensive) and incorrect customer information.


First – always include individuals responsible for creating and executing test scenarios as early as possible, especially during design activities.  I suggest to identify applicable negative testing scenarios during the normal course of design discussions.   Even the most elegant design will no longer seem elegant when a critical problem is discovered by a customer – and becomes critical enough to be solved yesterday.   This customer will be very disappointed by the delays caused by complex testing procedures.

Rule 1:  think of negative testing an another, essential element of the design process.  Any design improved by testing considerations will save a lot of time later.

It’s important to note that planning negative tests usually requires more experienced testing engineers, especially if the target of negative testing is a series of dependent steps.  For example, a batch interface between 2 systems can be seen as a collection of distinct functional components:

– Schedule / initiate data extraction from source system
– Process data extraction
– Apply transformation rules / create a new entity
– Stage
– Transmit
– Stage (again)
– Schedule / initiate update of the target system
– Verify & update logs

Every step of the above process will require some thought how to properly structure a negative testing scenario.

What happens when an empty file is generated?  Does it mean that no transactions were found?  Or – empty file means something unexpected took place?

What happens when the file is 8 times larger than previously anticipated?  Will the update of the target system take an extra 8 hours?  Is this acceptable?

Ensuring that errors receive proper attention (detection, logging) is also a part of negative testing.  Every error is an opportunity to show to the customer how much we care if an error takes place.   There is nothing more disappointing than a misleading error message.  But – customers will appreciate accurate and descriptive error messages, even if the error occurred.

Rule 2:  prioritize and plan specific negative testing scenarios based on the functional path of what’s being tested.  Do not forget to include error management and error messages in negative testing scenarios.

Finally, always include additional time in design and testing activities for negative testing.   I suggest to define specific negative user stores or use cases and include them in the project plan of choice.  Plan, plan, and plan again … which brings us to the final Rule 3.

Rule 3:  plan for negative testing or your customers will plan it for you.  However in the latter instance you will no longer control the schedule.