Home > Software Engineering > Agile can feel ungainly when migrating complex systems

Agile can feel ungainly when migrating complex systems

At some point, almost everyone in the software business either dealt or will have to deal with a complex system migration.

System migrations carry a much higher risk of being over budget and requiring more time to complete.   Can Agile development process help mitigate these risks?   By definition, system migrations require a significant amount of planning (I will talk about these elements shortly).   In some parts of the plan, Agile development process will help.  In others, it will feel very ungainly and may create more risks than benefits.

First – even before discussing planning elements, let’s talk about factors which will influence how the system migration will be planned.

– Time:  some systems have to be migrated before a busy shopping season or a critical financial milestone.  Time compression will almost always lead to a decision to split the project into parallel work streams, requiring more coordination and advance planning.

– Budget:   self-explanatory.

– Technology platform shift:   migrating a system within the technology family or between different technology families (for example, from .NET to Java or vice versa) will introduce separate considerations and additional risks.

– Technology paradigm shift:  migrating a system from a custom-built order management system to an enterprise ERP suite will introduce – again – a set of new considerations and risks.   In 1998, FoxMeyer Drug – a drug distributor – failed as a company after an implementation of a well-known ERP software also failed.    Quick Google search will produce many articles about this case.

– Data migration:  incremental, one time?  Data migration planning requires more rigor than any other stream of work in a complex system migration project.

– Operational performance:   will the new system be able to accept a 300% increase in orders?  In 5 new countries?  These – and other metrics – have to be clearly identified because the architecture required to support a certain level of business activity may be very different.

It’s important to recognize that the support from business owners (marketing, order processing, financial operations) is essential for success of any complex system migration project.   I won’t discuss business transformation elements in detail.  However, your plan should have a separate work stream with very clear milestones:  training instance ready, training program for Merchandisers developed, training scheduled, lessons learned from training processed, training delivered to 5 locations.  Anything less – and the project will fail even if all technology goals have been met.

It’s also important to recognize that the above factors make the planning phase by far the most important first step when considering how to migrate a system.  “Measure seven times, cut once”.

Applying Agile development principles can be very helpful in several areas of a complex system migration effort:

– Developing data extraction and conversion processes:  iterative data discovery approach can work very well when dealing with systems that are poorly documented

– Developing interfaces:  only after all architectural considerations have been addressed (especially performance and scalability), developing interfaces in an iterative manner can be also very effective.   I found that ‘design by contract’ approach also works very well.   Complex interfaces usually have stages of evolution, i.e. first – develop an ability to obtain real-time pricing, then – develop an ability to obtain real-time availability, etc.  Each stage of interface evolution can be designed and built in parallel with other efforts.  Again – the caveat is the same.  The design should have been already completed, including understanding of integration milestones with old and new systems.

– Developing functionality which – as a result of system migration – is new or has significant changes, for example new order approval workflow.

Can Agile feel very ungainly during complex system migration efforts?  Yes, it can.

– Any effort which has a high degree of architectural uncertainty, for example a critical interface that has significant performance implications on the rest of the system.    Once designed, the development approach to build the interface can be structured as a series of multiple iterations.

– Where there is a need to integrate with legacy systems that are not well understood or documented.

– Where there is a need to integrate with 3rd party external systems:  complying with their interfaces and requirements becomes the most important goal.

“The more complex the problem, the greater the need to consider a multitude of solutions techniques”.

Complex system migrations are a perfect example where the above holds true.

Categories: Software Engineering
  1. Michael Hayes
    March 1, 2010 at 6:02 am

    Listen to this everyone! This article does a terrific job of pointing out REAL BUSINESS implications of system migrations. It’s easy to sit in an ivory tower and say, “we do everything SCRUM,” or “we do a release every two weeks” or “we don’t write code until we have all the requirements.”

    I was once told, by a frustrated, extremely senior executive, that “we have too many ways to do things, we should have ONE way.” I took a deep breath, and said, “I’m not sure that’s the ideal goal. We should strive to things the RIGHT way.” (depending on the situation, e.g., budget, deadline, risk, etc.)

  2. March 4, 2010 at 2:40 pm

    The basic assumption here is that the more complexity a project has the less fit Agile methods become. This is because, something as complex as a migration requires extensive planning, while Agile methods are about doing. If I have understood the base assumption correctly and thus its conclusion, reserve Agile methods for moderately complex projects but stick with tried and true methods for those extremely complex situations, I’d like to offer a couple of thoughts which lead me away from embracing the assumption and its conclusion.

    Agile is a process of iterating through similar cycles and at its core I see only congruence with this and the approach needed to solve incredibly complex problems. Research, design, planning, prototyping, development, and testing all play a role in the sprint cycle. The difficulty becomes how can one break apart a vastly complex project in a way that draws out the iterative process’s value? It isn’t to say that we try to fit a square peg in a round hole, but it is to say that we attempt to decompose the square peg and reconstruct it with iterative cycles in mind.

    At the end of the day, a well developed plan, if not built and tested along the way, is only a plan. It has no inherent value and due to probable inaccuracy, will often cause followers to make poor decisions throughout the project’s lifecycle. I do not believe there is a problem that cannot be solved with Agile’s principles, but I do believe our minds are so set to solve problems linearly that when project complexity is overwhelming, we are aided and comforted by the linear approach to forming an entire solution, then executing.

    Clinton Rice

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: