Home > Hiring, Software Engineering > Software testing: from elusive and insufficient to attainable and just right

Software testing: from elusive and insufficient to attainable and just right

Even today, the state of software testing can be described as elusive and insufficient.

In short – insufficient software testing carries significant business risks.   Global enterprises invest millions of dollars developing & supporting mission critical software which should enable sales, marketing, supply chain, and customer service organizations to serve their customers better than  competition.

Another perspective at this multi million dollar investment: global enterprises literally fund and operate their own internal software company to gain a competitive advantage.

What if this internal software company just doesn’t do enough when it comes to software testing?  What is the true cost of insufficient software testing?  The answer is both obvious and sobering.

Because every functional area is enabled by a significant IT investment …

every functional area carries a higher cost due to insufficient software testing …

… leading to lower margins and the loss of a competitive advantage if critical IT projects are late.

Let’s ask (and later respond to) three questions.  These questions will help anyone determine the state of software testing in their own organization:

One:  do you design, build, and then test?  Or – think about testing after most of the development has already been completed?

Two:  how do you characterize individuals that perform software testing:  embedded & engaged? Or – disconnected?

Three:  who are they?    The best and the brightest? Or – second class citizens in the technology organization?

Let’s respond to question One.

Software testing – by definition – is an integral part of software delivery cycle.   Software testing must be integrated in all critical areas:

– Design:  should be questioned without any hesitation as early as possible by those who test.   How can this be tested?  Is there another design approach which might make it easier to create an automated test in the future?

– Process:  the benefits of writing automated tests during development cycle are well-known.  Set a realistic code coverage goal:  80% of all critical functionality must be covered by automated tests.   Monitor this goal during development.   Do not release software unless the goal is met.  Create and monitor measures of success to ensure individual accountability.

– Schedule:  schedule must reflect not only software testing but also all the different types of testing required.   Otherwise, schedules become unrealistic.    If the design specifies 10,000 transactions per hour, appropriate tests must be planned and scheduled well in advance to determine if this scalability level can be reached.   That’s why those who test should engage early as equal partners.

Responding to question Two is now much easier.  If those who test engage early as equal partners, they will be embedded and engaged.

I think it’s best to leave question Three unanswered so the reader can answer it themselves.

Instead I will briefly discuss changes that are essential to create the organizational culture where software testing is an equal partner.

– Recruit, hire and retain the best and the brightest for software testing roles

– Create a compensation model that rewards software testing excellence

– Define and practice software testing as a true engineering discipline, not just an exercise in writing scripts

– Make software testing an integral element of delivering great products

  1. March 15, 2010 at 2:22 am

    This is a great insight about software testing. Definitively, the software testing is a much underestimated task.

    Sometimes, I like to think there must be a software testing team as big as software development team. Specially, because if you are serious about to do proper testing you need to write as much code as to make the real function itself. Its awful thing, I guess, that may be why developers tend to avoid the testing topic. But a project leader may beneficiate a lot by reading an article like this: I will share it. Programmers do better if they know there is a testing department ready to laugh at their code.


  2. March 15, 2010 at 1:29 pm

    It isn’t necessarily true that your code volume for testing is as big as the codebase that you are testing. This is why process, and toolsets are so important. We use a flavor of test driving development (TDD) at my company. Once our data model gets fleshed out, we run a code gen session and get our corresponding business objects generated. It also generates HTML pages along with the test harnesses. Since we use an OR/M this is a straightforward task. All basic CRUD operations are tested including testing for failure with out of band data/invalid datatypes/null data etc etc. Since this is all code gen’d and tested with tools, almost no programming effort is required to validate that these basic functions work. After we get the OK on our data model we can get a framework running for a decent sized project in an afternoon.

    Now, where we DO spend a lot of time in testing is in the rest of the process – validating UI’s, business rules and performance goals. This is where it is critical to have a member of your team who is engaged deeply into the system and understands how, and why it functions the was it does (or should).

    • March 15, 2010 at 1:37 pm

      Michael, thank you for sharing. This is a great description of how a successful software testing approach could be implemented.

    • March 15, 2010 at 3:03 pm

      Hi Michael,

      Not in all development efforts the business objects meets the data model and tools like the ones you are mentioning (which I have also made use of) are not of too much use on that case. Those tools tends to control null, datatype and length. But there is also business rules, fields dependency, network connectivity, long etc. The development effort can even be behind a frameworks like CAB/SCSF or CSLA and one would like to validate some other things related to interoperability between objects with those frameworks. A lot of things you can control before having the UI testers dirtying their hands with the application.

      Probably, your suggestion works for you, but certainly not for us. Probably, viceversa, too.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: