This article is part of a series called Sensible Software Development. This series is designed to present practical, real-world information on software development. Each article discusses a topic in simple language with explanations that are easy to apply to your software development lifecycle.
The act of developing software is an expensive one, everyone knows that. When asked, most reasonable project managers and executives in charge of developing software will tell you that quality is important to them, and most software development teams will employ some form of testing. Yet, very few companies can tell you how much money their testing effort saves or costs the company. I believe that the reason for that is two-fold; first, they really don’t want to think about the ‘cost’ of testing, and second, they either don’t know what metrics to capture to measure the savings, or they have no faith in those metrics.
While I don’t prescribe to the idiom that all metrics are good, there are a few simple metrics that companies can gather to help them discover the relative costs associated with the quality of their software product.
The Relative Cost of Fixing Defects
To begin to calculate the cost associated with finding and fixing defects, you must know at which point during the development cycle those defects are found. The reason for this is because where a defect is found dramatically impacts the cost associated with fixing it. To illustrate:
IBM Systems Sciences Institute
According to available research, a defect found in the development phase (code) costs 6.5x as much to fix as a defect found in the requirements or design phase, and a defect found during testing costs 15x as much. The exact figures vary by researcher, but according to National Aeronautics and Space Agency (NASA) findings, “problems that are not found until testing are at least 14 times more costly to fix than if the problem was found in the requirements phase.” [1]
One complaint that I often hear from testing teams is that they are not involved in the requirements phase. In some industries this may be unavoidable, but in others, the solution is often a simple one; inject yourself into the requirements phase completely. Ask to be invited to the meetings, ask to review the requirements, etc. Requirements should be scrutinized just as severely as the product itself is. If you need to convince management of the necessity of your involvement during this phase, then use the material here and elsewhere to show them how finding defects in requirements will save the company money. Just remember, to speak to management you must speak their language. Managers love graphs like the one above. Keep the details at a high level and provide industry references for your data. Here is one for you; involvement of testing resources during technical design within iterative development life cycles is appearing in research proposals [2].
Interesting enough, a James Martin study revealed that the root cause of 56% of all defects is errors introduced in the requirements phase, with approximately half of these defects being the result of poorly written, ambiguous, and incorrect requirements. [3]
When you combine this with the common reality that the bulk of testing efforts occurs towards the end of the project, what you end up with is that test efforts are not providing the best return on investment that they could be.
Survey results from the Quality Assurance Institute annual software testing conference. [4]
It should be clear by now that companies who wish to reduce the overall cost of defects should be applying testing efforts earlier in the process.
Earlier I mentioned metrics, while the average cost of a defect found during the requirements stage is $1125, it is not an exact number, and can vary wildly from company to company. Thankfully though, there is a formula to measure this cost. According to Tom King and Joe Marasco[5]:
It is done in 5 steps:
- Estimate the number of screens and reports in your final project.
If you don’t know this number, you can estimate it based on the visual representation of use cases; just list the functions or activities that each actor performs in each use case, some of these activities will involve displaying screens or running reports, making it easier to count.Multiply this number by four person-weeks (pw)
development effort = 4pw * (# screens + # reports)
- Estimate the number of defects to expect at system test.
Your defect rate will very depending on your level of quality control in the development process. If you do not have historical data to base your estimate on, you can use the industry average of 1.0 defect per person-week.
- Estimate the number of requirement defects.
Again, historical data is best here, but lacking any, the industry average is 50% of the total number expected.
- Estimate rework cost
If you do not already measure the percent of rework on development projects through accurate time accounting, then estimate where you stand within the industry average of 20 to 40 percent of the development effort. If for example, your estimate is 30%, then rework cost = 0.3 * project development cost.
- Estimate the amount of rework cost due to requirement errors.
The Chaos Report [6] and Dean Leffingwell’s Managing Software Requirements [7] indicate that about half of all software defects are due to missing or bad requirements. However, the cost of finding and fixing requirements defects (because this most often occurs at test or post-release) is higher than other types of defects. They indicate that 75 to 80 percent of the rework cost is due to requirements defects.rework cost due to requirements defects = 0.75 * rework cost.
- Divide the rework cost due to requirements defects by the number of requirements defects.
cost per requirement defect = rework cost due to requirements defects / # of requirements defects
For Example:
# of screens plus reports = 25
estimated number of total defects = 100
estimated rework cost = 0.3 * $250,000 ($2,500 being average salary + benefits @ 40pw/year, making the project development cost $250,000) = $75,000
total development cost + estimated rework cost = $325,000
now estimate the rework cost that will be due to the requirements errors 0.75 * $75,000 = $56,250
and lastly, divide this rework cost by the number of requirements defects to determine the cost per requirement defect $56,250 / 50 = $1,125
So, what does all of this mean? Take a look again at the chart showing the relative cost to fix a defect at various stages. If you are not applying testing practices to your requirements and design phases, then it will cost you $1,125 to find and fix a defect in the testing stage. Whereas if you had found the defect during the requirements phase itself, it would have only cost you $75.
References
[1] Rosenberg, L., Hyatt,L., Hammer, T., Huffman, L. and Wilson, W. (1998) Testing Metrics for Requirement Quality, presented at the Eleventh International Software Quality Week, San Francisco, CA.
[2] Ramachandran, M. (1996) Requirements-Driven Software Test: Process-Oriented Approach, ACM SIGSOFT Software Engineering Notes, Vol. 21, Issue 4, pp. 66 – 70.
[3] Mogyorodi, G. (2003) What is Requirements Based Testing? The Journal of Defense Software Engineering, 12-15.
[4] riceconsulting.com, The Economics of Testing. www.riceconsulting.com
[5] Tom King and Joe Marasco, “What is the Cost of a Requirement Error?” stickyminds.com
[6] The Standish Group, “The Chaos Report”
[7] Leffingwell, Dean and Wildrig, Don. Managing Software Requirements: A Unified Approach. Addison-Wesley Professional, 1999.
