Earlier this month, Herb Krasner, an Advisory Board member of the Consortium for IT Software Quality (CISQ) joined our VP of Solution Engineering Eric Mizell for a webinar examining the cost of poor quality software. A recent CISQ report [1], The Cost of Poor Quality Software in the US: A 2018 Report, aggregates publicly available data from a wide variety of sources to highlight the devastating financial impact of poor quality software.

Below, we recap webinar highlights and salient takeaways about the state of software quality today, as well as how the new OverOps cost calculator can help organizations pinpoint the specific ways application errors are hurting their bottom line.

Quality is in the Eye of the Beholder

When evaluating software quality, Herb argues that defining what quality looks like should happen before you begin writing software. He notes,

“I think that quality is one of those things that if you don’t define it or make it an objective, then it tends to fall behind the other priorities like cost and schedule. If you’re trying to get something out fast and you’re trying to get something out at minimum cost or fixed cost, then the thing that might suffer would be quality. If we don’t have it as a goal and objective, then it doesn’t get to stand on an equal footing with the functional requirements that are typically specified at the beginning of any development project.”

More specifically, when evaluating software quality on a project by project basis, Herb recommends that organizations define a quality goal model that derives from the following perspectives:

  1. Does the product conform to the original requirements?
  2. Does it fit with user expectations and meet user needs?
  3. Does it comply with required or recommended industry standards, such as ISO 25000?
  4. Does it achieve the desired structure and aesthetics?

The Era of Nine-Digit Defects

Bill Curtis, a source cited in Herb’s report, says we are living in the era of nine-digit defects [2]. According to Herb, this means that, “when we encounter software bugs, we’re looking at things with an impact of $100 million or more. These are the things that cause massive disruptions and failures in the world we live in, and cause executives to lose their jobs.”

Herb’s CISQ report estimates that, in 2018, poor quality software cost organizations $2.8 trillion in the U.S. alone. To put that in perspective, that’s roughly 10% of the country’s GDP last year. Contributing factors to this staggering number include:

  • Problems with Legacy Systems: “75% of total IT spend goes to just the care, feeding, maintenance and evolution of legacy systems that have been around for a long time, and part of that deals with poor quality,” says Herb. This accounts for roughly $635 billion in expenses.
  • Losses from Software Failures: Estimated at a cumulative $1.28 trillion based on data from major outages in the news. This cost is comprised of many things, ranging from lawsuits and stock price dips, to company valuation and brand tarnishment. Herb believes this number really only scratches the surface as there are plenty of other software failures that don’t necessarily make the headlines of major news organizations but still have massive customer impact.
  • Troubled or Cancelled Projects: When we look at costs from troubled and canceled projects, we’re taking into account software projects that were either outright canceled, way behind schedule or way over budget – something many of us have experienced first-hand. Derived from research from The Standish Group [3], this aspect is  estimated to cost approximately $178 billion.
  • Finding and Fixing Defects: A report from Capers Jones found that 60% of US software engineering work-time centers around finding and fixing errors [6]. This represents roughly $500 billion in costs from the software developers, testers and QA specialists dealing with problems and deficiencies that pop up during the course of the software development lifecycle. This could even reflect lost employee productivity, delayed product roadmaps and employee turnover.
  • Technical Debt: Herb defines this as “the delayed work that happens when shortcuts are taken, mostly due to schedule pressure, or just unintentional bad work, and future costs of latent defects that might pop up.” A report from CAST Software [5] analyzed the latent weaknesses, vulnerabilities, and errors from 1,400 applications and what it would cost in the future to service those issues. The research found that the cost of technical debt for an average application was $1.1 million, based on the estimation that an average application in their data set is about 300,000 lines of code. In the case of Java applications, CAST estimates the cost is likely higher – closer to $1.6 million.

What’s more, Herb notes that we are quickly moving into the era of 10-digit defects:

“I think we are in a situation where we live in our society with a suite of highly vulnerable technologies that are increasingly disrupting our lives. At the same time, as we try to innovate and we try to see more things computerized with the internet of things, and with artificial intelligence, and virtual reality, and all these new technologies coming into the picture, I think we’re gonna see more disruption and more problems. I only see things getting worse, not better – at least in the short-term.”

A 2017 report from Tricentis [4] examined more than 600 headline-making software failures featured in English language media outlets and found that those failures alone resulted in a cumulative 268 years of downtime and $1.7 trillion in financial losses. If the Boeing 737 MAX 8 fiasco of 2019 is any indication, these newsworthy and expensive failures  are likely to persist.

It probably comes as no surprise that the biggest culprit behind these headline-making failures is software bugs.

Calculating the True Cost of Your Own Software Errors

So how can you really know what errors are costing you?  

Herb recommends that every software/IT organization and project collect specific data that would allow them to compute the costs of both good quality software and poor quality software. He has created a taxonomy and categorical model for that purpose. However, for many organizations, those mechanisms are not yet in place and maturity in this area can take time.

As an alternative starting point, OverOps put together a calculator with the input of several large banks and credit card companies to help organizations gauge just how much their software quality issues are costing.

Examining things like the number of customer-facing applications, release frequency and average number of SLA violations per month, we’re able to provide you with the estimated costs of everything from infrastructure overhead to lost developer productivity to brand tarnishment.

Answer a few quick questions here, and see how much errors are affecting your bottom line!

Sources

  1. Herb Krasner, Consortium for IT Software Quality, Sept. 26, 2018 – The Cost of Poor Quality Software in the US: A 2018 Report,
  2. Bill Curtis, Oct. 14, 2016 – The era of nine-digit defects
  3. The Standish Group – CHAOS Report 2016: Outline
  4. Tricentis – Software Fail Watch
  5. CAST – Technical Debt Estimation, Crash Reports, How to Monetize Application Technical Debt, CAST Software Analyst Report, 2011
  6. Capers Jones, WASTAGE: THE IMPACT OF POOR QUALITY ON SOFTWARE ECONOMICS, Version 8:0 September 3, 2017
Tali has a degree in theoretical mathematics and covers topics related to software monitoring challenges. In her free time, Tali enjoys drawing, practicing yoga and spending time with animals.