In software development, a defect can generally be considered any circumstance where the software doesn’t behave as expected or desired. Today, digital products go to market with 10-15% of defects or more unaddressed, even when they are known. One thing we hear often as quality assurance professionals is, “If you guys would just quit finding defects, the software could go live!”
This leads to the question: what makes a defect “acceptable”? When should a team worry about resolving an issue or letting it slide? Those questions and their answers are different for every project, team, and industry. It all depends on how much risk a company is willing to tolerate. As you develop your risk profile and decide when to stop taking defects seriously, here are a few high-level factors to consider.
High Risk Software Defects? The Definition is Relative
Before any development work begins, it’s essential to understand not only the end goal, but how essential it is to achieve it. Defining user stories and requirements clearly helps developers and testers understand when the vision isn’t going to be realized in execution due to a found defect. However, that doesn’t mean everyone always agrees that a defect is critical or severe—or even when they agree, that the same defect will get addressed.
For example, an unsuccessful installation on a certain operating system, or the complete failure of a single program feature, will stand out as critical defects to a quality assurance professional. But the developer or owner of a product may feel confident those issues will be resolved in future releases without the need to correct the original code before release. Other teams might simply choose to assume the risk associated with the defect rather than spend the time correcting it.
However, the cost of poor-quality software going to release in the United States was $2.84 trillion in 2018, with 37.5% of that spend driven by failures of the software, and another 16.8% driven by fixing and finding defects. This is why every team must have their own risk profile and be comfortable with risks presented by the defects they allow into the release.
Software Defects Under Limited Resources
The main excuse people use for allowing defective software products to be released is the presence of one or more constraints. Maybe it’s a desire not to spend money resolving the issue, or the deadline that passed a few weeks back, or the exhausted and grumpy team of professionals who don’t want to dig back into the code and find the source of the malfunction. Increased competition has led many organizations to especially prioritize speed-to-market over quality-on-the-market. But this is another huge gamble, especially if your system isn’t designed to expect or manage failures. All you are doing is taking on technical debt that may end up costing you more in the long run to save resources in the short-term.
Software Defects That Aren’t Defects
Non-functional defects often come about because the requirements weren’t well-defined. For instance, the developer might know there needs to be a button or a form on the page, but the color of the button, or the size of the fields, weren’t clearly described. This means you get buttons of three different shades of grey, or fields that allow totally different numbers of characters, or some that allow certain characters while others don’t. It’s worth noting that these “non-defects” can present some of the biggest hurdles to user adoption. Internally, a manager, product owner, or developer may pass these over because they are eager to reduce the rework required. But the costs of these issues being allowed in a release could be incalculable, from wasted marketing spend to a tarnished reputation or excess demand on the customer service team. Plus, that work will still have to be done again anyway.
No software will ever be totally defect-free, especially with constant updates to the systems it integrates with using APIs. The key to determining which defects to permit and manage, and which ones to dial down and resolve before release, is in assessing your willingness to be exposed to risk.