Search This Blog

Sunday, August 22, 2010

Bug Severity vs. Priority

Severity and priority are two ways of thinking about software bugs and deciding which ones should get fixed and in what order. Not everyone uses both. I’ve been in places where bugs are categorized by only one of them. But inevitably in those places, I find that the two ways of thinking get mixed and people end up confused. As I use them, bug severity and priority are different and we need both.

Severity is about the risk a bug poses if it gets out into the wild. I’ve written about testing and software risk elsewhere. See http://www.quality-intelligence.com/articles/DeterminingBusinessRisksForTesting.pdf.

Briefly, software risk can be characterized as:

The potential for
some fault, failure or other unintended happening
in the implemented system
to cause harm or loss
to one or more persons or organizations.

We assess the risk of a bug by asking questions about impact and probability. How much harm could this bug cause to some thing the customer cares about, such as human safety or the bottom line? How likely is this bug to manifest and how likely is that harm to occur if it does?

If a bug could cause significant harm but only manifests under very unlikely circumstances, then we might decide it’s less severe than a bug that could cause less harm but manifests frequently. Or not, depending on the context.

Most organizations have standard criteria for classifying bug severity, such as:

Severity 1 – Catastrophic bug or showstopper. Causes system crash, data corruption, irreparable harm, etc.

Severity 2 – Critical bug in important function. No reasonable workaround.

Severity 3 – Major bug but has viable workaround.

Severity 4 – Minor bug with trivial impact.

Typically, Severity 1 and 2 bugs must be fixed before release, where 3’s and 4’s might not be, depending on how many we have and on plans for their subsequent disposition.

Priority, on the other hand, is about the order in which bugs need to be fixed. Often, priority and severity run hand-in-hand: a bug is both high severity and high priority to fix. But that’s not always true. Occasionally in testing we’d like to have lower-severity bugs fixed so we can explore an area more thoroughly and see if they’re masking something else. Particularly on large projects, we can also find that we have a larger number of high severity bugs open than the programmers can readily fix. In this case, we need to specify the order for fixes based on where we plan to test next. To some extent also, ease of programming kicks in. A programmer working on high severity bugs in a particular module may choose to fix the low severity bugs in the same session.

At least theoretically, bug severity doesn’t change. The potential for business or technical impacts stays pretty much the same throughout the development project. (The passage of time can affect risk, but that’s a subject for another post.)

Priorities for fixing bugs do change depending on where we are in the project. At first, it’s testing priorities that matter most. But the closer we get to release, the more important the customer’s priorities become, to the point where they take over entirely.

And that brings us to an essential question about both severity and priority. Who gets to decide?

Ultimately, it’s the customer’s prerogative to decide both severity and priority (using “customer” as the stand-in here for “key decision-makers”). We—testers, project managers, and programmers alike—can make educated guesses about business risk and even about business tolerance for risk, but we can’t really know and we certainly can’t decide for the customer. Similarly, we can’t decide bug fix order for the customer, who frequently has different priorities from ours.

That’s not to say we abdicate responsibility. It’s a tester’s job to try and represent the customer’s point of view when they are absent. It’s also our job to help customers make those decisions. We do that by attempting to understand the true significance of a bug and communicating our understanding. We also ask questions to help our customers assess relative risk. (As anyone who has ever supported UAT knows, customers frequently assume everything is equally high risk until we ask those questions.)

It’s good to engage with customers and ask those questions early, ideally as we go through the development project. (Agile projects have it easy in this regard.) Early engagement is especially important for assigning severity.

But until we reach the point in the project of determining bug fix priorities for release, it’s only practical for the entire development team to set priorities according to what’s needed to move the project forward. Most often that means the testers’ needs: the bug fix priorities for testing.

Of course, there are other factors affecting priority during a development project, including the relative risk and cost of fixing a bug. But generally speaking, it makes sense for testers to drive priority until we switch over to the customer’s priorities for release.

6 comments:

Ajay Balamurugadas said...

>> Ultimately, it’s the customer’s prerogative to decide both severity and priority (using “customer” as the stand-in here for “key decision-makers”).

Isn't that our(testers) role to highlight the severity and leave the priority decision to the customers (key decision-makers) as we might not have information about the business decisions.

Regards,
Ajay

Anonymous said...

"As I use them, bug severity and priority are different and we need both."

I agree.

"If a bug could cause significant harm but only manifests under very unlikely circumstances, then we might decide it’s less severe..."

That's where I disagree. Severity to me is static and likelihood-independent. In fact, you hint at that above: "At least theoretically, bug severity doesn’t change."

The only difference, in my experience, it isn't theoretical. Once in awhile, the triage team will overrule a tester's notion of severity, but not because of risk, but because they interpret the severity differently than the tester.

I've always thought that risk = likelihood + impact. But to me, risk does not change severity. If the likelihood is high, the *priority* of the fix might change, but not the severity of the bug.

For example, to me, a bluescreen is always severity 1 -- the highest (crash, hang, data loss). But if it only happens at midnight when you set the system clock to February 29, 2048, I would expect it to get priority 4, unless there's some discussion that the *fault* of the bug could cause something bad that's more likely.

To me, a copyright infringment on the splash screen is a severity 4 -- the lowest. But legal issues like that are almost always a priority 1 because of the dire legal consequences. A wonky tab order is another severity 4 bug, but could be a pri 1 because the customer's satisfaction relies on being able to make quick keyboard moves to improve their workflow speed.

"We can make educated guesses about business risk and even about business tolerance for risk..."

I agree...

"but we can’t really know and we certainly can’t decide for the customer."

I disagree that we can't decide for the customer. As a member of triage (as test manager), I certainly decide on behalf of the customer, because, well, they're not there.

"Similarly, we can’t decide bug fix order for the customer, who frequently has different priorities from ours."

We can and do decide bug fix order (again, because the customer is not there). I know there's this notion of onsite customers in Agile Development, and I have seen that. But I have also seen them not know what they want until a suggestion is made on what can be done. And even then, they tend not to know what they want until they discuss it with the project staff.

Rarely has a customer been in the room during any of triage meetings I've been on for any project -- it's always been a proxy (customer support, sales, or our own experience with them) and I've seen thousands of good decisions made on their behalf.

Rikard Edgren said...

I have also always used Severity = impact (and no risk evaluated.)
I use the same examples as Jon:
crash that probably won't happen - highest severity, lowest priority
product name spelling error - lowest severity, highest priority

This makes it easier to set the Severity, and it doesn't change over time.
It also gives you a chance of doing some analysis of which type of issues you find.
The categories are more separated than in your suggestion.

On the other hand, Severity/Priority is often misunderstood intuitively, people tend to think they are the same.
Your suggestion might be faster to explain:
'severity is the importance, priority is the order we will address them in.'

/Rikard

Fiona Charles said...

Hi Ajay,

Yes, it's the tester's job (I'm not sure about our right) to highlight the severity for customers, but they also have the right to disagree and change the severity.

Fiona Charles said...

Thanks, Jon & Rikard -- interesting. This is a fundamental difference in how we use severity. In my model, sevrity is entirely about risk -- both business risk and technical risk (which inevitably has a business effect, whether or not it's immediately meaningful to business owners).

I prefer to use other fields on a bug log to categorize bugs in other ways, e.g., problem type.

Unknown said...

Very nice and interesting post. I also wrote similar lines on prioritizing bugs. Hope you would like it - http://bit.ly/1M5SEAS