How and why should you apply an inbox zero policy when it comes to your exceptions
Inbox zero is a concept that has been around for a while, and one that tries to help you keep a clear email inbox and a focused mind. Now imagine, what if you could take this concept, and apply it to your exception handling process? If this question made you raise your eyebrow, keep on reading.
In the following post we’ll try and tackle the inbox zero concept from a new perspective, and see how it can be incorporated into the world of production monitoring. Let’s go clear some errors.
[This blog post is included as chapter 3 of our free Guide to Automated Root Cause Analysis. Download the full eBook here.]
— OverOps (@overopshq) August 2, 2017
An inbox for exceptions
A production environment is a lot like a living organism, and that’s where the real-time action is happening. There’s a lot going on behind the scenes, and we need to be aware of everything that might damage this delicate operation.
One of the core elements of a production environment is exceptions. No matter if they’re new or recurring; they indicate a warning signal that requires attention. In Joshua Bloch’s book “Effective Java” he says that “When used to best advantage, exceptions can improve a program’s readability, reliability, and maintainability. When used improperly, they can have the opposite effect.”
In addition to that, when exceptions are misused, their sheer number may cause performance issues and leave you unaware of them.
That’s why we need a better way of managing exceptions, and this is where the concept of an exception inbox shines. We heard about this developing practice from engineering teams we spoke to, and thought it would be worth to cover the subject in this blog post for our readers.
An exception inbox is an actual inbox that holds all of the events that happened in a certain time frame, with the aggregate information about each unique event. It allows us to go through different issues and manage them as needed. On OverOps, it looks like this:
Exception Inbox Zero: A similar concept for a similar issue
The Inbox Zero concept was developed by the productivity expert Merlin Mann, and it centers on handling every email as an immediate task, until there are no emails left in the inbox folder. Emails, whether they hold specific requests inside them or not, translate to a to do list since they require our attention.
Each email should be translated to an action: acknowledge and archive, delete, reply, or defer for a later date when you can take care of it. Yet, sometimes we choose to push aside the “known” and “familiar” email, to be taken care of at a later unknown date. Or in other words, never.
Our exception inbox will be where we collect, view and handle exceptions, and to that we’ll apply the inbox zero concept. Just like email, we have the tendency to treat them at a “later” date, but we need to acknowledge each exception, take care of it and eventually eliminate it. Otherwise, we’ll have to face major downsides, some we might not even be aware of:
The top 5 disadvantages of not handling exceptions
Disadvantage #1: Experiencing unnecessary overhead
Every application’s boogieman is inefficient use of memory and CPU resources, and it makes sense. It can be downright destructive. Whenever an exception is thrown, it creates an object and sends data to the log. It’s not much when it comes to a single occurrence of an exception, but what happens if it fails millions of times? Before we know it, the heap gets clogged, the CPU is wasting precious cycles on meaningless tasks, and we can kiss performance goodbye.
Aleksey Shipilëv, Principal Software Engineer at RedHat, who has been working on Java performance for more than 10 years, has an optimistic rule-of-thumb. According to him, the frequency of common exceptions should be 10^-4, which is a single occurrence of an exception for 10,000 calls into the method it’s thrown from. An error rate of 0.0001%. Anything beyond that deserves your attention.
What should we do? Keep an eye out for overhead. If an exception happens more often than 0.01% of the times the method it’s in has been called (at the very least), its root cause should be taken care of. Poof.
Disadvantage #2: Not understanding how the application really works
The best way to figure out how a toy work is by taking it apart. Same goes for applications. We need to increase our application’s Observability. In control theory, Observability is a measure for how well internal states of a system can be inferred by knowledge of its external outputs.
Exceptions are the external outputs that can give us knowledge regarding the state of the application. They give us an exclusive look backstage, which could be especially handy when debugging someone else’s code or working on legacy code.
What should we do? Promote Observability. Exceptions are a big part of this process, and you need to use the proper tools and techniques in order to understand what’s going on within the application.
Disadvantage #3: Filling your logs with noisy events
When an error happens, everyone’s go-to-solution is usually to look at the log. In it they hope to find the needed information to know why did the error happen and if there’s an exception that could help shed more light on it.
The thing is, logs usually contain massive amounts of information that often has no real use. If something breaks unexpectedly you’ll have to sift through the logs, and it feels like looking for a needle in a haystack.
What should we do? As their name suggests, exceptions should be used for exceptional scenarios that should be dealt with as soon as possible. It could be a ticket, a hotfix or even acknowledgment of the issue – but it should get the right attention.
Disadvantage #4: Causing developers to be unhappy
According to a study that focused on “The Unhappiness of Software Developers“, the number one reason that leads to unhappiness is being stuck on an issue. Unfortunately, it’s one of the most common tasks we face throughout the day, usually when debugging issues.
Being stuck on issues and debugging means that there’s no time to work on features or write new code. We usually find ourselves playing Whac-A-Mole with exceptions, trying to eliminate them before they pop-up again.
What should we do? Apply better exception handling techniques. It will require a deep dive into your existing exceptions and taking care of each one, a required step towards applying an Inbox Zero technique to manage your exceptions.
Disadvantage #5: Inability to focus on what actually matters
Exceptions cloud the developer’s view. As exception numbers increase, it’s harder to know which exceptions are more important than others. This could lead to missing a major issue or dismissing an exception that requires immediate attention.
What should we do? Unexplained performance issues that have been haunting your application could be easily solved when you have your exceptions under control. It’s like the cholesterol of your application. And you should keep their level under control.
Next step: Run a check up for your application
You should be aware of when an exception happens more often than 0.01% of the times the method they’re in is called. But how can you tell what’s your exception rate? That’s one of the questions we were asking ourselves, that lead us to create OverOps.
OverOps gives you a complete overview of your exceptions, pinpointing the root cause for each error that occurred in production. With one click, OverOps shows you the complete source code and the variable state that caused them, across the entire call stack.
With OverOps you’ll be able to see how many times each exception happened, and get the detailed error rate of how many times it failed. In less than 5 minutes you can start counting (and solving!) your exceptions. Get a personal walkthrough of the Exception Inbox Zero concept and start calculating your exception rate within minutes.
The official Java documentation states that “An exception is an event that occurs during the execution of a program that DISRUPTS the normal flow of instructions”. But more often than not, exceptions are abused and used as part of the normal application flow, and that’s less than ideal to say the least.
Just as the name implies, exceptions should be exceptional. Each one should be actionable and result in code changes, in order to eliminate the exception altogether. The Inbox Zero method will get you one step closer to a zero-exception environment, which can lead to a better performing application and a cutting edge user experience.