What Makes a Good Alert

Today I learned what makes an alert to be a good alert. As a developer, we sometimes forget about what happens AFTER we release our code. The monitoring part is an important stage of making the software or product a good and stable one.

Based on podcast episode “Alerts (or – Avi was the first to detect)” of No Tarbut Developers

On the one hand, we want to know immediately about a case of an error without missing one, and we want to know that before the customer does. But on the other hand, if we face a huge amount of them we’ll get alert fatigue and start ignoring them. A good alert contains all H.U.D.I.N.A. attributes:

1. Human required – if the alert can be automated, then automate it. If no Human is required or can solve the alert problem, then why alert one?
2. Urgant – this can be devided into 3 levels of urgancy. Whatever dont fall into one of these should not be alerted (but do need to notify somehow)
   1. Now – for a NOC or on-call person to answer the alert
   2. Tommorow morning – so the alert can be supressed, maybe to shutdown/restart a service for the meantime. But this is the first thing the team will do next day.
   3. Next relesae – important to alert but not THAT urgant.
3. Descriptive and Informative – the alert MUST be informative and contains descriptive data of how/what may cause is. err - network failure is much less descriptive than err pulling config from server.domain.com - got 502 resspone code. you might want to check server disk
4. Nessacary – this usually refer to error suprration. if some user get 4XX code it is not that nessacery to solve right now. but if you see some user experiancing 1-2 401 unauthorized so it is not that important. But many 401 unauthorized might mean that someone is trying to brute-force your service
5. Actionable – might be the first thing to actually think of – is there something I can do about the alert? But also refer to 1- it must be human required. Ususally it will be simple like restting or deleting some files and it is OK to require human operator for that to double check, to make sure all other componants are working, etc… but if you say “it happen all the time nothing to do”, “It will fix in a minute, you can ignore it” than you got it wrong.

You can hear more about it in the podcast [Hebrew]: notarbut.co/ep104-alerts/