SMS, Telegram or Bale? Choosing Alert Channels Without Alert Fatigue.

An alert is only useful if the right person sees it at the right time and still trusts it. Send every hiccup by SMS and people start muting their phones; send everything to a busy group chat and the one alert that matters scrolls away. Choosing channels by urgency fixes both problems.

Key takeaways

• Use SMS for the few alerts that must wake someone up.
• Use Telegram or Bale groups for team awareness and lower-urgency problems.
• Route by monitor: checkout and payments are not the same as a marketing page.
• Send recovery messages too, so nobody keeps working on a problem that is already gone.
• Review alerts monthly and remove the ones nobody acts on.

What each channel is good at

Alerts in two channels at once are not wasteful for important monitors: if one network has a problem, the other path still arrives.

Route by urgency, not by habit

Split monitors into two or three levels:

1. Critical - customers can't use the service: homepage down, checkout failing, API unreachable. SMS to the on-call person plus a message in the team group.
2. Warning - something degrades or will break soon: slow responses, certificate expires in 14 days, one location failing. Team group only.
3. Info - useful to know, no action needed now: maintenance started, a monitor recovered. Group or daily summary.

In noc724 every monitor chooses its own channels, so the payment API can page someone while a blog page only posts to Telegram.

Make each alert readable at a glance

A good alert tells the reader what broke and how badly before they open anything:

• the monitor name in plain words ("Checkout API"), not an ID,
• what failed (timeout, status 503, keyword missing) and from which locations,
• when it started,
• a link to the monitor for details.

Avoiding alert fatigue

Alert fatigue sets in when people get more alerts than they can act on. Signs: alerts acknowledged without looking, muted groups, the phrase "it does that sometimes". Remedies:

• Confirm before alerting. Require failures from more than one location, or two consecutive failed checks. See how to cut false alarms.
• One incident, one alert. Don't send a new message for every failed check of the same outage.
• Pause planned work. Put deploys and maintenance in maintenance windows.
• Delete alerts nobody acts on. If an alert never leads to an action, change it to a report or remove it.

Send recovery messages

"Back up" messages close the loop. Without them, someone woken at 3 a.m. starts investigating a problem that fixed itself two minutes later. Include the total downtime in the recovery message so the scale is clear.

Keep contact details current

• Re-verify phone numbers when people change roles or SIM cards.
• Make sure team groups contain the people currently on call, not only the founders.
• Test the path: trigger a test alert after any change to channels.

Frequently asked questions

• Should every alert go by SMS?

  +

  No. Reserve SMS for problems that need action now. If people receive more than a few SMS alerts a week that they don't act on, they will start ignoring all of them.

• What if Telegram is unreachable during an outage?

  +

  That is a good reason to add a second channel, such as Bale or SMS, for critical monitors. Different channels rarely fail at the same time.

• How quickly should an alert arrive?

  +

  Within a minute or two of confirming the failure. Faster isn't better if it comes from an unconfirmed single check that turns out to be a false alarm.