Heartbeat Monitoring: How to Know When a Cron Job Silently Stops.

Most monitoring asks a service "are you there?". Heartbeat monitoring flips that around: the job tells the monitor "I'm done", and silence is the alarm. It's the only reliable way to catch the most common cron failure of all - the job that simply stopped running.

Key takeaways

• The job sends an HTTP request (a "ping") to a unique URL each time it completes.
• If no ping arrives within the expected interval plus a grace period, you get an alert.
• Ping only on success (&&) so failed runs and missing runs both alert.
• Set the grace period to cover normal variation in run time.
• Use it for anything that runs on a schedule: backups, reports, syncs, queue workers.

Why normal monitoring can't see this

An uptime check needs something to connect to. A nightly backup has no port and no URL - it runs for a few minutes and exits. If its crontab line was removed during a deploy, the server was rebuilt, or the job hangs forever, no error is ever produced. Weeks later someone needs the backup and finds the last one is from August.

How heartbeat monitoring works

1. You create a heartbeat monitor and get a unique URL.
2. You tell the monitor how often the job runs (for example every 24 hours) and a grace period (for example 30 minutes).
3. The job requests the URL when it finishes.
4. The monitor expects the next request before *last ping + interval + grace*. If it doesn't come, the monitor goes down and alerts.

This pattern is also called a dead man's switch: the alarm fires unless someone keeps pressing the button.

Adding the ping

In a crontab

0 2 * * * /opt/app/backup.sh && curl -fsS -m 10 --retry 3 https://YOUR-HEARTBEAT-URL > /dev/null

• && sends the ping only when the backup exits with code 0.
• -f makes curl fail on HTTP errors, -m 10 caps the wait, --retry 3 survives a brief network blip.

Inside a script

#!/bin/bash
set -euo pipefail
pg_dump mydb | gzip > "/backup/mydb-$(date +%F).sql.gz"
find /backup -name 'mydb-*.sql.gz' -mtime +14 -delete
curl -fsS -m 10 --retry 3 https://YOUR-HEARTBEAT-URL > /dev/null

With set -e, any failing step stops the script before the ping.

From application code

Long-running workers can ping from their main loop - for example once a minute after processing a batch - so a worker that freezes without crashing is caught too.

Choosing the interval and grace period

Too short a grace period alerts on every slow run; too long delays real alerts. Base it on the slowest normal run you have seen.

What a heartbeat catches

• The crontab entry was deleted or never installed on a new server.
• The cron daemon stopped.
• The job fails (non-zero exit) - when you ping only on success.
• The job hangs and never finishes.
• The server is down or unreachable at the scheduled time.

It does not check that the output is correct - a backup that writes an empty file still exits 0. Add a sanity check (file size, row count) before the ping for important jobs.

Setting it up in noc724

Create a cron job monitor, set the expected interval and grace period, and copy the URL into the job. If a ping is late you are alerted by SMS, Telegram or Bale, and the monitor shows each check-in in its history, so you can see run-to-run timing at a glance.

Frequently asked questions

• What if the monitoring service itself is unreachable?

  +

  Use --retry in curl so short network problems don't cause false alarms. If the ping really fails, the monitor alerts - which is correct, because you can't confirm the job ran.

• Can I report failures explicitly?

  +

  Yes - many setups ping a separate URL or include the exit code when the job fails, so you hear about failures immediately instead of after the grace period.

• Does the ping slow the job down?

  +

  No. It's one small HTTP request at the end of the run, typically under a second.