The agent needs a heartbeat.

Sometime before four this morning, the always-on machine at home that runs my overnight jobs went offline.

It didn't send an error. Machines that are switched off don't send anything. The machine itself never said a word all morning.

What did speak up were the jobs around it. A little before four, a job on a different machine tried to copy files from it, couldn't reach it, and flagged that. At eight, the job that picks up the overnight output went looking for a file that should have been there. It wasn't. The job waited about a quarter of an hour, looked again, and wrote one plain line: blocked, nothing to pick up. It was the second morning in a row that the file hadn't shown up.

That is the whole essay. The most dangerous thing a production agent can do is go quiet. Errors are easy, because errors announce themselves. Silence looks exactly like a system with nothing to report. So every agent needs a heartbeat that proves it ran, a receipt that proves what it did, and something outside the agent that raises an alarm when either one doesn't show up.

How do you know an AI agent actually did its job?

You don't, unless something other than the agent checks. An agent's own "done" tells you the run ended. It doesn't tell you the work happened.

There are three separate questions hiding inside "did it work?" Did the agent run at all, at the time it was supposed to? Did it finish? And did the thing it finished actually change the world the way it was meant to: the file exists, the email went out, the row moved? Most setups only ever answer the second one.

Anthropic's guide to building effective agents (December 2024) makes the same point from the agent's side. During execution, it says, "it's crucial for the agents to gain 'ground truth' from the environment at each step (such as tool call results or code execution) to assess its progress." The agent should check reality, not its own memory of what it meant to do. The people running the agent need to hold it to the same rule.

Why "succeeded" is not the same as "done"

One of the schedulers I use is honest about this in its own documentation. A run that hits a login wall, handles it gracefully and reports "I couldn't sign in" is still marked as succeeded. That is correct from the scheduler's point of view. The run started, ran its code and exited cleanly. From mine, nothing happened.

This is the most common failure I see in agent setups, including my own. The status says green. The work didn't happen. And because the status is green, nobody looks.

Google's SRE book has language for this. In the chapter on monitoring distributed systems, it splits monitoring into white-box (looking at a system's internals, like logs) and black-box, which it describes as symptom-oriented: "The system isn't working correctly, right now." A run status is white-box. It's the agent describing itself. What you actually want to know is the symptom: is the output there?

An agent that crashes tells you it's broken. An agent that goes quiet looks exactly like an agent with nothing to say.

What is a heartbeat for an AI agent?

A heartbeat is a small signal an agent sends on a schedule, or at the start and end of every run, that says "I'm alive and I'm on time." The monitor listens for it. As long as it keeps arriving, the monitor stays quiet. The moment it's late, the monitor raises an alarm.

Ops people have a name for this pattern: a dead man's switch. It inverts the usual logic. Instead of alerting when something bad happens, it alerts when something good stops happening.

Healthchecks.io, a tool built for exactly this with cron jobs, describes it in two sentences: it "keeps silent as long as pings arrive on time. It raises an alert as soon as a ping does not arrive on time." That's the whole mechanism. It works for a nightly agent run as well as it works for a backup script.

The Prometheus monitoring world takes the idea one step further and points it at the monitor itself. The Watchdog alert in the Prometheus Operator runbooks is "always firing" on purpose. Its job is to prove the alerting pipeline works. If it ever stops firing, the runbook says, that absence "should alert external systems that this alerting system is no longer working." The alarm has its own heartbeat.

How to monitor AI agents in production

If you're running agents on a schedule this year, this is the short version of what I'd set up first:

1. Treat silence as a failure. There's one line in the operating notes my agents read at the start of every session: silence on an overdue job is a failure, so don't wait for someone to report "blocked." If a job was due at nine and there's no sign of it at ten, that is the incident.

2. Alert on the missing output, not just on errors. The useful check isn't "did the agent throw?" It's "does the thing it was supposed to produce exist?" The file, the published URL returning 200, the sent message, the updated row. Check the world.

3. Make every run leave a receipt. A line in a log that a human can read in five seconds: what ran, when, what it touched, and a link to the result. If the receipt is missing, the run didn't happen, whatever the status says.

4. Separate the checker from the doer. An agent grading its own homework will usually pass. The heartbeat monitor, the output check and the alert should live outside the agent, ideally on a different machine from the one doing the work. My Sunday problem was only visible because a job on another machine was expecting a file.

5. Trace the agent's steps in a standard format. OpenTelemetry now has semantic conventions for generative AI agents, with span types such as invoke_agent and execute_tool. They're still marked as in development, but they mean an agent run can show up in the same tracing tools as the rest of your stack instead of in a bespoke log nobody opens.

6. Page a human only when it's actionable. The SRE book's rule holds for agents too: "Every page should be actionable." A heartbeat alarm that fires every night for a job you've retired trains you to ignore the one that matters.

The layer underneath

In The agent needs a wallet I wrote about limits. In The agent needs a memory I wrote about receipts for what an agent remembers. In The agent needs its own login I wrote about knowing which agent did what. All three assume you'd notice if an agent stopped doing anything at all. Most of the time, you wouldn't.

My Sunday morning wasn't a model failure. No prompt was wrong. A machine went offline, and the only reason I knew was that the jobs around it refused to stay quiet when what they expected didn't show up.

If you're building agents, stop asking only "what happens when it fails?" Ask the harder question: how long would it take you to notice if it just stopped?

Give it a pulse. Then listen for the silence.