Silent automation failures: how four of our own systems broke while reporting success

Our stack runs on automations we built ourselves: prospect list refueling, cold email, social posting, backend health checks, a morning briefing. For months, the monitoring on all of it was me occasionally remembering to check whether something had happened lately. The audit behind this post started with me admitting, in writing, that the only way I found out about broken automations was going and looking.

We suspect most small-business automation is monitored the same way. Somebody wires a workflow, watches it succeed twice, and walks away. From that day on, the system reports whatever its happiest code path reports, and the owner finds out it died whenever they next happen to wonder about it.

Every row shares the same shape. The process ran. The process exited cleanly. The process was wrong about the one thing it exists to do.

The LinkedIn failure deserves a closer look, because it was built with good intentions. After a real outage, we added a connectivity check so the poster would skip runs while the network was down. That check happened to execute under a Python installation with no certificate authority bundle wired in, so every single HTTPS probe failed certificate verification. A broad exception handler caught the error and reported the network as offline. The safety check could never once pass. A guard that cannot distinguish the network is down from I am misconfigured will spend its life reporting the flattering one.

The cold-email failure was even smaller: one line, half finished. The script that promotes verified prospects was updated to write to a new file one evening, and the sender that reads the queue was not updated to match. Both jobs kept running daily. Both kept exiting cleanly. One filled a file nothing read, the other read a file nothing filled and logged queue empty, for 20 days. Two healthy-looking automations, connected by a filename, doing nothing.

None of this is exotic, and none of it required unusual bad luck. Error handling gets written for the errors the author imagined, so the surprise arrives as a calm log line instead of a page. Schedulers know whether a process ran, not whether the work happened. Alert channels ship behind flags, and flags default to off. Each of these is a reasonable engineering choice on its own. Stacked together, they guarantee that the failure you did not imagine is exactly the one nobody hears about.

The deeper problem is that no errors and working are different claims. A sender with an empty queue has no errors. A poster that believes it is offline has no errors. A muted alert system has, by definition, no errors. If your definition of healthy is the absence of red, a system can be dead for a month and stay green the whole time. Ours did.

Everything we wire now has to carry a signal tied to the outcome, not the process. The specific fixes from the audit took an afternoon each. The rules they left behind:

• Assert the outcome, not the exit code. The email sender now checks the delivery count it actually achieved against the queue it believed it had. Zero sends on a day the queue says otherwise is an alert, not a log line.
• Alert on quiet, not just on error. If a system that normally does something daily does nothing for days, that silence should page a human, because silence is exactly what the worst failures produce.
• Un-mute by default. Alert switches that need a flag turned on will spend part of their life off without anyone having decided that. We found two of those stacked on one system.
• Pin the environment. Scheduled jobs get explicit interpreter paths and explicit certificate bundles. Whatever the environment resolves to is a different program on different days.
• Read the whole log once. The tail shows the weather. The full log shows the climate.

If someone built automation for you, or you built it yourself, you can audit it without reading a line of code. Ask three questions of each automation you depend on:

• What number should move when this works, and where do I see it? If the answer is a log file, that is a no.
• Who finds out, and how fast, if that number stops moving?
• When did a human last confirm the outcome end to end, instead of the dashboard saying fine?

When we build automations for a business now, the outcome signal and the alert wiring ship with the workflow, as part of the same job. Not because it is sophisticated. Because we watched four of our own systems fail quietly in one week, and quiet is the expensive kind of broken.