Silent automation failures: how four of our own systems broke while reporting success

Our stack runs on automations we built ourselves — prospect list refueling, cold email, social posting, backend health checks, a morning briefing. For months the monitoring on all of it was basically me, occasionally remembering to go check whether something had actually happened lately. This whole audit started because I had to admit, in writing, that going and looking was the only way I ever found out something had broken.

I'd guess most small operators run the same way. You wire something up, watch it succeed a couple times, and move on. After that the system reports whatever its happiest code path tells it to report, and you find out it died whenever you happen to remember to check again.

Every one of these has the same shape: the process ran, it exited clean, and it was simply wrong about the one thing it was supposed to do.

The LinkedIn one deserves a closer look, since it was built with good intentions. After a real outage, we added a connectivity check so the poster would skip its run while the network was down. That check happened to execute under a Python install with no certificate authority bundle wired in, so every HTTPS probe it made failed certificate verification. A broad exception handler caught that and reported it as the network being offline. The check literally could not pass, ever, because it had no way to tell "the network is down" apart from "I'm misconfigured" — so it just kept reporting the version that let it quietly do nothing.

The cold-email one was even dumber: one line, half finished. The script that promotes verified prospects got pointed at a new file one evening, and the sender that reads the queue never got the memo. Both jobs kept running daily. Both kept exiting clean. One filled a file nothing read, the other read a file nothing filled and logged queue empty, for 20 days straight. Two automations that each looked completely healthy, connected by a filename that no longer matched, doing nothing.

None of this was exotic and none of it needed particularly bad luck. You write error handling for the errors you can imagine, so the ones you can't just show up as a calm log line instead of a page. A scheduler knows whether a process ran, not whether the work got done. Alert channels ship behind flags, and flags default to off. Every one of those is a fine decision on its own. Stack them and you've basically guaranteed that whatever failure you didn't think of is exactly the one nobody hears about.

The real problem is that no errors and working are not the same claim, and it's easy to forget that. A sender with an empty queue looks fine. A poster that thinks it's offline looks fine. A muted alert system looks fine too, since there's nothing left to alert on. If your bar for healthy is just nothing red, a system can sit dead for a month and never once fail your dashboard. Ours did, four times.

Everything we wire now has to carry a signal tied to the outcome, not the process. The actual fixes from the audit took an afternoon each. What we changed going forward:

• Check the outcome, not the exit code. The email sender now compares the delivery count it actually hit against what the queue says it should've sent. Zero sends on a day the queue disagrees is an alert, not a log line.
• Alert on quiet, too, not just on error. A system that normally does something every day and suddenly does nothing for three is exactly the pattern the worst failures produce.
• Un-mute by default. We found two separate flags stacked on the same alert path, both defaulting off, and neither was a decision anyone had actually made.
• Pin the environment down. No more whatever-python3-happens-to-resolve-to-today — explicit interpreter paths, explicit certificate bundles.
• Actually read the whole log sometimes, not just the last twenty lines. It's the only reason we caught the intermittent one at all.

If somebody built this for you, or you built it yourself, you don't need to read a line of code to audit it. Just ask three questions about anything you depend on:

• What number should move when this works, and where do I actually see it? If the answer is a log file, that's a no.
• Who finds out, and how fast, if that number stops moving?
• When's the last time an actual person checked the real outcome end to end, instead of trusting a dashboard that says everything's fine?

These days, when we build automation for a client, the outcome signal and the alert wiring go in as part of the same job, not an add-on later. Not because it's some best practice we read somewhere. Because four of our own systems went quiet for weeks at a time and we didn't notice, and quiet turned out to be the expensive kind of broken.