A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. October 16, 2026.
An hourly job hunter also reads two mailboxes for employer replies. An hour that found no new jobs ended early, before the mailbox step, so replies were read in 278 of 466 cycles, with gaps of up to 14.4 hours.
What it looked like from outside
While the bot's help screens were being rewritten, a read-only check compared every sentence with the service's own log, including how often the job-search inboxes were read. Over the 466 hourly cycles in the retained log (27 September to 16 October 2026), the mailbox step started 278 times. The hours that skipped it did not look broken: each logged a calm, true line, "No new jobs this cycle (N consecutive) - waiting for next run", and the service never crashed; its restarts in that period were all clean, deliberate ones.
What was actually happening
One function runs the whole cycle as numbered steps: find jobs, score and route them, send a summary, send follow-ups and warm introductions, then read the mailboxes for replies. When the search found zero new jobs, the function returned right after saying so. That was correct for the scoring steps, which had nothing to score, but it also skipped the mailbox step, which has nothing to do with whether new jobs exist. Reading replies shared the fate of finding jobs: 187 of the 466 cycles found nothing new, so about four hours in ten never looked at the inbox. A 24-hour look-back in the mailbox scan hid the gap. The longest stretch between two checks was 14.4 hours, fourteen quiet cycles in a row, inside the window, but that was margin, not design: a quiet spell longer than a day would have made replies invisible.
The fix
In the zero-jobs branch, the same reply-check block now runs before the early return. Nothing else in the cycle changed, and the existing de-duplication still alerts each email once. Because the check now runs every hour, the reply-check mail connections also got a 60-second timeout, so a hung mail server can block the single event loop the cycle shares with the chat bot and a daily marketing post for about a minute per call instead of indefinitely. The change does not touch the marketing post's code or schedule.
How I know it worked
From the service log. Before the fix, over the retained log from 27 September to 16 October 2026: 466 cycles, 187 with zero new jobs, 278 mailbox checks, and a longest gap of 14.4 hours between two checks. An independent fact-check recounted those numbers from the same log and matched them. After the deploy, the first quiet cycle, at 17:55:59 UTC, logged "Found 0 new jobs", then "No new jobs this cycle (1 consecutive) - waiting for next run", then "Checking for responses", and "Response check complete" at 17:56:43: the step a quiet hour used to skip now runs. The cycles that found jobs that afternoon completed the check as before.
The rule this earned
Count how often each duty actually runs from its own log line, not how often the loop runs. When one function carries several duties, an early return is a decision about all of them: put a duty that does not depend on the branch before the branch, or give it its own trigger.
The named concepts behind it
Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.
Fate-sharing
Two duties that should be independent ride the same code path, so when one has nothing to do, the other quietly stops too.
In network design, fate-sharing is a deliberate choice: David Clark's 1988 paper on the design of the Internet's protocols argued that the state of a connection should live with the endpoints that use it, so that the state is lost only when those endpoints themselves are lost. Two things that share fate fail together, and that is the point.
The bug is the unintended version: two duties that have nothing to do with each other end up sharing fate because they happen to live in the same function, the same job, or the same branch. The usual carrier is an early return. A loop does A, then B, then C; someone adds "if A found nothing, stop here" — correct for B, which depends on A — and C, which never depended on A, now runs only when A happens to be busy.
Ordinary-life version: a night guard is asked to sign for deliveries and, while at the door, to check the mailbox. One day the instruction becomes "if there is no delivery tonight, go back to your desk." Quiet nights get quieter. Nobody told the guard to stop checking the mail; it just stopped happening on every night without a parcel.
The usual shapes:
- The early return. "Nothing new — waiting for next run" ends the whole cycle, including the steps that had their own reason to run.
- The shared try block. One failing step aborts every step after it, so a broken scraper also silences the health report written at the end.
- The chained schedule. Job 2 is triggered by job 1's output; when job 1 legitimately produces nothing, job 2 never runs.
- The monitor on the patient. The thing that watches a system runs on that system, so the outage it should report also takes the reporter down.
It is a special case of silent failure: the skipped duty leaves no trace, and the line that is logged ("no new jobs") is true and calm. It also hides behind liveness is not correctness: the loop runs every hour, so every liveness signal is green, while one of its duties runs only some of the hours.
The defences. Put a duty that does not depend on a branch before the branch, or give it its own trigger. Treat every early return as a decision about everything after it, not only about the step that motivated it. And measure each duty's real frequency from its own log line (verify from logs, not config) — count "mailbox checked" per cycle, not "cycle ran".
Silent failure
The system did something reasonable, and told nobody.
The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.
A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.
The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:
- Log the outcome, not the attempt. "sending notification" tells you nothing. "notification DELIVERED (id 4661)" versus "notification REJECTED 400" tells you everything.
- Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.
Liveness is not correctness
A dead job announces itself. A job that runs perfectly and emits slightly wrong output never will.
Almost every check you own measures liveness: did it run, did it return, did it exit zero, did it publish. Almost none measures correctness: was the thing it produced the right thing. These are different properties, and the gap between them is where the expensive incidents live.
The asymmetry is what makes this dangerous. A job that stops firing is loud — the output is missing, someone notices within a day. A job that fires on schedule and produces output that is subtly wrong is silent, and it stays silent for as long as nobody reads the output, because every signal you have is reporting the truth. The scheduler really did fire. The API really did return 200. The file really was written. Each check passes honestly while the only thing that matters fails.
Two shapes to watch for:
1. The safety net that adjusts instead of refusing. A guard catches a bad condition, then modifies the input so the operation can proceed — renaming a colliding key, truncating an over-long field, coercing a bad type. The error disappears from the logs and the bad condition ships anyway. A guard that never refuses is not a control; it is a laundering step, converting a real signal into a clean log line. Prefer failing closed: a skipped run is cheap and visible, a wrong run is expensive and invisible.
2. The record that drifts from the reality. Any check that compares against a cache, a state file, or a local ledger is only as good as that memory. When the memory can be truncated by a restart, a fresh machine, or a path that writes to one place and reads from another, the check degrades quietly and keeps returning "fine". Seed the memory from the artifacts themselves wherever you can, and periodically assert that the two still agree.
The practical defence is to add one check that reads the output rather than the exit code, and to make it something a human would actually notice — a count that should be stable, a uniqueness constraint, a spot comparison against what shipped last time. You are not trying to verify everything. You are trying to have at least one signal that fails when the job succeeds incorrectly.
Verify from logs, not config
Configuration tells you what somebody intended. Logs tell you what happened.
A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.
The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.
Each of those has a cheap, decisive check that costs seconds:
- Probe the dependency, do not read its credential. A key that exists proves nothing about the balance behind it.
- Grep for the action line, not the setup line. A startup banner proves the process started, not that it ever did its work.
- Compare timestamps after a deploy. If the running process is older than the file on disk, it is still executing the previous version from memory.
The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.
---
This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.
No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.