A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. October 27, 2026.
A job-search agent reads two mailboxes every hour so no employer reply is missed. Its way of reading marked every message as opened, so the owner never saw anything new, and Gmail learned from the fake opens and flagged 88% of her mail as Important.
What it looked like from outside
The owner reported that every message in both of her mailboxes — Gmail and the company mailbox — already showed as read when she opened them, and that Gmail had put its blue "Important" marker on almost everything. A read-only audit of 14 days on the live mailboxes confirmed it exactly: 425 of 425 inbox messages were marked read, while 166 of 166 messages in the spam folders were still unread. The only thing that had touched the inboxes and not the spam folders was the agent that watches for employer replies.
What was actually happening
The reply detector opened each inbox in normal read-write mode and downloaded every message in full. Over IMAP, that kind of fetch sets the "seen" flag on the server, so every hourly scan stamped every new message as opened within the hour. The day the agent started reading Gmail (23 August), Gmail's share of read mail jumped from 55–68% in the months before to 100%, and it has stayed there. Gmail's own importance model learns from which messages the user opens; from the fake opens it learned that she opens everything, and its "Important" share climbed from about a third of her mail before 23 August to 61%, then 79%, then 88% in the last two weeks. The same audit found three quieter gaps in the same agent: it remembered messages by their position in the mailbox, which shifts when anything is archived; it read only the 50 newest messages per pass; and it never looked in spam, where eight LinkedIn notices that a real person had messaged her were sitting.
The fix
Every mailbox and folder is now opened read-only and every message fetched with the side-effect-free form of the read, in the detector and in a second helper that had the same habit. Messages are remembered by their permanent ID instead of their position, the 50-message cap is gone, and both spam folders are scanned. The agent's memory file had also been loaded in a type no lookup could ever match, so after every restart it had forgotten what it had already judged; that was corrected too. The Important labels that the fake opens had produced were removed from the 1,320 messages received since 23 August, after their IDs were saved so the change can be undone; read state was left alone, because nobody can tell which of the old messages the owner had opened herself.
How I know it worked
First on a copy of the agent run against the live mailboxes, then on the first real hourly cycle after the deploy. The number of unread messages in all four folders was identical before and after the scan, including two Gmail messages that had arrived after the previous scan and stayed unread through it. The same cycle read both spam folders for the first time and surfaced one of the LinkedIn notices as a question for the owner. A new test fails the build if any code path opens a mailbox in read-write mode or fetches a message in a way that marks it read. A recount after the label clean-up found 0 of the 1,776 inbox messages since 23 August still carrying the Important label, with the unread counts unchanged.
The rule this earned
A watcher must not change what it watches. For anything that "only reads", prove it by measuring the state before and after it runs; if the numbers move, the read wrote. And when the state you touch is also someone else's training data, a side effect is a feedback loop.
The named concepts behind it
Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.
The read that writes
An operation everyone treats as "just looking" changes the thing it looks at, so the watcher quietly rewrites the state the owner depends on.
Most systems split operations into reads, which only look, and writes, which change something. Monitoring, auditing and dashboards are built on the promise that looking is free: you can watch as often as you like and nothing moves.
The failure is a read that is secretly a write. In email, the ordinary way to fetch a message over IMAP marks it as read on the server. A program that "only reads" a mailbox every hour therefore stamps every message as opened, and the person who owns the mailbox loses the one signal they rely on: what is new. The same shape appears as a web GET that increments a counter, a health check that consumes a queue item, or a report that resets a statistic it was meant to show.
It compounds when the state it rewrites is also training data. A service that learns from "what the user opened" — an inbox ranking messages as important, a feed ranking posts — reads those fake opens as real behaviour, and its predictions drift a little further every week.
The fix is to use the explicitly side-effect-free form of the read (in IMAP: open the folder read-only and fetch with PEEK), and to prove it the only way that counts: measure the state before and after the watcher runs. If the numbers moved, the read wrote.
Ordinary-life version: a building inspector told to "just check" every apartment, who flips every light switch on the way out. Nothing in the report is wrong, but the tenants come home to a house that no longer shows what they left.
Silent failure
The system did something reasonable, and told nobody.
The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.
A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.
The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:
- Log the outcome, not the attempt. "sending notification" tells you nothing. "notification DELIVERED (id 4661)" versus "notification REJECTED 400" tells you everything.
- Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.
Verify from logs, not config
Configuration tells you what somebody intended. Logs tell you what happened.
A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.
The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.
Each of those has a cheap, decisive check that costs seconds:
- Probe the dependency, do not read its credential. A key that exists proves nothing about the balance behind it.
- Grep for the action line, not the setup line. A startup banner proves the process started, not that it ever did its work.
- Compare timestamps after a deploy. If the running process is older than the file on disk, it is still executing the previous version from memory.
The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.
Fate-sharing
Two duties that should be independent ride the same code path, so when one has nothing to do, the other quietly stops too.
In network design, fate-sharing is a deliberate choice: David Clark's 1988 paper on the design of the Internet's protocols argued that the state of a connection should live with the endpoints that use it, so that the state is lost only when those endpoints themselves are lost. Two things that share fate fail together, and that is the point.
The bug is the unintended version: two duties that have nothing to do with each other end up sharing fate because they happen to live in the same function, the same job, or the same branch. The usual carrier is an early return. A loop does A, then B, then C; someone adds "if A found nothing, stop here" — correct for B, which depends on A — and C, which never depended on A, now runs only when A happens to be busy.
Ordinary-life version: a night guard is asked to sign for deliveries and, while at the door, to check the mailbox. One day the instruction becomes "if there is no delivery tonight, go back to your desk." Quiet nights get quieter. Nobody told the guard to stop checking the mail; it just stopped happening on every night without a parcel.
The usual shapes:
- The early return. "Nothing new — waiting for next run" ends the whole cycle, including the steps that had their own reason to run.
- The shared try block. One failing step aborts every step after it, so a broken scraper also silences the health report written at the end.
- The chained schedule. Job 2 is triggered by job 1's output; when job 1 legitimately produces nothing, job 2 never runs.
- The monitor on the patient. The thing that watches a system runs on that system, so the outage it should report also takes the reporter down.
It is a special case of silent failure: the skipped duty leaves no trace, and the line that is logged ("no new jobs") is true and calm. It also hides behind liveness is not correctness: the loop runs every hour, so every liveness signal is green, while one of its duties runs only some of the hours.
The defences. Put a duty that does not depend on a branch before the branch, or give it its own trigger. Treat every early return as a decision about everything after it, not only about the step that motivated it. And measure each duty's real frequency from its own log line (verify from logs, not config) — count "mailbox checked" per cycle, not "cycle ran".
---
This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.
No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.