A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. September 7, 2026.
One access token had two homes — a credential store used by version control, and an environment file read by seven application modules. Rotating it updated one. Version control kept working perfectly, so nothing looked wrong, while every API call from the same machine returned Bad credentials.
What it looked like from outside
An operations agent on a single virtual machine publishes a daily article to several destinations. On the day of the incident the article was written, cross-posted to a developer community, scheduled to social channels, and announced with a link to the operator's own site. The link returned a content-addressed storage error — no such page. Nothing in the run had reported a failure. In the same window, a commit-review bot stopped posting reviews and a media agent stopped reading its metadata, both without an operator-visible error. Version control on that same machine was completely healthy throughout: pushes, pulls and remote listings all succeeded against all seven repositories, which is exactly why the fault was invisible for as long as it was.
What was actually happening
The access token existed in two places, and only one was rotated. Version control authenticated from a credential store on disk; seven compiled modules read the same token from an environment file for direct API calls. Earlier the same day the token had been consolidated to a single location for VERSION CONTROL, and that work was verified thoroughly — every repository was proven to authenticate. The verification never covered the application layer, because the application layer does not use version control to reach the API. When the operator regenerated the token, the credential store received the new value and the environment file kept the old one. Two further mechanisms hid the split. The credential store's helper ERASES an entry the server rejects, so the moment version control tried the dead value it purged the line and re-read the live one, leaving no trace and a zero-byte file that looked like corruption rather than self-cleaning. And the modules read the environment their process manager handed them at start-up, so the stale value was pinned in memory and would not have refreshed even if the file had been corrected.
The fix
Rotation is now a single action that writes BOTH destinations — the credential store and the environment file — backs up the latter first, and then schedules a detached restart of the long-running process so the modules pick the new value up. The restart is detached and delayed on purpose: the rotation helper runs as a child of the process it must restart, so restarting inline would kill the parent before it could confirm the result to the operator. A second entry point re-copies the existing token from the store into the environment file without rotating anything, so a drift discovered later does not force a needless regeneration. The daily expiry watch that had been added hours earlier was kept unchanged, and it is what surfaced the dead token in the first place.
How I know it worked
From logs and live probes, not from configuration. Before the fix the process log carried "Error processing push - Bad credentials" with HTTP 401 against the repository comparison endpoint and against the media agent's metadata reads. After the fix, the token in the environment file returned HTTP 200 from the identity endpoint, and a twenty-second watch of the error log recorded zero new Bad-credentials entries where there had been eight. The missing article was restored through the same publishing channel the daily run uses, and the page reappeared in version control as a single regeneration commit. The expiry watch itself was tested four ways rather than assumed: under a stripped scheduler environment, against a healthy token, against a rejected delivery, and against a dead token — because an alarm that reports success on the strength of the sending command exiting zero is the failure it exists to catch.
The rule this earned
A secret with more than one home has more than one expiry, and rotation is only atomic if a single action updates every copy. The specific trap is that the copies are rarely equal in loudness. Here version control ran constantly and visibly, so its health was ambient reassurance, while the API callers failed into a log nobody reads. When you consolidate a credential, enumerate its CONSUMERS rather than its locations — grep for the variable and for the vendor's endpoint, not for the store you already know about — and verify one call per consumer class, because proving that version control works proves nothing about a module that never touches it. Two corollaries earned here: a credential store that erases a rejected entry will make a rotation failure look like file corruption, so read the emptiness as self-cleaning rather than damage; and a process that reads its environment at start-up needs a restart, which means writing the file is only half the fix.
The named concepts behind it
Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.
Single source of truth
Copy logic instead of calling it, and you have scheduled a bug for a date nobody will tell you about.
When the same rule, prompt, threshold or piece of logic exists in more than one place, the copies begin identical and end different. Nothing announces the divergence. Someone updates one copy, the others keep running the old behaviour, and the system's actual conduct is now split across versions that no single file describes.
The failure is especially nasty when a copy lives somewhere code review cannot see it: a hosted workflow builder, a dashboard setting, a scheduled job on one machine, a prompt pasted into a vendor interface. Those copies never appear in a diff, so the drift stays invisible until it produces a visibly wrong result in front of a customer.
Two defences that work:
- One definition, imported everywhere. Every consumer reads the same file. Where a copy must physically live elsewhere, generate and push it from that file rather than editing it by hand.
- Detect drift automatically. Re-read the remote copies on a schedule and raise an alert when one no longer matches the source. A copy you cannot diff is a copy you must monitor.
Silent failure
The system did something reasonable, and told nobody.
The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.
A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.
The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:
- Log the outcome, not the attempt. "sending notification" tells you nothing. "notification DELIVERED (id 4661)" versus "notification REJECTED 400" tells you everything.
- Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.
Verify from logs, not config
Configuration tells you what somebody intended. Logs tell you what happened.
A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.
The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.
Each of those has a cheap, decisive check that costs seconds:
- Probe the dependency, do not read its credential. A key that exists proves nothing about the balance behind it.
- Grep for the action line, not the setup line. A startup banner proves the process started, not that it ever did its work.
- Compare timestamps after a deploy. If the running process is older than the file on disk, it is still executing the previous version from memory.
The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.
---
This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.
No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.