A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. September 2, 2026.
"Move this card" created a card called "Move this card". Three independent defects stacked — an ASCII word boundary that could never match Cyrillic, a hand-rolled call that bypassed the five-provider chain, and an empty catch that discarded the API's exact explanation.
What it looked like from outside
A voice assistant turns spoken notes into task-board cards, and also understands management commands — move this card, archive those. The create path worked. Every management command produced a new card instead: "Move this task to the September board" created a card titled "Move task to September board"; "Archive this card" created a card titled "Archive this card". Nothing errored, nothing was logged as a failure, and each reply was a cheerful confirmation that a card had been created. The operator hit it three times in a row, in two languages, before reporting it. Because the create path was healthy and the replies were success messages, the system looked like it was working and merely misunderstanding.
What was actually happening
Three defects in series, each individually silent, and each one hiding the next. FIRST — the gate deciding "is this a management command" tested a regex of the form \b(move|archive|...|Cyrillic verbs...)\b. JavaScript's \b is defined on ASCII word characters, so a Cyrillic letter is not a word character and a boundary can never occur beside one. Every Russian command had been unmatchable since the feature shipped, including the exact example printed in the assistant's own help text — the documentation advertised a capability whose regex could not fire. SECOND — English commands passed the gate and reached a classifier that POSTed the LLM vendor's API directly instead of using the project's five-provider fallback chain. That vendor's balance had been at zero for two weeks. The chain routed everything else around it flawlessly; this one hand-rolled call could not be routed because it never entered the chain. It returned an empty action list, and the caller treated "no actions" as "not a management command" and fell through to create. THIRD — once those were fixed and a real move was finally attempted, it failed with the message "undefined". The move helper sent only the destination list id; the API requires the destination board id as well whenever the list is on a different board, which is the common case. The API said so precisely on the first call. The caller had wrapped it in an empty catch, so the exact explanation was discarded and replaced with nothing.
The fix
Replaced the ASCII word boundary with Unicode lookarounds under the u flag, with stems taking a letter-class suffix so inflected forms match in both languages while "remove" still does not match "move" — verified against a case set covering both languages and the near-miss. Routed the classifier through the five-provider chain and made its failure path log loudly rather than returning an empty list, because falling through to create is a real behaviour change the operator must be able to see. Added the destination board id to the move call, and replaced both empty catches with collected reasons surfaced in the reply. Separately, the create path was extended to honour a board and column named out loud: its routing fields were fixed enumerations that could not represent a specific board name, so an explicit instruction was being collapsed into whichever enumeration the topic suggested — a credit-card task filed itself under finance despite naming a different board. While testing that, a fourth instance of the same bypass surfaced: the create classifier ran on a two-provider pair rather than the chain, and when its remaining provider's reply did not parse it degraded to a default that discarded every hint, reporting confidence 0.3 with the reason "parsing failed" while looking exactly like a routing bug.
How I know it worked
Each layer proven separately rather than by trying the feature again. The word boundary was demonstrated to be impossible before it was fixed: a bare test of the pattern against the Cyrillic verb returned false while the same pattern without boundaries returned true, and the replacement then scored 6/6 across both languages plus the "remove"/"move" near-miss. The classifier fix was confirmed by log lines naming which provider answered — the previously dead vendor was skipped and a different one responded — with the command classified as move and archive rather than create. The move fix was proven in both directions against a real record: sending only the list id returned HTTP 404 with the message "list does not exist on board", and adding the board id returned HTTP 200 with the record on its new board. The explicit-routing fix was verified end to end with a temporary record that was deleted afterwards, landing on the named board and the named column — matching that column despite a stray punctuation character in its real name — with confidence 1 instead of 0.3. Final confirmation came from the operator's own next-morning transcript: one spoken note created a task on the board and column she named, and a second moved it to a different column, with no manual editing.
The rule this earned
Three rules, one per silence. A regex word boundary is ASCII-only in most engines — never use it to gate non-Latin input, and test a matcher against a string you know must match before trusting it to route anything. A fallback chain protects only the calls that route through it, so audit for direct vendor calls rather than assuming the wrapper is universal; every hand-rolled call is a second, unprotected system wearing the first one's reputation. And never write an empty catch around a network call: the API's own error message is usually a complete diagnosis, and discarding it converts a thirty-second fix into a multi-session investigation. The cost here was entirely in the discarded message, not in any of the defects.
The named concepts behind it
Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.
Resilience is opt-in
A fallback chain protects only the calls that route through it. Every hand-rolled call is a second, unprotected system wearing the first one's reputation.
You build a provider chain: five vendors, ordered by cost and quality, each failure falling through to the next. You test it. A vendor goes dark and the system keeps answering. The claim "we survive a provider outage" is now true, demonstrated, and written on the architecture diagram.
Then, somewhere in the codebase, one function calls the vendor directly. Not maliciously — it was written before the chain existed, or in a hurry, or by someone who only needed one quick classification and reached for the SDK. It works perfectly. It goes on working perfectly for months.
The day the primary vendor's balance hits zero, the chain routes around it exactly as designed, and that one function returns nothing.
The failure is invisible in a specific and dangerous way. The system is not down — most of it demonstrably still works, which is the strongest possible argument that the outage is not your problem. The broken path usually has a fallback of its own: an empty array, a default value, a "not classified" branch. So it does not error. It quietly does the other thing, and the other thing is often plausible enough to look like a product decision rather than a defect.
Three properties make this worth naming as its own failure mode:
- The bypass is invisible from the resilient side. Nothing in the chain's code, tests or metrics can see a call that never enters it. Coverage of the chain tells you nothing about coverage of the system.
- It survives exactly as long as the primary works. Which means it is introduced, reviewed, tested and shipped without ever being wrong. There is no moment where the mistake is observable — until the outage.
- Its blast radius is the opposite of its footprint. One function, five lines. The behaviour it silently disables can be an entire product surface.
The defences are unglamorous and cheap:
- Grep for the vendor, not for the wrapper. The audit question is "what calls
api.vendor.comor imports the SDK?", not "does everything use our chain?" One of those has an answer. - Make the wrapper the only thing holding the credential. A function that cannot reach the key cannot bypass the chain.
- Never let a classifier fail into a default. Returning
[]on error is the mechanism that converts an outage into a silent behaviour change. Fail loudly, or fail into a state the operator can see. - Name the responder. If every routed call logs which provider answered, a bypass is visible as an absence — the one path that never names anybody.
The rule this earns: resilience is a property of calls, not of systems. Audit the call sites, because the chain cannot audit them for you.
Silent failure
The system did something reasonable, and told nobody.
The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.
A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.
The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:
- Log the outcome, not the attempt. "sending notification" tells you nothing. "notification DELIVERED (id 4661)" versus "notification REJECTED 400" tells you everything.
- Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.
Verify from logs, not config
Configuration tells you what somebody intended. Logs tell you what happened.
A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.
The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.
Each of those has a cheap, decisive check that costs seconds:
- Probe the dependency, do not read its credential. A key that exists proves nothing about the balance behind it.
- Grep for the action line, not the setup line. A startup banner proves the process started, not that it ever did its work.
- Compare timestamps after a deploy. If the running process is older than the file on disk, it is still executing the previous version from memory.
The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.
---
This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.
No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.