AI OpsWIKI

When you hand work to something asynchronous — a queue, a webhook, a workflow tool, a background job — the response you get back means "I have received this". It does not mean "I have done this", and very often it does not even mean "I intend to do this".

This is the trap behind a large share of “the data just vanished” incidents. The sending side logs a success, the receiving side never processes anything, and both halves look healthy in isolation. A queue that accepts your message and never reads it looks exactly like one that works.

Defences, in order of strength:

1. Do not branch on the acknowledgement. If your fallback logic reads "if the handoff failed, do it myself", it will never run, because the handoff reports success. Make the local path unconditional and let idempotency absorb the duplicate.
2. Confirm from the other side. Check that the work actually completed — a status endpoint, a result record, a callback — rather than trusting the receipt.
3. Set a deadline. If the expected outcome has not appeared within N minutes, treat it as failed and act, rather than waiting forever.

An operation is idempotent if repeating it changes nothing beyond the first time. Setting a value to 5 is idempotent. Adding 5 is not.

This is the property that makes reliability affordable. Networks time out, retries fire, and redundant paths overlap — so in any real system some operations will happen more than once. If those operations are idempotent, that is a non-event. If they are not, your safety net becomes the thing that corrupts the data or spams the customer.

The usual implementation is a fingerprint: a hash of the inputs that identify the work. Before acting, check whether that fingerprint was already handled inside some window; if so, do nothing, and say so in the log.

The detail that separates a junior implementation from a senior one is what goes into the fingerprint. Too narrow and genuine repeat work gets swallowed; too wide and duplicates slip through. Hashing only “who” would silently discard a real follow-up message from the same person an hour later. Hashing “who plus what they said” collapses the duplicates while letting a genuine second message through.

When a record moves through stages — an order, a ticket, a deal, a deployment — the sequence usually carries meaning: later stages represent more progress. A monotonic state machine enforces that an update may move a record forward, never backwards.

Without that rule, a routine automated update can quietly destroy information. An order that goes from Shipped back to Processing has lost the fact that it shipped. Nobody notices, because no error was raised: the write succeeded perfectly.

The damage is worst when the stages encode who needs to act next. If “we contacted them” sits before “they replied to us”, then an automation that stamps “we contacted them” on every outgoing message will drag replied-to records out of the human's action queue — burying exactly the items that most needed attention.

Implementation is simple and worth doing every time:

• Keep the ordering in one list, not in scattered comparisons. Internal state identifiers frequently do not resemble their display labels, so comparing them by name is guesswork waiting to break.
• Compare positions before writing, and skip if the record is already at or past the target.
• Leave unrecognised states alone rather than guessing where they belong.

The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.

A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision — drop this message, skip this record, return an empty string — that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.

The defence is not “add more logging”. It is to make the healthy state provable, so that “nothing happened” can be distinguished from “nothing was supposed to happen”. Two things do that:

• Log the outcome, not the attempt. “sending notification” tells you nothing. “notification DELIVERED (id 4661)” versus “notification REJECTED 400” tells you everything.
• Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.

Every system has a critical path — the sequence of steps that must all succeed for the thing to work. A single point of failure is any step in that path with no alternative.

The trap is that these are usually invisible until they fire, because they hide behind something that has never failed before: a vendor account, a prepaid balance, one API key, one machine, one person who knows how the deploy works.

The lesson that generalises, and the one most teams get wrong: redundancy only counts if it is in the path. Having five interchangeable providers configured somewhere in your estate does nothing if the one place that actually makes the call can only reach one of them. Spare tyres in the garage do not help on the motorway.

Practical test: for each external dependency on your critical path, ask "if this returns an error for the next 72 hours, what does the user see?" If the answer is “nothing at all”, that dependency is a single point of failure, and the fallback belongs where the call is made — not elsewhere.

When the same rule, prompt, threshold or piece of logic exists in more than one place, the copies begin identical and end different. Nothing announces the divergence. Someone updates one copy, the others keep running the old behaviour, and the system's actual conduct is now split across versions that no single file describes.

The failure is especially nasty when a copy lives somewhere code review cannot see it: a hosted workflow builder, a dashboard setting, a scheduled job on one machine, a prompt pasted into a vendor interface. Those copies never appear in a diff, so the drift stays invisible until it produces a visibly wrong result in front of a customer.

Two defences that work:

• One definition, imported everywhere. Every consumer reads the same file. Where a copy must physically live elsewhere, generate and push it from that file rather than editing it by hand.
• Detect drift automatically. Re-read the remote copies on a schedule and raise an alert when one no longer matches the source. A copy you cannot diff is a copy you must monitor.

A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.

The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.

Each of those has a cheap, decisive check that costs seconds:

• Probe the dependency, do not read its credential. A key that exists proves nothing about the balance behind it.
• Grep for the action line, not the setup line. A startup banner proves the process started, not that it ever did its work.
• Compare timestamps after a deploy. If the running process is older than the file on disk, it is still executing the previous version from memory.

The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.