AIdeazz Blog About Portfolio

The queue marked 175 items done before anyone read them, and logged it as success

· by

A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. August 30, 2026.

A newly added data source ran on schedule for a full day, logged a healthy count every hour, and delivered nothing at all downstream. The dedup ledger was stamped before the processing cap was applied, so everything past the cap was recorded as already handled without ever being handled.

What it looked like from outside

A curated source had been wired into an autonomous discovery pipeline the previous day and appeared healthy on every measure taken. It fetched on schedule, its hourly log line reported a stable count of live items, a hygiene filter reported dropping expired ones, and the overall relevance-gate pass rate rose after it was added. Nothing errored, no timeouts, no retries. The failure surfaced only when a human asked why one specific high-value item that was demonstrably present in the source had never appeared in the CRM at the far end of the pipeline.

What was actually happening

Three independent defects, each sufficient on its own to hide the item. First and most serious: the discovery stage wrote every gate-passing item into the persistent seen ledger, saved it, logged a success count, and only then truncated the returned list to a processing cap. Two consecutive lines read "295 NEW accepted" and "Found 120 new" -- the 175 in between were permanently recorded as already seen without anyone having looked at them, and a 21-day time-to-live on that ledger buried them past the expiry of the items themselves. Second: the ordering that was supposed to protect high-value sources sorted them into a binary group rather than ranking them, and a stable sort preserves insertion order inside a group, so the newest and densest source was appended last and roughly 888 items from other sources consumed the cap before it was reached. Measured afterwards, 279 of its records sat in the ledger with a status of merely seen and not one had ever reached the processing stage. Third: the model-based relevance judge carried a character description asserting the candidate did not write code by hand, so it vetoed roles for requiring the two languages the operator ships production systems in daily, contradicting its own criteria which listed several such titles as approved. A fourth, upstream: the source republished an empty eligibility field from the origin ATS as a hard single-country restriction, so a globally-open role was correctly rejected as geographically ineligible on incorrect input.

The fix

Made the ledger write conditional on the work actually being handed on -- the loop now breaks at the cap before touching anything it cannot process, marks seen only what it returns, and logs the remainder explicitly as deferred rather than dropped, so the next cycle reconsiders them immediately instead of in three weeks. Replaced the binary priority group with round-robin interleaving across sources, richest first within each round, which fixes the class rather than the instance: no source can crowd out another however much volume it brings, so adding a large new source can never again silently starve an existing one. Corrected the judge's character description to the true constraint -- production code daily, but no degree-gated or algorithm-screen hiring -- leaving the same categories filtered out with the false premise removed. Added an origin check to the source that re-verifies single-country tags against the employer's own record and only ever relaxes, on the employer's explicit declaration, failing soft to the original value. Released the burned ledger entries under a guard that touched only records still at status seen, leaving anything already acted on untouched.

How I know it worked

Before the fix the ledger showed 279 records from the new source with a status breakdown of seen for every one and zero at any later status, which is the outcome check that should have been run on day one. After deploy, one cycle showed 120 accepted and marked seen against the cap and 686 gate-passing items explicitly left unseen for the next cycle, where previously those would have been burned; 88 of the 120 items actually processed came from the source that had contributed nothing for its entire life. The specific item that prompted the investigation now clears both the relevance gate and the judge, with the judge citing the correct reasons. The judge was regression-tested at eight of eight on a fixture set spanning three role types that must pass and five that must be rejected, including a genuinely geography-restricted role which is still correctly rejected. The origin check corrected two of 103 re-verified records, confirming the source is right most of the time and the correction is conservative. The evaluation harness ran 136 passing and one failing, identical to before the change, that single failure being an unrelated provider whose credits are exhausted and which the harness is correctly reporting.

The rule this earned

A queue must not acknowledge work it has not done. If the ledger write happens before the capacity limit, everything above the limit is recorded as handled and disappears without an error -- so mark complete only what actually completed, and log the remainder as deferred so the difference stays visible. Priority expressed as membership is not priority: a stable sort preserves insertion order within a group, so the item you most wanted first ends up wherever it happened to be appended. And when you add a component, verify what came out of the far end, not that the component ran. A source that logs a healthy count every hour and produces nothing is indistinguishable from a working one until someone asks what it actually delivered.

The named concepts behind it

Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.

Acknowledgement is not completion

A receipt proves delivery. It never proves processing.

When you hand work to something asynchronous -- a queue, a webhook, a workflow tool, a background job -- the response you get back means "I have received this". It does not mean "I have done this", and very often it does not even mean "I intend to do this".

This is the trap behind a large share of "the data just vanished" incidents. The sending side logs a success, the receiving side never processes anything, and both halves look healthy in isolation. A queue that accepts your message and never reads it looks exactly like one that works.

Defences, in order of strength:

1. Do not branch on the acknowledgement. If your fallback logic reads "if the handoff failed, do it myself", it will never run, because the handoff reports success. Make the local path unconditional and let idempotency absorb the duplicate.
2. Confirm from the other side. Check that the work actually completed -- a status endpoint, a result record, a callback -- rather than trusting the receipt.
3. Set a deadline. If the expected outcome has not appeared within N minutes, treat it as failed and act, rather than waiting forever.

Liveness is not correctness

A dead job announces itself. A job that runs perfectly and emits slightly wrong output never will.

Almost every check you own measures liveness: did it run, did it return, did it exit zero, did it publish. Almost none measures correctness: was the thing it produced the right thing. These are different properties, and the gap between them is where the expensive incidents live.

The asymmetry is what makes this dangerous. A job that stops firing is loud — the output is missing, someone notices within a day. A job that fires on schedule and produces output that is subtly wrong is silent, and it stays silent for as long as nobody reads the output, because every signal you have is reporting the truth. The scheduler really did fire. The API really did return 200. The file really was written. Each check passes honestly while the only thing that matters fails.

Two shapes to watch for:

1. The safety net that adjusts instead of refusing. A guard catches a bad condition, then modifies the input so the operation can proceed — renaming a colliding key, truncating an over-long field, coercing a bad type. The error disappears from the logs and the bad condition ships anyway. A guard that never refuses is not a control; it is a laundering step, converting a real signal into a clean log line. Prefer failing closed: a skipped run is cheap and visible, a wrong run is expensive and invisible.
2. The record that drifts from the reality. Any check that compares against a cache, a state file, or a local ledger is only as good as that memory. When the memory can be truncated by a restart, a fresh machine, or a path that writes to one place and reads from another, the check degrades quietly and keeps returning "fine". Seed the memory from the artifacts themselves wherever you can, and periodically assert that the two still agree.

The practical defence is to add one check that reads the output rather than the exit code, and to make it something a human would actually notice — a count that should be stable, a uniqueness constraint, a spot comparison against what shipped last time. You are not trying to verify everything. You are trying to have at least one signal that fails when the job succeeds incorrectly.

Silent failure

The system did something reasonable, and told nobody.

The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.

A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.

The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:

Verify from logs, not config

Configuration tells you what somebody intended. Logs tell you what happened.

A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.

The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.

Each of those has a cheap, decisive check that costs seconds:

The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.

---

This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.

No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.