In Q4 2025 we shipped an automated content pipeline for a client that ran cleanly for 7 weeks and then began producing subtly wrong output for 9 days before anyone noticed. Nothing errored. No alert fired. Our monitoring reported 100 percent success throughout, because every request returned a 200 and every job completed.
The vendor had changed a default parameter. Our integration had never pinned it, and our tests had never checked the one thing that changed.
What the system did
The pipeline took long-form source material, called the Anthropic API to produce structured summaries, validated the output against a schema, and wrote the results into the client's CMS. It was not complicated. It ran on a schedule, processed a few hundred items a week, and had been through a normal review before launch. Job state lived in Supabase and failures were meant to surface in Slack.
Our monitoring was what most teams would build. We alerted on error rate, on job duration, and on throughput. We had a schema validation step that rejected malformed output. All three were green for the entire incident, and all three were telling the truth about what they measured.
What actually changed
The provider adjusted a default on their side, which is a documented and entirely reasonable thing for a provider to do. The effect on our output was a shift in how the model handled ambiguous source passages. Previously it had omitted them. Now it inferred.
Every inference was plausible. Every output still validated against our schema, because the schema checked structure and the problem was content. A summary that confidently states something the source document does not say is structurally identical to a correct one, and no amount of validation on shape will separate them.
Over 9 days roughly 40 items went through the pipeline carrying at least one inferred claim. The client caught it, not us, when a subject-matter expert reading a published summary said that a particular detail was not in the original.
We then audited the full window. Across 6 weeks of output, over 9 days of affected runs, 40 of 118 items carried at least one claim we could not locate in the source. The other 78 were clean, which is part of why nobody spotted it earlier: the failure was intermittent by nature, because it only appeared where a source passage was ambiguous enough to invite an inference.
What we did not understand at the time
We had built monitoring for a system that fails loudly, and integrated a component that fails quietly.
Traditional integrations break in ways your instruments are designed to catch. A schema changes and parsing throws. An endpoint moves and you get a 404. A rate limit hits and you get a 429. Model integrations have all of those failure modes plus one more that none of the standard tooling addresses, which is that the output stays well-formed while its relationship to the input changes.
The bug was that our entire verification strategy tested the contract between our code and the API, and the thing that mattered was the relationship between the input document and the output text. We had no assertion anywhere in the system that the output was grounded in the source.
In retrospect the warning was in our own review notes from launch. Somebody had written down that we should spot-check outputs weekly. We had done it for 3 weeks, found nothing, and stopped, because everything was green and the check felt redundant. A manual check that only ever confirms what the automated monitoring says will always feel redundant, right up to the moment it is the only thing that would have worked.
What we changed
Three changes, and the first is the one that generalises.
First, we added a grounding assertion. For a sample of every batch, the pipeline now checks that key factual spans in the output appear in, or are directly derivable from, the source. It is imperfect, it produces some false positives, and it would have caught this incident on day 1 instead of day 9.
Second, we pin every parameter we depend on, explicitly, including the ones that currently match the default. Relying on a default is relying on a decision somebody else can change without telling you, and the cost of writing it down is one line. We also record the model version in the output metadata, so a change in behaviour can be correlated with a change in what produced it.
Third, we treat any provider changelog entry as a trigger for a verification run rather than as reading material. Over 5 months that has fired 4 times and found nothing 3 of those times, which is the correct ratio for a check like this.
What I would tell another technology leader
The thing I would push hardest on is that this class of integration needs a semantic assertion, not just an operational one, and almost nobody builds one because the operational monitoring is so much easier and looks so much like coverage.
Ask this about any component you have integrated: if the output became subtly wrong while staying well-formed, which alert fires? For a large number of teams shipping model-based features right now, the honest answer is none, and the second honest answer is that a customer tells you. We had reached that state deliberately in the sense that we had built the monitoring on purpose, and accidentally in the sense that nobody had asked that question out loud.
The second thing is smaller and worth saying. Do not let a manual check die because it keeps finding nothing. A check that never fires is either unnecessary or it is the only instrument you have pointed at a failure mode your automation cannot see, and those two look identical from the outside until one day they do not. We stopped ours after 3 clean weeks and the incident arrived in week 7.
The pipeline still runs. It has processed considerably more since the fix than before it, and the grounding assertion has fired twice on genuine drift, both times within a day. Our broader approach to shipping AI into live work is in our founder-led content piece.