Yesterday morning fifty-nine of our automated workflows stopped producing output at the same time. Nothing had been deployed. No server had gone down. On the diagram those workflows are separate systems, owned by different people, serving different parts of the business.
They shared one thing. A single language model API key, bound into 143 nodes across all of them, and the account behind it had run out of balance.
That is the failure mode I now watch for, and it is not really an integration problem. It is a counting problem. An architecture diagram shows you how many systems you have. It does not show you how few distinct things they actually depend on, and that second number is the one that decides how bad your worst day is.
Uptime Institute's Annual Outage Analysis for 2025 makes the same point at industry scale. Across nine years of tracking publicly reported outages, third-party IT and data center providers account for roughly two thirds of them. Outages caused by IT and networking issues rose to 23 percent of impactful outages in 2024, which the analysis links to growing complexity and the change management and misconfiguration problems that come with it. The systems are not fragile. The concentration underneath them is.
Seven of these have caught me. Here they are, ordered by what each one cost us.
1. One credential sitting behind dozens of unrelated systems
That one took twenty minutes to diagnose, and only because I had seen it before. The first time, I spent a day hunting for a code fault that did not exist.
Credentials spread by convenience. Somebody builds a workflow, picks the credential that already exists, and ships. Nobody decides to put 143 nodes on one key. It accumulates, one reasonable choice at a time.
The check is a single query, and most platforms will answer it: how many running systems reference each credential you hold? Anything above about ten deserves a second credential and a deliberate split. Not because the key will be revoked. Because billing limits and provider policy changes both arrive at the level of the key.
2. A vendor's free tier quietly promoted to infrastructure
We run a scraping layer that once depended on free tier keys. It worked, so it grew. Eventually 49 workflows sat on a rotating pool of those keys, with a lock so that two jobs never drew on the same one at once.
That design is fine as an experiment. As infrastructure it has a property nobody wrote down: the provider can change the free tier whenever it likes, and the failure will look like a bug in your code rather than a commercial decision made somewhere else.
Free and cheap tiers are not the problem. Not knowing which of them your revenue now depends on is the problem. Every quarter I list the external services in the stack and mark which ones are on a paid contract and which are on somebody's goodwill.
3. Shared disk that nobody is metering
An automation server of ours became unresponsive and stayed that way. The cause was not a workflow. Execution logs had grown to 25 gigabytes and filled the volume, and once the volume was full the database stopped accepting writes, which stopped everything.
Nothing was integrated wrongly. Every component behaved as designed. The shared resource underneath them had no owner and no alarm.
Disk, database connections, memory on a shared host, API quota pooled across an organization. These are integration dependencies even though they appear in no integration. If two systems can starve each other of something, they are coupled, whatever the diagram says.
4. A management plane that reaches everything you own
We run more than a hundred web properties. For years the operational advantage was that one management layer could push an update to all of them, which is exactly the efficiency you want at that scale.
Then we found an administrator account on the network that nobody on my team had created. Because the management layer touched every property, we had to treat all of them as compromised rather than the one we found it on. The blast radius was defined by our convenience, not by the intrusion.
The uncomfortable question for any technology leader with a control plane: if credentials to it leaked tonight, how many systems would you have to assume are affected? If the answer is all of them, the plane needs segmentation, separate credentials per tier, and a real audit trail before it needs any new features.
5. Schedules that all fire at the same moment
This one is almost funny until it is not. We set staggered schedules across a set of jobs so they would spread their load through the day.
They all ran within the same four-hour window instead. The scheduler was interpreting times in a different timezone than the one we had reasoned in, so every carefully separated slot collapsed together and the whole set hit shared rate limits at once.
Correlated timing is a dependency. Two jobs that never talk to each other are still coupled if they always run together and draw on the same quota. When you stagger something, verify the actual fire times from the execution history rather than from the configuration you intended.
6. Assets that expire on a calendar nobody owns
We keep a large outbound email setup. During an audit we found two sending domains had already expired, while every dashboard above them showed their mailboxes as healthy. The mailboxes were healthy. The thing they stood on was gone.
Domains, TLS certificates, API tokens with expiry dates, developer program memberships, business verifications. All of them fail on a date rather than under load, which means load testing will never find them and monitoring usually will not either, because the component being monitored is still up.
Every expiring asset needs a named owner and a renewal date in a place a human reads. It is the least interesting item here. It has also caused more of our silent outages than anything else.
7. The copy and the original living on the same host
While recovering a database, we brought up a second instance pointed at the same data directory as the first. Two processes. One set of files underneath. The corruption that followed was ours, not the platform's.
The general shape is broader than that mistake. Backups on the same disk. A staging environment sharing a production database. A failover that reads its configuration from the system it replaces.
Anything whose job is to survive the failure of something else must not share substance with it. So when you review a recovery plan, trace the dependencies of the recovery path itself. Would each one survive the scenario you are recovering from?
Conclusion
None of these were exotic. Each one came from a decision that was locally sensible and never revisited once the system it belonged to grew up.
The exercise I would recommend takes an afternoon and no tooling. List the systems you run. For every credential, host, schedule, quota, control plane and expiring asset, write down how many of those systems stop if that single item stops. Sort the list by that number.
The top of that list is your real risk register, and it will not look much like your architecture diagram. Give every item near the top an owner. Concentration you have counted and assigned is a manageable engineering tradeoff. Concentration nobody has counted is a bad day waiting on a calendar.