Software resilience: what happens to your process when a vendor goes down?

Almost no modern system works alone. A typical business application depends on a payment API, an authentication provider, a messaging or email service, maps or data-lookup APIs, third-party AI models, a cloud platform (AWS, Google Cloud or Azure) and, in front of everything, edge services like Cloudflare, which carries a huge share of the internet's traffic.

Each of these pieces exists for a good reason: it is faster and cheaper to use specialists than to build everything in-house. The side effect is that your business process now depends on a chain of vendors you do not control, and the recent major outages of cloud providers and of Cloudflare itself, which took everything from banks to social networks offline for hours, showed how long that chain is.

The myth of perfect operation

The problem is rarely in using external vendors. It is in how processes are designed around them.

In most companies, the workflow assumes perfect operation: the payment always approves in seconds, the external lookup always responds, the login always authenticates, the cloud is never unavailable. Screens, automations and even the team's procedures are built on that premise, as if every dependency were immutable and immune to error.

While everything works, the premise is invisible. It only shows up the day it stops being true.

In real life, the chain breaks at the link you are not watching

When one of those dependencies fails, the process that assumed perfection simply stops. And a stopped process has a different price depending on the area: the e-commerce stops selling at peak traffic, logistics cannot issue documents and trucks do not leave, support loses access to customer history, finance misses settlement windows and accumulates SLA penalties. In sectors like healthcare and financial markets, the bill is even more serious, involving regulatory risk and, at the limit, risk to people.

The uncomfortable detail: in most of these scenarios, your system is technically up. It was a vendor that went down. For the customer, the distinction does not exist, the one who failed was you.

A well-designed system assumes failure will happen

That is what software resilience means: designing the system knowing that every external dependency will, at some point, slow down, respond incorrectly or go offline. It is not pessimism, it is statistics: with dozens of vendors in the chain, the probability of all of them working perfectly all the time is practically zero.

In engineering, that translates into a set of known practices: timeouts and retries with progressive backoff, so the flow does not freeze and a struggling service does not get bombarded; circuit breakers, which isolate the sick dependency before it takes down the rest; queues, which absorb work when the destination is unavailable and process it later; caching the last valid data, which keeps the operation running on slightly stale information; idempotency, which guarantees a repeated attempt does not generate a duplicate charge or order; and degraded mode, in which the system gives up the accessory to preserve the essential, like a checkout that switches off product recommendations but keeps selling.

For truly critical dependencies, real redundancy comes in: a second payment provider ready to take over, an alternative messaging route, a documented manual procedure for the team to follow while the technology recovers.

Not everything deserves redundancy, and that is where the process comes in

Redundancy costs money, complexity and maintenance. Duplicating everything is as wrong as duplicating nothing. The right question is not technical, it is a business question, and our colleagues at Dink summarized it well in the article that inspired this post: if this service goes down for six hours, what will your customer experience?

Answering that question requires understanding the process as a whole, end to end, not system by system. That map is what reveals which dependencies are a tolerable nuisance and which are a business vulnerability, where a cache solves it, where a backup vendor is needed and where a well-trained manual procedure is the best contingency. Without the process map, the company spends redundancy where it is not needed and discovers the fatal link during the incident.

It is the same principle we apply to maintaining critical systems: knowing the whole operation before touching the technology. And it is the difference between a system that works in the demo and one that survives the real world, a distance we explored in the myth of the 90% done project.

Where to start

An honest exercise that fits in one afternoon: list the ten most important external dependencies of your operation, and for each one answer what the customer experiences if it goes down for six hours, what the team does in that scenario and how much each stopped hour costs. If any answer is "we don't know", that is the place to start.

At Espresso Labs, this mapping is part of the diagnosis we run for systems in production, together with our colleagues at Dink, the Belgian software house and Espresso shareholder that serves critical operations in Europe. If your operation depends on systems that cannot stop, bring us the challenge.