Distributed systems fail in ways single applications never do. A service that works perfectly in isolation can still bring down an entire platform the moment a downstream dependency slows down or drops a connection. I learned this the hard way while leading the design of a distributed BNPL platform that integrated fraud detection across multiple services, and again while building financial systems where a single unhandled timeout could mean a failed transaction for a real customer.
Resilience in microservices isn't a feature you bolt on later. It has to be designed in from the start, and in .NET, the tool I reach for most is Polly.
Why "it usually works" isn't good enough
In a monolith, a slow database call is annoying. In a microservices architecture, that same slow call can cascade. Service A waits on Service B, which is waiting on Service C, which is having a bad day. If nothing intervenes, threads pile up, connection pools exhaust, and the failure spreads to services that had nothing to do with the original problem.
The fix isn't "never fail." Failure is guaranteed at scale. The fix is designing so that failure stays contained.
Retries: useful, but not blindly
The instinct is to retry every failed call. That instinct is half right. Transient failures, like a brief network blip or a momentary spike in latency, genuinely benefit from a retry. But naive retries without backoff can make an already struggling service worse by hammering it with repeated requests.
var retryPolicy = Policy
.Handle<HttpRequestException>()
.WaitAndRetryAsync(3, attempt => TimeSpan.FromMilliseconds(200 * Math.Pow(2, attempt)));
Exponential backoff, ideally with a bit of jitter added, gives a struggling downstream service room to recover instead of getting buried under repeated retries from every caller at once.
Circuit breakers: knowing when to stop trying
Retries handle short blips. They don't help when a dependency is genuinely down. If a payment gateway is unreachable, retrying five times just delays the inevitable and ties up resources you need elsewhere.
This is where a circuit breaker earns its place:
var circuitBreakerPolicy = Policy
.Handle<HttpRequestException>()
.CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));
After a threshold of failures, the circuit "opens" and calls fail fast without even attempting the request, giving the downstream service breathing room. After the break duration, it moves to a half-open state to test whether the dependency has recovered. On the fraud detection integration for the BNPL platform, this pattern was the difference between a degraded but functioning checkout flow and a complete outage every time the detection service had a rough patch.
Timeouts: the policy everyone forgets
I've reviewed more production incidents caused by missing timeouts than by almost any other single issue. Without an explicit timeout, a call can hang indefinitely, holding a thread and a connection while your service quietly starves.
var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(5));
Every outbound call to another service should have an explicit timeout that reflects a realistic worst case, not the framework default.
Combining policies
Polly lets you wrap these together so retries, circuit breaking, and timeouts work as one coherent strategy instead of three uncoordinated pieces:
var resiliencePolicy = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy, timeoutPolicy);
The order matters. Timeout should sit closest to the actual call, with retry and circuit breaking layered around it, so a single slow call doesn't get retried into an even longer delay.
Fallbacks: degrade gracefully instead of failing loudly
Not every failure needs to be user-facing. When a recommendation service is down, showing a generic list instead of a personalized one is a far better experience than an error page. Polly's fallback policy lets you define exactly what happens when everything else has failed:
var fallbackPolicy = Policy<IEnumerable<Product>>
.Handle<Exception>()
.FallbackAsync(GetDefaultProducts());
On the loan management portal I built earlier in my career, this kind of graceful degradation kept core application and disbursement flows working even when a secondary verification service was struggling, instead of blocking the entire process.
Bulkheads: isolating failure domains
A single overloaded dependency shouldn't be able to consume every available thread in your application. Bulkhead isolation limits how many concurrent calls can be made to a particular resource, so a failure in one area doesn't starve unrelated parts of your system.
var bulkheadPolicy = Policy.BulkheadAsync(10, 20);
This one is easy to skip because its value is invisible until the day it saves you. I only started using it consistently after watching one slow third-party integration quietly consume the thread pool for an entire service.
The real lesson
None of these patterns are exotic. They're well documented and Polly makes them straightforward to implement. The hard part isn't the code, it's the discipline to apply them consistently before an incident forces the conversation. Every outbound call in a distributed system is a place where something can go wrong. Treat it that way from the start, and resilience stops being a firefighting exercise and becomes just how the system is built.
