The "Hope" Strategy

Most developers write resilience policies and then "hope" they work. They might write a unit test that mocks a dependency to throw an exception, but that usually tests the exception handler, not the resilience policy.

If you wrap a call in a retry policy, you don't just want to know that the app doesn't crash. You want to know:

• Does the classifier actually see this specific error as Transient?
• Does the retry budget bound the amplification, or are you about to DDoS your own database?
• Does the circuit breaker trip exactly when it should, and does it probe for recovery?

To answer these, you need to break things on purpose, and you need to do it in a way that exercises the entire pipeline.

Through-the-Pipeline

The core philosophy of the NResilience Chaos tool is that faults must traverse the real pipeline.

If you inject a fault outside your resilience call, you are bypassing the very logic you're trying to test. You're testing the "outer" world. To test the policy, the fault must be injected inside the callback that the policy executes.

When a fault is injected inside, it is treated exactly like a real failure:

1. It is passed to the Classifier.
2. The resulting Verdict determines if it retries.
3. It is counted against the Circuit Breaker.
4. It consumes the Retry Budget.

How to Inject Chaos

Wrapping Callbacks

For general async work, you can wrap your callback using the Inject extension method. This replaces your normal work with a version that occasionally fails or slows down based on a Chaos profile.

using NResilience;
using NResilience.Testing;

// 1. Define the chaos profile
var chaos = new Chaos 
{ 
    Enabled = true, 
    FaultRate = 0.1, // 10% of calls fail
    Latency = TimeSpan.FromMilliseconds(500), 
    LatencyRate = 0.2 // 20% of calls are slowed
};

// 2. Wrap your actual work
Func<CancellationToken, Task<string>> work = async ct => 
{
    return await _service.GetDataAsync(ct);
};

var chaoticWork = chaos.Inject(work);

// 3. Run it through your policy
var result = await policy.RunAsync(chaoticWork, ct);

Chaos in the HTTP Pipeline

If you're using HttpClient, the ChaosHandler lets you inject faults into the DelegatingHandler chain. The key is to add it after AddResilience(). In .NET's handler chain, "after" means "inner to," so the fault is injected before the request leaves the app but after the resilience handler has wrapped it.

services.AddHttpClient<OrdersClient>()
    .AddResilience() // Outer: handles the retries/breakers
    .AddHttpMessageHandler(() => new ChaosHandler(chaos)); // Inner: injects the fault

Testing Outcomes vs. Exceptions

Real failures aren't always exceptions. Sometimes a dependency returns a 503 Service Unavailable or a 429 Too Many Requests. If your classifier is tuned to retry these specific status codes, testing with throw new Exception() is useless.

The Chaos tool allows you to inject outcomes - specific return values that the classifier must judge.

// Inject a 503 response instead of throwing an exception
var chaosHandler = new ChaosHandler(chaos, response: () => 
{
    return new HttpResponseMessage(HttpStatusCode.ServiceUnavailable);
});

Game Days and Production Safety

You might be hesitant to leave testing code in production. However, Chaos is designed to be low-cost. When Enabled = false, the Inject method returns the original callback untouched.

At the call site, this is just one boolean check at composition time. There is zero overhead per call when chaos is off. This means you can safely leave Inject in your production code and toggle a "Game Day" on and off via configuration without redeploying.

Determinism and Seeds

Randomness is the enemy of a stable CI build. To make chaos tests repeatable, Chaos accepts a Seed. If you use the same seed and drive your calls sequentially, you will get the exact same sequence of faults every time.

var chaos = new Chaos { Seed = 42, FaultRate = 0.1 };

What's Next?

Chaos testing is great for verifying that your policies react correctly to single faults. But real-world outages are rarely single events - they are sequences of degradations, partial failures, and recovery cycles.

In our next deep dive, we'll look at Simulation. While Chaos injects random noise, Simulation lets you script precise failure scenarios to see exactly how your system evolves over time.

Go deeper: Read the Fault Injection documentation for a full API reference.

The Flakiness of "Real Time"

If you want to test how a circuit breaker recovers from a 30-second outage, you have two bad options:

1. Actually wait 30 seconds. Your CI pipeline slows to a crawl, and you'll eventually get a flaky test because a background thread took 1ms too long.
2. Use a mock clock. You spend half your day fighting Task.Delay and trying to figure out why your mock doesn't trigger the actual timer inside the library.

Simulation solves this by using Virtual Time.

How Simulation Works

At the heart of a simulation is a SimulationClock. It's a TimeProvider that doesn't track wall-clock time. Instead, it only moves when the driver tells it to move.

When the executor asks to "delay for 10 seconds," the simulation doesn't sleep. It queues a timer and immediately jumps the virtual clock forward to the next due event. Five simulated minutes cost whatever the arithmetic costs and no more.

Because everything runs on a single driver thread, there are no races. The sequence of events is identical every single time you run it with the same seed.

Scripting the Failure

While Chaos injects random noise, Simulation uses Sequence to define a precise script of outcomes. This allows you to model a specific disaster scenario: "The dependency fails twice, then slows down for three calls, then recovers."

Scripting a Callback

You can script any callback by defining a sequence of returns, throws, and delays.

using NResilience.Testing;

// Script a specific sequence of events
var calls = Sequence.For<string>()
    .Throws(new IOException("Connection reset")) // Call 1: Fail
    .Throws(new IOException("Connection reset")) // Call 2: Fail
    .Delays(TimeSpan.FromSeconds(2))            // Call 3: Slow...
    .Returns("Success!")                         // ...then succeed
    .Returns("Success!");                        // Call 4: Succeed

// Run this through your policy
var result = await policy.RunAsync(attempt => calls.NextAsync(attempt), ct);

Scripting the HTTP Layer

For HTTP clients, ScriptedHttpHandler lets you define a sequence of responses without needing a real network transport.

var transport = new ScriptedHttpHandler()
    .Responds(HttpStatusCode.ServiceUnavailable, times: 2)
    .Responds(HttpStatusCode.OK);

using var client = HttpResilience.CreateClient(policy, innerHandler: transport);
var response = await client.GetAsync(uri, ct);

// Assert that we actually made 3 attempts
Assert.Equal(3, transport.CallCount);

The Simulation Report: Facts, Not Anecdotes

The goal of a simulation isn't just to see if the code finishes; it's to measure the impact.

The SimulationReport captures the total cost of the resilience policy during the run. Instead of saying "it felt like it recovered quickly," you get hard numbers:

• Load Multiplier: Exactly how much extra traffic did the retries send to the dependency? (e.g., 1.1x)
• Availability: What fraction of caller requests actually succeeded?
• TimeToRecover: How long after the impairment ended did the system return to 100% success?
• P99 Latency: What was the worst-case experience for the caller?

// Example report output
// Simulation seed 42 over 00:05:00
//   Calls            1000
//   Succeeded        950
//   Failed            50
//   Availability     0.9500
//   LoadMultiplier   1.120
//   Amplification    1.050
//   Latency p50      00:00:00.012
//   Latency p99      00:00:02.450
//   BreakerOpens     1
//   TimeToRecover    00:00:04.200

Because the report is deterministic, you can turn it into a CI gate: Assert.True(report.LoadMultiplier <= 1.2);. If a change to your policy causes a retry storm, the test fails.

Chaos vs. Simulation

If you're wondering which to use:

• Use Chaos when you want to smoke-test your system. It's great for "Game Days" where you want to see if the app survives random instability.
• Use Simulation when you need to prove a specific behavior. It's for regression testing, tuning timeouts, and asserting that your retry budget actually bounds amplification.

Go deeper: Check out the Testing documentation to see how to combine Sequence with FakeTimeProvider for deterministic unit tests.

Why a stress suite?

Most benchmarks measure one call at a time: how many bytes, how many nanoseconds. That is useful, but it is not the same as 80 concurrent callers hitting a library against a dependency that is failing.

NResilience already has an allocation gate in CI to catch regressions, and a latency harness to trend per-operation cost. Neither answers the actual operator's question: what does this process sustain, and what does the dependency feel from the other side?

I built bench/NResilience.Stress to find out. It runs both libraries through the same harness under sustained load and measures throughput, memory, GC pauses, and how much load a guard keeps off a failing dependency. 90 configurations, three repetitions each, 270 runs in fresh processes, about 32 minutes.

The numbers

I paired each NResilience configuration with its closest Polly equivalent - same gate, same suspension count, same test callbacks.

Read the caveats before the scoreboard convinces you of something the run did not measure. No saturation was tested - 80 workers is high concurrency, not a saturated thread pool. Polly ran its cheapest configuration, pinned at 8.7.0. Throughput ratios moved 10-30% between runs and are quoted as ranges; anything without one should be read as "on this machine."

What was surprising

Deterministic allocations. I expected noisy medians. Instead: 0% spread across three repetitions, and a second, independent full run byte-for-byte identical. At the floor - every feature off - NResilience costs 224 B/call, exactly what the raw test callback allocates with no library at all. Turning NResilience off is free, byte-for-byte.

The retry tie. Both libraries at three attempts, zero delay: 1.00x-1.13x, a tie. I expected the retry loop itself to favor the flat executor. It does not - the advantage is in the frame around the call, not the retry loop. The win is not "we retry faster."

The p99 spread. Identical repetitions of the same cell produced p99s spanning 1.02 ms to 8.49 ms - a factor of 8 - with GC under 1% of the window. Multi-millisecond stalls of a whole process on a fanless laptop. That is the environment, not the library, and it means this run makes no tail-latency claim.

The budget's price. Against a dependency failing half its requests, the budget cut retries-per-operation 18.9x - and completed 53% as many operations, refusing the rest. Client-side availability for load relief on a shared dependency is a good trade across a fleet and a bad one for a single client with no fleet to protect.

How I kept it honest

Benchmarks are easy to rig by accident. A few decisions mattered:

• Fresh process per test, so GC state belongs to exactly one library.
• Shuffled order, so thermal drift does not land on one library.
• Three repetitions, spreads published - a single run cannot support a claim that is a few percent.
• The HTTP dependency in its own process, so its garbage does not pollute the client's numbers.

One caveat: Polly has no equivalent of the deadline, classifier, or retry budget, so in the "default machinery" pair NResilience is doing more work, not the same work. Both sides also ran their cheapest configuration - no telemetry on either. A claim that survives both of those notes is worth more than one that does not.

Try it yourself

The suite ships with the library:

dotnet run -c Release --project bench/NResilience.Stress -- --quick   # one DOP, one rate, single repetition
dotnet run -c Release --project bench/NResilience.Stress             # the full matrix, ~32 min

Reports land under artifacts/stress/ as timestamped Markdown and JSON. The JSON holds every repetition, not the medians - so anything asserted in the report can be recomputed from it. Run it on your hardware and the numbers will differ; throughput ratios here moved 10-30% between runs.

One machine, one run: a 10-core Apple Silicon laptop, .NET 10, Polly pinned at 8.7.0.

The full figures are in bench/stress-results.md. For how the same performance discipline is enforced per-change in CI, Shipping a resilience library covers the gates. For the executor the suite is measuring, One loop, one allocation explains the flat executor. The library and docs live at docs.nresilience.net - and if this is useful, a GitHub star helps others find it.

Stop calling it

When a dependency starts failing, the instinct is to retry until it goes through. If a hundred instances of your app all do that at once, you have not fixed the outage - you have organized a distributed denial of service attack against your own infrastructure. That is how a database hiccup becomes a system-wide outage.

The kindest thing you can do for a struggling dependency is to stop calling it. That is the breaker's whole job.

The breaker is a state machine with three states you will actually see in operation:

• Closed - normal operation. The circuit is complete, traffic flows, and the breaker quietly samples every attempt.
• Open - the breaker has tripped. Every call is refused with a CallRejectedException without touching the network. The dependency gets silence, which is the one thing it needs.
• HalfOpen - after the break duration elapses, a small number of trial calls are let through. One concurrent probe by default. Two successful probes close the breaker; a failed probe re-opens it immediately.

That last number is worth pausing on. Closing after a single lucky probe is how you get oscillation: a dependency that is only intermittently available passes one probe just as a fleet of accumulated retries arrives, and the cycle repeats. Two consecutive successes prove the dependency actually recovered.

There is a fourth state, Recovering, but it is opt-in - we get to it later.

The break doubles

If a probe fails, the breaker re-opens - and the next break is longer. The break duration doubles on each consecutive open, up to a configured maximum (15 seconds by default, 2 minutes at the top). It is exponential backoff applied to the breaker itself.

This is what bounds a flapping dependency. A service that comes back, accepts its probe, and dies again does not get probed every 15 seconds forever: 15, 30, 60, 120. Unstable dependencies sit in the timeout corner longer, preventing an endless open/half-open loop.

Tripping without guessing a rate

The classic breaker asks you for a flat error threshold: trip at 5%. That number cannot fit two dependencies. For a payments API that normally runs at a 0.02% error rate, 5% is a page-everyone catastrophe. For a flaky third-party search backend that has always failed 30% of calls, 5% is a quiet Tuesday. Hardcoding a threshold before the code has seen production traffic is guesswork either way.

NResilience measures instead. By default the breaker tracks the dependency's own error rate over a rolling 5-minute baseline and trips when the current window's rate exceeds 5 times that baseline. A payments API spiking to 0.1% - five times its flawless normal - trips, even though no absolute rule would notice. The search backend's 30% is accepted as its normal; only a significant deterioration trips anything.

Both relative trips come with guards, because a relative measurement against a tiny sample is a way to trip on noise:

• A floor of 5% absolute. Below that, nothing is wrong no matter how quiet the baseline was. Five times a baseline of nearly zero is still nearly zero, so without the floor the first error of the day would open the circuit.
• At least two failures in the trip window. One failure is not a rate, it is an event. At a 5% floor and 20 minimum calls, a single transient error would be exactly the threshold - so the relative trip needs a second one.
• Enough samples. The error baseline needs 100 outcomes before the relative trip arms (Failures.MinimumSamples); the latency baseline needs 20 successful calls. Until then the breaker behaves exactly like a consecutive-failures breaker - which is also exactly what it does at cold start, because the consecutive-failure trip is on and needs no history.

The consecutive-failure trip (5 by default) is the default's backbone for a related reason: a rate-based trip needs throughput to mean anything, and a service doing two calls a minute would never arm one. The breaker samples attempts, not logical calls, so a first attempt that trips the breaker stops the second attempt immediately.

Brownouts: failing slowly

A dependency does not have to return errors to be broken. A 30-second 200 OK is often more dangerous than a 10ms 500 Internal Server Error. The hard failure is fast and frees your threads; the brownout ties up your thread pool and connection pool while claiming everything is fine.

The breaker tracks a latency baseline next to the error baseline: the median of successful call duration over the last 5 minutes. An attempt counts as slow when it exceeds 3 times that median, and half the trip window being slow trips the breaker. Nothing here asks you for a millisecond figure.

Why the median and not a high quantile? Because a threshold defined as "the p99 of recent latency" has 1% of calls above it by construction, whatever the dependency does - it can never trip. The median survives a brownout until it accounts for more than half the window it is measured over, and the baseline lives on a 5-minute clock against the trip window's 30 seconds, so the trip window fills with slow calls long before the baseline forgets what healthy looked like.

The baseline is fed only by successful attempts, and it survives an open, a close, and a Reset. It measures the dependency; it does not judge it. That is what makes the first probe against a still-degraded dependency recognizable as slow - the breaker remembers the pre-brownout latency.

The refusal that breaks the spin

Here is a failure mode the breaker itself can cause. A developer wraps a resilient call in a while loop - a critical background job polling until the service is back. The breaker opens, the call is refused... instantly. No network latency, no timeout, just an exception at the speed of a method call. The loop catches it, calls again, gets refused again. Thousands of iterations a second, CPU at 100%.

The mechanism built to shed load has become a load generator - self-inflicted, and invisible on the network dashboards, because outbound traffic is zero. Your system looks safe while the client's CPU melts.

NResilience fixes this with guarded rejection: a refused call pauses 100ms before returning the refusal. 100ms is imperceptible to a user, but it's a speed bump that caps a polling loop at 10 iterations a second instead of 10,000.

The pause is not configurable, on purpose: its job is to set a floor under the rate of a rejection loop, not to be a tuning knob. It is bounded by the time remaining on the call's deadline and observes the cancellation token, so a refusal never pushes a call past the caller's budget and cancelling aborts it immediately. If you schedule your own retries, CallRejectedException.RetryAfter says when to come back - the break actually being served, not the nominal figure.

if (result.Exception is CallRejectedException rejection)
{
    Console.WriteLine(rejection.Reason);   // DependencyUnavailable, or BudgetExhausted
    Console.WriteLine(rejection.RetryAfter); // when to come back, when there is an answer
}

This is the general truth the breaker makes visible: you have to protect the dependency from the client, and the client from its own protective mechanisms.

Hand the traffic back over a ramp

Two successful probes prove the dependency can serve two calls. A naive breaker reads that as proof it can serve two thousand, closes, and the entire backed-up load arrives in one millisecond. The recovering dependency has a cold cache and an empty connection pool. It fails, the breaker re-opens with a doubled break, and you have the crash/wait cycle all over again.

Recovery adds a transitional Recovering state between half-open and closed. It admits a small fraction of traffic - 5% to start - and refuses the rest. The fraction is the lower of two terms:

• A clock, climbing linearly across a ramp whose length is a fraction of the break just served (default 0.25, so a 15-second break ramps over about 4 seconds).
• Evidence, which is AIMD - the same algorithm TCP uses for congestion control. Admitted calls coming back fast add a step, slowly and linearly. A call comes back slow: the admitted fraction is slashed in half on the spot. Aggressive on stress, cautious on recovery.

One real failure during the ramp abandons the hypothesis and re-opens the breaker with the grown break. The ramp is opt-in because a call refused during it would have been served by a breaker that closed immediately - but the alternative is not "all served". It is "all served once, then all refused again for twice as long".

The synchronized probe, and equal jitter

The ramp solves the pod-level cliff. There is a fleet-level problem just before it: 200 pods watching the same dependency all trip within a second of each other, all set the same break, and all fire their one probe in the same second. The recovering gateway receives a synchronized pulse of 200 probes, fails them, and all 200 breakers double their backoff together. Fleet-wide oscillation.

The standard fix is jitter - randomizing the break so the pods do not wake together. But Jitter.Full has a flaw: the random delay could be anywhere from zero to the full break, so a pod could probe 200ms into a 15-second break. The break duration is not just de-correlation - it is the guaranteed rest period the dependency gets.

So NResilience defaults to Jitter.Equal: at least half the break is strictly enforced, and only the second half is random.

var breaker = new Breaker(settings: new BreakerSettings
{
    Name = "search",
    BreakDuration = TimeSpan.FromSeconds(15), // now half of that, plus up to half again
    BreakJitter = Jitter.Equal,               // the default
});

Every pod waits at least 7.5 seconds, no exceptions, then adds a random 0 to 7.5 on top. The probes spread over a wide window, and the rest period survives.

Who the breaker trips for

One more operational question: the blast radius. If you wire one global breaker for a dependency, tenant A's malformed requests trip it, and tenant B - sending perfectly valid traffic - finds the door locked too. Tenant A has weaponized your breaker against tenant B.

NResilience isolates state by default where it can: the HTTP handler scopes breakers per host, so a dead analytics API does not trip calls to the auth API; the gRPC integration scopes per service. For the multi-tenant case specifically, PolicyScope<TKey> gives every tenant, shard, or channel its own breaker, retry budget, and latency estimate:

private static readonly PolicyScope<string> Tenants = new(
    Resilience.Default with { Breaker = new Breaker() });

await Tenants.For(tenantId).RunAsync(
    attempt => calls.NextAsync(attempt), cancellationToken);

Tenant A's failures only accumulate in tenant A's breaker. One bound to know: the scope holds at most maximumKeys entries (default 1024), because every entry carries a breaker and a budget, and an evicted tenant that returns gets a fresh breaker that does not remember being open.

What counts as evidence

One subtlety that keeps all of the above honest: the breaker does not count every non-200 as a failure. Outcomes are classified into verdicts, and only Transient outcomes are evidence against the dependency. A 500 is a genuine failure. A 429 - or a 503 carrying a Retry-After header - is classified Throttled: the dependency is up and defending itself with its own rate limiter. The breaker ignores throttled responses entirely, because tripping on a functioning rate limiter turns a working defense into a hard outage. And Permanent outcomes are usually client-side facts, not dependency health.

The breaker as a mirror

One last detail. When a breaker opens, the health check reports Degraded, not Unhealthy. That is deliberate: the dependency is down, but this process is doing exactly what it was designed to do - shedding load and failing fast. Reporting Unhealthy invites Kubernetes to restart the pod, which discards the breaker's state and unleashes the traffic storm all over again. Restarting the pod does not fix the database.

That visibility is worth more than it first appears. When a breaker opens and a seemingly unrelated part of your application degrades with it, the breaker has done more than stop a crash - it has revealed the true blast radius of your architecture: how it actually works, not how you drew it on the whiteboard. It is worth putting breaker state on a dashboard for that reason alone.

For more information, see Breaker internals has the full state machine, the arithmetic behind every default, and the recovery-ramp failure modes.

The multiplication nobody sees

Retries compose multiplicatively. A frontend that allows three retries calls a backend that allows three, which calls a database that allows three: one user action that fails at the bottom becomes 4 x 4 x 4 = 64 database attempts. Each layer's configuration looks reasonable in isolation - three retries is a modest number - and every layer is blind to the aggregate, because no single layer can see the whole call chain.

This is what makes a retry storm different from ordinary overload. It is not one misbehaving client. It is a fleet of clients, each independently behaving exactly as configured, delivering hammer blows to a dependency at the one moment it can least afford them. The mechanism every layer installed to survive the failure is the thing that finishes the dependency off.

The obvious fix - lower the attempt counts - does not work, and it is worth seeing why before reaching for it. Attempts compose multiplicatively: whatever fixed limit each layer picks, the product across layers grows faster than any of them. Two retries per layer is still 27 attempts at the bottom of a three-hop chain. No per-call number fixes a cross-call problem.

The fix is a fraction, not a count

A budget expressed as a fraction of traffic composes differently. If every client independently funds retries at 10% of its successful traffic, fleet-wide amplification is bounded at 1.1 times the original traffic - whatever the topology, however deep the call chain, with no coordination protocol anywhere.

That bound comes from arithmetic, not cooperation. Each client holds its own retries at one-tenth of its own successes, and the fleet's total is the sum of those ratios - so the fleet's aggregate amplification is 1.1x too, however the clients differ in size. Google SRE uses the same figure; Finagle and Envoy budget 0.2.

This is why the retry budget is on by default in NResilience and the attempt count is only a maximum. Attempts answers "how many times may this one call retry". The budget answers "how much traffic may this process add to a struggling dependency" - which is the question the dependency actually cares about.

The token bucket

The budget is a token bucket with two sources of tokens and one rule about spending:

• Deposits. Every successful attempt deposits fraction tokens - 0.1 by default. Ten successes fund one retry. In steady state, retries can add at most 10% on top of your successful traffic, whatever your traffic is.
• A floor rate. minimumPerSecond (3 per second by default) refills the bucket at a constant rate regardless of traffic, so a quiet client whose successes are too sparse to fund anything can still retry. Set it to zero and the fraction becomes the only source of tokens.
• A burst bound. The bucket's capacity is ten seconds of the floor rate - 30 tokens at the defaults. This caps the burst a recovering client can spend at once without changing the sustained rate the floor and deposits set.
• A full bucket at cold start. A new process begins with a full bucket. Throttling a fresh deployment's first retries would penalize deployment rather than a storm, and the budget's job is the opposite.

One charging rule sits above all of this: only retries are charged. The first attempt of every call always executes. The budget is not a rate limiter - it refuses amplification, not traffic. A dependency cannot be starved by its clients' budgets, because no budget can stop the calls the caller actually wanted to make.

The lock holds arithmetic and nothing else

Deposit runs on every successful attempt, which makes it the one gate the library takes on the success path. Hold one static policy - the shape every application settles into - and all of that traffic funnels through a single lock inside the budget.

So the lock's critical section is arithmetic and nothing else. The clock read happens before the lock is taken; the timestamp arrives with the caller. A thread can arrive holding a timestamp another thread has already refilled past, and the guard for that already existed for non-monotonic clocks: a refill with no elapsed time accrues nothing rather than rewinding the refill point. The lost accrual is the sub-microsecond gap between two clock reads, which no per-second rate can observe - the lock remains a thin layer of arithmetic on the success path.

It is a small thing, and it is the difference between a bucket that costs nothing on the hot path and one that shows up in a flame graph.

The hint that stays honest

When a retry is refused, the call stops with StopReason.BudgetExhausted after the library's short guarded rejection pause, and CallRejectedException.RetryAfter says when to come back. The number is worth trusting, because of when it is computed.

The executor asks the budget for the hint after serving the 100ms rejection pause, and the hint refills the bucket first. Get that ordering wrong and the answer is the shortfall as it stood 100ms ago - an overstatement, which is the one thing a RetryAfter must not be. A caller that schedules its own polling on the hint gets the wait the floor rate actually needs to accrue the next token, not a padded version of it.

And when the floor is zero, the hint is null. With no floor rate, only successful traffic can refill the bucket, and no one can honestly predict when your next success arrives. The budget would rather say nothing than make something up.

What the budget exempts

Two kinds of retry never draw on the bucket, and both exemptions are about protecting the mechanism from itself.

A refusal this process issued. When your own rate limiter refuses an attempt, the outcome is classified Throttled with a self-imposed bit set - the dependency was never contacted, so the retry costs it nothing. Charging that retry against the budget would let a burst of self-throttling quietly drain the capacity that real transient failures need, and it would drain it at exactly the moment a client is under load, which is when that capacity matters most. So a self-imposed refusal neither spends nor deposits; it is not evidence in either direction.

The breaker needed a subtler version of the same care. Its recording is still called for a self-imposed refusal - it records nothing, but it returns the in-flight probe slot the refused call occupied. Skip the call to save the work and the probe count leaks downward until a half-open breaker has no slots left to give and never recovers.

Half the bucket, for work nobody is waiting on. A nightly backfill and a checkout draw on the same bucket, and by default the executor cannot tell them apart. But a budget is a fraction of traffic, not a queue - the only way to guarantee capacity for the checkout is refusing to hand the last of it to work nobody is waiting for. A Sheddable call must leave half the bucket behind: while the dependency is healthy, nothing is refused; once the bucket drains, the backfill stops first.

Every other criticality level spends the bucket exactly as before. The refusal is the ordinary one - same pause, same StopReason.BudgetExhausted - because criticality changes when a refusal happens, not what it is.

Sharing, and where buckets go to die

Budget state is per process and unshared. There is no coordination between pods, and that is not a defect - the statistical argument from the fraction is what makes coordination unnecessary. But it decides where a bucket can live:

• RetryBudget.Automatic, the default, resolves to a bucket private to each policy instance - or to each key when the policy is scoped. The privacy is the point: it is why a storm against payments cannot throttle retries to search. Policies derived from the same preset deliberately do not share a bucket.
• RetryBudget.Shared("payments") is a process-wide budget looked up by name. Two policies naming the same string share it, and the first caller's parameters win. Share one across the policies that hit the same dependency, so three clients of the same API do not each get their own 10% allowance.
• RetryBudget.None is the one spelling of off, and it is right only when the dependency is known not to be shared.

The per-process rule has a consequence worth stating bluntly: a budget allocated per HttpClient instance, or resolved from a scoped DI container, is worthless. It is discarded before it observes enough traffic to bound anything. Hold one instance, or name it.

Two lifetime details keep the mechanism intact where you would expect it to break. In DI, the bucket is pinned to the registration name, so a configuration reload creates a new policy instance but keeps the traffic history. And the bucket an Automatic marker resolves to lives in a ConditionalWeakTable keyed by policy identity, so lazy initialization does not affect record equality - two equal policies still compare equal after one of them has warmed a bucket.

Reading the budget

A budget near capacity is the library telling you it is preventing a storm right now:

if (result.Exception is CallRejectedException rejection)
{
    Console.WriteLine(rejection.Reason);     // BudgetExhausted
    Console.WriteLine(rejection.RetryAfter); // when the floor funds the next token
}

Utilization reads 0 to 1 for dashboards. A budget sitting near 1 is a client whose retries are being refused - a symptom to alert on, not a steady state, because it means your dependency is failing and the budget is absorbing what would otherwise be your contribution to the incident. The ratio of nresilience.attempts to nresilience.calls is the same story from the other side: it is the retry fraction you actually ran, and it is the one number that says whether the amplification you designed for is the amplification you got.

The loop no single layer can see

The budget bounds what this process spends. The other half of the amplification problem is structural: your frontend retries onto a backend that retries onto a database, and every layer sees only its own three attempts - the same blindness that makes fixed limits useless.

NResilience's HTTP handler detects nesting two ways: a flag in the execution context catches it within a process, and an HTTP header catches it across process boundaries. The handler stamps X-NResilience-Retrying: 1 on every request it can retry, and the stamp is on the first attempt, not the retries - because by the time a retry goes out, the amplification has already happened. The gRPC interceptor carries the same marker as x-nresilience-retrying: 1, because the chain that amplifies is rarely all one protocol.

In an ASP.NET Core app, receiving the marker is one line:

app.UseResilienceNestedRetry();

Registered before anything that makes an outbound call, it publishes the flag this request carries, so the middle hop of a chain reports NestedRetry events itself and can see an amplification it is part of.

Detection reports and does not intervene: silently dropping configured retries would be surprising and hard to debug. The right response is architectural - the layer closest to the failing dependency should do most of the retrying, and the layers above it should pass the call through. The library finds the loops; you decide what they mean.

The pacing around it

The budget decides whether a retry happens. The backoff decides when, and its defaults each carry a reason:

• Two bases, because two verdicts. A Transient failure waits 100ms before the first retry; a Throttled one waits 1s. A delay tuned for connection resets is too aggressive for a rate limiter - a 429 is the dependency telling you about its token bucket, not about how long its queue is.
• Full jitter, always on. A brief outage ends and every client's accumulated retries arrive together - a second traffic spike on top of the recovery. Jitter removes the correlation; without it, a fleet turns a two-second blip into a thundering herd.
• Server pushback wins verbatim. A Retry-After header on a 429 or 503 replaces the computed delay exactly - no jitter, no curve - capped only by the configured maximum and the remaining deadline. Where the server knows, it says so.
• The measured base, opt-in. Backoff.Measured(1) sets the transient base to one recent normal call's duration instead of a millisecond count that is only right for one dependency. It is the one measured term the library does not turn on for you, because it is the one that can lengthen a delay rather than only shorten one - so it is clamped to a factor of 10 either side of the constant you wrote, and the constant stays the anchor.

The trade, priced

None of this is free. A budget is a claim that surviving a dependency's struggles matters more than the availability your retries would have bought during them - and that trade can be measured before shipping. A five-minute simulation of a dependency that goes eight times slower for one minute, at 500 calls a second, puts the worst second the dependency feels at 2.1x load with the budget off and 1.1x with it on, for six points of this client's availability.

That is the whole idea in two numbers. The budget converts your worst second into the dependency's survivable one, and it is on by default because the alternative - every client independently being reasonable - is how the storm starts.

For more information, see Retry budget internals, which has the full token-bucket mechanism and the reasoning for bounding by traffic rather than by attempts.

The Resilience Journey: A Synthesis

Over the last 26 posts, we have explored the anatomy of a resilient system. If you started at post 1, you've seen the progression from a simple HttpResilience.CreateClient() call to the complex machinery of live latency quantiles and Native AOT probes.

The overarching theme is bounding.

Software fails when it is unbounded. A request that can hang for 30 seconds is an unbounded resource. A retry loop that multiplies traffic by 10x during an outage is an unbounded load. A memory allocation that grows with every layer of a pipeline is an unbounded cost.

NResilience is a tool for stating those bounds as plain values and letting a single, flat executor enforce them. Whether it is a Deadline capping the whole call or a RetryBudget bounding fleet-wide amplification, the goal is the same: ensure that when a dependency fails, your application degrades gracefully instead of crashing.

The Core Pillars of NResilience

If you take away only three things from this series, let them be these:

1. Policies are Values

Resilience should be a configuration, not a complex builder chain. By treating policies as immutable values that you derive with with, the library removes the "wrong ordering" bugs that plague layered architectures.

2. Measured, Not Guessed

The most dangerous constant in a system is a "guessed" timeout. By using measured bounds - where the library adapts to actual latency quantiles - you get the protection of tight timeouts without the fragility of hard-coded constants.

3. Correctness is a CI Gate

Performance and correctness cannot be "hope-driven." By enforcing allocation budgets and AOT compatibility as blocking CI gates, we ensure that the library's promises are backed by evidence, not assertions.

What's Next: The Road to v1.0

NResilience is an evolving project. While the core engine is stable, the journey to v1.0 involves hardening what is here and widening what it covers.

Coming Soon:

• More protocol coverage: Deepening the gRPC integration and evaluating support for message brokers.
• More adaptive defaults: Refining the Automatic budget and limiter settings based on real-world usage patterns.
• More docs and guides: The docs gate keeps every snippet compiling and running; the next push is breadth, not correctness.

Final Word: The Discipline of Failure

The most important lesson of this series isn't about a specific API or a memory budget. It's the realization that failure is a first-class citizen of distributed systems.

The goal isn't to build a system that never fails - that's impossible. The goal is to build a system that knows exactly how it is allowed to fail, how much it is allowed to retry, and when it must simply stop.

Thank you for following along with the NResilience series. Now, go bound something.

The GitHub Repository has the full source, CI pipeline, and contribution guide if you want to dive in.

The Promise vs. The Proof

When a library claims to be "allocation-free" or "correct by design," those are bold assertions. In a distributed system, a tiny regression in a core loop can cause a p99 spike that takes down a production cluster.

To move from "asserted" performance to "measured" performance, you cannot rely on manual benchmarks. You need a system of gates that make it impossible to merge a regression.

The Five Gates of CI

NResilience's CI pipeline isn't just about "passing tests." It is a series of five specialized gates that must be cleared before any code hits the main branch.

1. The Behavior Gate

The standard functional suite. It ensures the logic is correct across thousands of permutations of retries, deadlines, and breakers. dotnet test tests/NResilience.Tests/...

2. The Allocation Gate

This is the most critical gate for performance. We use GC.GetAllocatedBytesForCurrentThread() in Release builds to enforce strict memory ceilings. If a change adds even a few bytes to the core execution path, the build fails. dotnet test tests/NResilience.Gates/...

3. The Integration Gate

Real-world verification. This gate runs loopback tests against actual HTTP and gRPC listeners to ensure the library survives the "wire" (TCP resets, timeouts, and partial writes). dotnet test tests/NResilience.IntegrationTests/...

4. The Analyzer Gate

Since the library provides diagnostics to help users avoid bugs, the diagnostics themselves must be bug-free. This gate tests the analyzers against a suite of broken and fixed code samples. dotnet test tests/NResilience.Analyzers.Tests/...

5. The Native AOT Gate

To guarantee AOT compatibility, we don't just check for warnings. We publish a native binary and execute it. If the published binary crashes or fails to bind configuration, the gate closes. dotnet publish ... -p:PublishAot=true

Bundling the Intelligence

A common pattern for .NET libraries is to ship analyzers in a separate NuGet package (e.g., NResilience.Analyzers). We rejected this.

A diagnostic that isn't installed is a diagnostic that doesn't exist. If a user misses the analyzer package, they will ship a policy with a 30s deadline and a 90s retry loop, and it will fail silently in production.

By bundling the analyzers internally in the main package (analyzers/dotnet/cs), we ensure that the moment you add the library, the compiler starts reviewing your resilience design. The diagnostics travel with the code.

API Manifests and Pinned Dependencies

To prevent "API drift," NResilience uses manifest-based reviews. Every change to the public surface is tracked in PublicAPI.Shipped.txt. When a dev changes a method signature, they must update the manifest. This turns a casual code change into a deliberate architectural decision during the PR review.

We apply the same rigor to dependencies via Directory.Packages.props:

• Competitive Baselines: Dependencies like Polly.Core are pinned. We only bump them when we have re-measured our performance claims against the new version - which is what the sustained-load stress suite in bench/NResilience.Stress is for.
• Lockstep Pinning: All Grpc.* packages are pinned to the exact same version to prevent the notorious "mixed version" crashes common in gRPC-dotnet.
• Compatibility Pinning: Roslyn (Microsoft.CodeAnalysis) is pinned low to ensure our analyzers load in as many compiler versions as possible.

The Docs-to-Code Pipeline

The most frustrating part of using a library is a code sample in the docs that doesn't compile.

NResilience treats documentation as code. Every snippet in our markdown files is actually extracted from a real .cs file in tests/NResilience.Docs, marked up with snippet: tags.

The pipeline works like this:

1. Verification: CI runs dotnet test on the docs project. If a sample fails, the build fails.
2. Extraction: A custom tool extracts code between // <snippet:name> markers.
3. Synchronization: The tool scans the markdown for matching tags. If the markdown content differs from the source code, the CI fails with a "drift" error.

This ensures that the documentation is never a "best guess" - it is a verified, executable truth.

Conclusion

Shipping a resilience library is a lesson in humility. You are building the safety net for other people's systems. By treating performance as a CI gate, analyzers as a core product, and documentation as executable code, we ensure that NResilience's promises are backed by evidence.

The GitHub Repository has the full CI pipeline and source. For the sustained-load evidence behind the allocation and GC claims - a full NResilience-vs-Polly run with every repetition published - Stress-tested against Polly is the summary. To close the series, Beyond the basics synthesizes the core lessons and outlines the road to v1.0.

The Philosophy of the Small Decision

When you build a resilience library, you aren't just writing code to handle errors; you are writing code that manages the relationship between two systems under stress. In that environment, "standard" logic often fails.

A simple while loop for retries is fine for a demo, but in production, that loop can become a weapon. If not carefully guarded, a resilience library can accidentally amplify a brownout into a total outage. To prevent this, NResilience is built on a series of small, deliberate decisions - most of them invisible to the user - that ensure the library remains a protector, not a load generator.

Guarded Rejection: Breaking the Spin

When a circuit breaker trips, it enters the Open state. In a naive implementation, every call to an open breaker returns a "Failed" result immediately.

This sounds efficient, but it creates a dangerous edge case: the CPU Spin. If a developer accidentally puts a resilient call inside a tight while loop (or a polling mechanism) without any delay, an open breaker will return thousands of failures per second. This consumes 100% of the CPU, starving other parts of the application and potentially crashing the process.

NResilience implements Guarded Rejection. When a breaker is open, the library pauses for 100ms before reporting the refusal. The pause is capped by whatever is left on the deadline and observes your cancellation token, so a refusal never pushes a call past the caller's budget, and cancelling during the pause aborts immediately.

This pause is non-configurable on purpose. It exists to set a floor under the rate of a rejection loop, not to be a tunable performance parameter. It is short enough that a human user won't notice it, but long enough to make a CPU spin impossible. It turns a potential crash into a manageable delay. For callers who manage their own retries, CallRejectedException.RetryAfter says when to come back.

Admission Control and the "Self-Imposed" Bit

In NResilience, every attempt ends with a Verdict. Whether it's a 503 from a server or a timeout from the library, the outcome is mapped to a standard verdict (Ok, Transient, Throttled, or Permanent).

But there is a critical distinction when the failure comes from inside the library - for example, when a ConcurrencyLimiter refuses a permit.

If a limiter refuses a call, NResilience classifies it as Verdict.Throttled with a self-imposed flag. This distinction is vital for two reasons:

1. The Breaker ignores it: If your own limiter is refusing calls because you've hit your concurrency cap, that isn't a sign that the dependency is failing. It would be a bug if your local limiter tripped your circuit breaker.
2. The Budget ignores it: Retry budgets are designed to prevent you from overwhelming a struggling dependency. If you are throttled by your own internal limiter, you aren't adding load to the server. Therefore, self-imposed refusals are exempt from the retry budget.

The executor enforces this from the inside: a RateLimitedException is always classified as self-imposed, and no classifier rule can turn a refusal this process imposed on itself into evidence against the dependency. Value equality follows the flag too - Verdict.Throttled() and Verdict.Limited() carry the same kind but are not equal.

By distinguishing between "the server is full" and "I am full," NResilience ensures that your resilience settings don't fight each other.

Why Live Quantiles are the Only Safe Hedge

Hedging - sending a duplicate request when the first one is slow - is the sharpest tool for killing tail latency (p99). It is also the easiest to misuse.

The classic approach is a fixed delay: "If the call takes more than 200ms, send a second one." This is a trap. If your dependency slows down and every call starts taking 210ms, you have just doubled your traffic to a service that is already struggling. You've turned a brownout into a retry storm.

NResilience uses a live latency quantile (e.g., p95). Instead of a fixed 200ms, the library tracks the actual latency of recent calls. If the p95 is 10ms, the hedge fires at 10ms. If the dependency slows and the p95 moves to 2s, the hedge fires at 2s.

Because the threshold moves with the traffic, the fraction of calls that hedge remains constant (roughly 5%). The load on the dependency is stable, regardless of whether it is responding in milliseconds or seconds.

Data-Driven Design: Verdicts and Events

To keep the library performant, the internal data structures are stripped to the bone.

The Verdict is a 16-byte struct. It has to be: a verdict is live across the attempt await, so every byte it carries is paid in the state-machine box of every suspending call in the library, whether or not anything is ever throttled. That is why the self-imposed flag shares the kind's byte rather than sitting beside it as a bool, and why RetryAfter is a biased long rather than a TimeSpan? - each packing saves eight bytes for everyone.

Furthermore, NResilience follows a strict "One Event Per Call" rule. While a call might have five attempts, you only ever receive one terminal CallEvent. This event summarizes the entire journey: the final verdict, the number of retries, and the total delay. This prevents your telemetry and logs from being flooded by "Attempt X started" noise, giving you a clean, high-signal view of your system's health.

Conclusion

Resilience is not about adding more features; it's about removing ways for the system to fail. Guarded rejections, self-imposed throttles, and live quantiles are small decisions, but they are the difference between a library that "works in a demo" and a library that survives a production outage.

The Classification and Verdicts guide documents every outcome. For the story of how these decisions are enforced as the library ships, Shipping a resilience library covers the five CI gates, analyzer bundling, and the docs-to-code pipeline.

The Problem: The Layered Tax

In a traditional resilience pipeline, you stack strategies like a cake: Retry -> Timeout -> Circuit Breaker -> Callback.

In .NET, every async method that suspends (waits) heap-allocates a state machine "box" to track its progress. In a layered architecture, each strategy is its own async method. This means if you have three layers of resilience, you pay the allocation tax three times for every single call.

As you add more complex policies - hedging, adaptive limits, or nested retries - the allocation cost grows linearly. This creates GC pressure that manifests as tail-latency spikes (p99), which is exactly what resilience libraries are supposed to prevent.

The Solution: The Flat Executor

NResilience rejects the pipeline model. Instead of a chain of strategies, it uses a Flat Executor.

The core of the library is a single async loop. This loop handles the entire lifecycle of the call - admission, deadline checks, attempt looping, backoff, and classification - inside one method frame.

One Frame, One Box

Because there is only one main async method, there is only one state-machine allocation per call. Whether your policy is a simple retry or a complex combination of adaptive limits, budgets, and deadlines, the overhead remains constant.

The cost is not based on how many settings you have, but on the execution path taken.

Measured Allocation Budgets

We don't guess about performance; we measure it in CI. NResilience enforces strict allocation ceilings using GC.GetAllocatedBytesForCurrentThread() in Release builds.

By "quarantining" the higher cost of hedging, standard callers never pay for features they aren't using. If you don't hedge, you stay at the floor. Every figure is a ceiling enforced by CI, with roughly 15% headroom for cross-architecture variation - and NResilience deliberately does not claim to be "zero allocation" in an unqualified sense. The claim is one box instead of one box per layer, and the build gates enforce it.

These budgets hold under sustained load, too. The stress suite measured a whole policy end to end against Polly: with every feature off, NResilience allocates 224 B/call - byte-for-byte what the raw test callback allocates with no library at all - while Polly's empty pipeline adds 304 B. The full default policy sits at 624 B/call, and the process spends about 2.3x less time paused in GC than Polly's retry + timeout pipeline. The full run, with every repetition published, is in Stress-tested against Polly.

Native AOT and the AOT Probe

The flat executor's design makes it a natural fit for Native AOT. Layered pipelines often rely on reflection or complex generic compositions that can trip up AOT compilers or lead to oversized binaries.

NResilience is reflection-free. To ensure this stays true, our CI pipeline includes an AOT Probe:

1. It publishes the library with PublishAot=true.
2. It treats all AOT warnings as errors.
3. It executes the resulting native binary to verify that DI, configuration binding, and telemetry still work.

This means you can ship NResilience in a tiny, fast-starting native binary without worrying about runtime crashes due to missing metadata.

The Trade-off: Composition vs. Performance

The cost of the flat executor is a loss of "compositional extensibility." In a pipeline library, you can write a custom IStrategy class and plug it into the middle of the chain. In NResilience, you cannot.

To solve this without breaking the flat loop, we provide Extension Hooks. These are specific, high-leverage points where you can inject logic without adding new async frames:

• Admit: A hook to decide if a call should proceed before the attempt starts.
• BeforeAttempt: Setup logic that runs before every attempt, including the first; its time counts toward the deadline.
• OnEvent: A stateless, allocation-free listener for logging and telemetry.
• Backoff.Custom: A delegate to compute the delay between attempts.

You get the customization you need, but the library keeps the performance of a single loop.

Conclusion

Resilience should not come at the cost of performance. By replacing the layered pipeline with a flat executor, NResilience ensures that adding safety doesn't add latency. One loop, one allocation, and zero reflection.

The flat executor exposes specific extension hooks instead of a general strategy pipeline. Bending the library covers Admit, BeforeAttempt, Backoff.Custom, and OnEvent - the targeted escape hatches that let you inject custom logic without breaking the performance model.

The Problem: gRPC isn't just HTTP

If you look at a network trace, every gRPC call is an HTTP/2 POST. But at the application level, the semantics are entirely different. You don't have status codes like 404 or 503 in the response header; you have a trailing grpc-status.

This creates several technical challenges:

1. Blast Radius: A single gRPC client often connects to a server that hosts ten different services. If the "Payments" service is down, you don't want to trip the breaker for the "Catalog" service.
2. The Timer Race: The gRPC stack has its own internal deadline management. If you set a timeout in the library and a timeout on the wire to the exact same millisecond, you have a race condition. You don't know who won, and your telemetry becomes a lie.
3. Streaming State: Unlike a unary call, a stream is a long-lived conversation. If a stream fails after the client has already received five messages, you can't just "retry" the call - you'd be asking the server to start the whole stream from the beginning.

The Solution: gRPC-Aware Resilience

NResilience integrates directly into the gRPC channel via a ResilienceInterceptor, allowing it to understand the specific needs of RPCs.

1. Per-Service Scoping

By default, NResilience does not use a global breaker for the entire client. Instead, it scopes guards by the gRPC Service.

If you are calling OrdersService and InventoryService on the same host, they each get their own circuit breaker and retry budget. This prevents a failure in one domain from taking down your entire integration.

// Registered on the builder that AddGrpcClient<T>() returns
services.AddGrpcClient<Orders.OrdersClient>(o => o.Address = new Uri("https://orders.internal:5001"))
    .AddGrpcResilience();

// Default behavior:
// ScopeBy = static m => m.ServiceName;

You can tighten this to per-method scoping if you have a specific RPC that is notoriously flaky and shouldn't affect other methods in the same service.

2. Idempotency and the "Single Shot"

In gRPC, calls are considered repeatable by default. While this is usually true for reads, it is dangerous for mutations (like ChargeCard).

You can define which methods are repeatable in the options, or use GrpcResilience.SingleShot() for call-site overrides. This creates an ambient scope that disables retries for any call made within that block, regardless of the policy.

// 1. Policy-level: mark a specific method as non-repeatable
services.AddGrpcClient<Orders.OrdersClient>(o => o.Address = new Uri("https://orders.internal:5001"))
    .AddGrpcResilience(options => {
        options.RepeatableWhen = static m => m.Name != "ChargeCard";
    });

// 2. Call-site: force a single attempt
using (GrpcResilience.SingleShot())
{
    await client.ChargeCardAsync(request); // Will not be retried
}

3. The Math of Wire Deadlines

One of the most critical parts of the gRPC integration is how NResilience handles grpc-timeout.

The library calculates the wire deadline as: min(AttemptTimeout, remaining overall deadline) + DeadlineSlack

The DeadlineSlack (defaulting to 50ms) is a deliberate design choice. By giving the wire deadline a small buffer, NResilience ensures that its own internal timer expires first. This allows the library to throw an AttemptTimeoutException and correctly track the attempt as a failure, rather than receiving a generic StatusCode.DeadlineExceeded from the gRPC stack. It ensures the library, not the transport, remains the authoritative source of truth.

4. Handling Server Streams

Retrying a stream is tricky. NResilience supports server streaming with a specific rule: retries are only attempted until the first message is received.

Once the consumer has received the first element of the IAsyncEnumerable, the call is considered "started." Any failure after that point is propagated directly to the consumer. This prevents the library from accidentally triggering a full stream restart, which would likely lead to duplicate data or missing messages. For a stream, the wire deadline is the whole call's remaining budget rather than one attempt's ceiling. Client-streaming and duplex calls pass through untouched - wrapping one means re-enumerating something the failed attempt already partially consumed, which is not a resilience feature.

5. Why No Blocking Calls?

You will notice that BlockingUnaryCall is not supported and will throw a NotSupportedException.

NResilience is built as an async-first library to maximize throughput and minimize allocations. A blocking call that silently ran without the policy would be the one call in the client with no retry, no breaker, and no deadline - a worse outcome than a visible failure at the one call site that uses it. The same reasoning is why a synchronous Send throws in the HTTP handler. Use the generated client's Async overloads.

Trade-offs and Limitations

The per-service scoping adds a small amount of overhead to the interceptor to resolve the service name for every call. While negligible for most, in extremely high-throughput environments, this is the cost of preventing a total client brownout.

The "first message" rule for streams means that if your stream is designed to be resumable (using offsets or tokens), NResilience's built-in retry won't help you after the first message. You would need to implement resumption logic at the application level.

Conclusion

gRPC resilience isn't about wrapping a call in a loop; it's about managing the contract between the interceptor and the wire. By scoping breakers to services and carefully timing the grpc-timeout, NResilience provides protection that respects the semantics of RPCs.

The Deadlines documentation details the wire format and DeadlineSlack. For the engine underneath both the HTTP and gRPC integrations, One loop, one allocation explains the flat executor and how it keeps a single async frame regardless of policy complexity.

The Problem: The "One Client" Fallacy

When you build an HttpClient with resilience, it is easy to think of it as a single pipe. But in a real app, that client often calls multiple different hosts - a payment gateway, a user profile service, and a legacy SOAP API.

If you use a single global circuit breaker, a brownout at the payment gateway trips the breaker for everyone. Your user profile service is healthy, but your app stops calling it because the "client" is broken. This is a blast-radius problem.

Then there is the wire problem. Retrying a GET is safe. Retrying a POST that creates an order can be catastrophic if the server processed the first request but the network failed on the way back. You need a way to signal to the server that "this is a retry of request X" so it doesn't charge the customer twice.

Finally, in a microservice chain, if Service A retries Service B, and Service B retries Service C, a single failure at the bottom can trigger an exponential explosion of requests. This is a retry storm, and it can turn a minor glitch into a full-site outage.

The Solution: HTTP-Aware Resilience

NResilience doesn't treat HTTP as just another callback. It implements specific wire conventions and scoping rules to handle these distributed systems problems.

1. Per-Host State

To solve the blast-radius problem, NResilience supports per-host breakers and budgets. Instead of one breaker per client, you can have one breaker per unique host.

When you configure the client, you can set MaximumHosts to bound the memory used by the host registry. If you call a new host, the library creates a new breaker for it. If the registry hits the limit, it evicts the least recently used host.

// In practice you get the handler through HttpResilience.CreateClient() or AddResilience()
// on a client builder - both of which accept HttpResilienceOptions.
var options = new HttpResilienceOptions
{
    MaximumHosts = 64 // Track up to 64 unique hosts independently
};

using var client = HttpResilience.CreateClient(options: options);

// The breaker for api.payments.com can be Open
// while the breaker for api.users.com remains Closed.

This ensures that a failure in one dependency does not starve your other dependencies. You can inspect the state of these breakers via handler.BreakersByHost() to see exactly which host is struggling.

2. Resolving the Timeout Conflict

There is a classic conflict between HttpClient.Timeout and resilience deadlines. The transport timeout defaults to 100 seconds and covers the entire request sequence - including all attempts and backoff delays - so it silently caps any policy with a longer deadline, invisibly to the policy.

NResilience solves this with OwnTransportTimeout. When enabled (it is on by default), the library sets HttpClient.Timeout to infinite so the policy's deadline is the only active bound.

var options = new HttpResilienceOptions
{
    OwnTransportTimeout = true // HttpClient.Timeout becomes infinite (the default)
};

This makes the NResilience deadline the single source of truth. Whether it is a 30s global deadline or a 2s attempt timeout, the library controls the clock, and the transport just executes the bytes.

Which raises the obvious question about the bytes. Taking ownership of the transport timeout is the right trade - two silent timeout systems are worse than one - but it removes the only bound that ever applied to the response body. An attempt ends when the headers arrive; the body is a live stream you read afterwards, so it is covered by neither the deadline nor the attempt timeout nor the retry, and now not by a 100-second backstop either. A server that sends headers and then stops writing used to give you a call that never returned.

BoundProgress closes that, and it is on by default. The bound is the AttemptTimeout you already set, applied to the gap between two reads: a body that goes quiet for longer than that fails your read with an AttemptStalledException rather than hanging on it. Because it bounds the gap and not the total, a large download that keeps arriving is never cut off.

The stall is bounded, not retried - the retry loop was over before the body existed. BufferResponses is the other side of that trade:

var options = new HttpResilienceOptions
{
    BufferResponses = true // read the body inside the attempt, so a stalled body is retried
};

The body is now read before the attempt is judged, so both clocks cover it and a broken body is one more transient failure. The cost is memory: the whole body is held before the call returns. Leave it off for a client that downloads large files. For a JSON API whose caller was going to buffer the body a moment later anyway, it is close to free.

3. Idempotency on the Wire

Retrying a POST or PATCH is usually a bad idea. NResilience defaults to not retrying these methods. However, if your server supports idempotency keys, you can safely mark a request as repeatable.

The MarkRepeatable extension method does two things in one call: it tells this client the request may be sent again, and it stamps an idempotency key header (defaulting to Idempotency-Key) so the server can deduplicate the operation. Both halves matter - the option without a key duplicates the order, and the key without the option is never used.

using var request = new HttpRequestMessage(HttpMethod.Post, "https://api.example.com/orders")
{
    Content = JsonContent.Create(new { Item = "Book", Amount = 1 })
};

// Mark as repeatable with a unique key
request.MarkRepeatable(idempotencyKey: "order-123-abc");

// The client now allows retries for this POST request
using var response = await client.SendAsync(request, cancellationToken);

If the first attempt succeeds on the server but fails on the wire, the second attempt sends the same key. The server sees the key, recognizes the duplicate, and returns the original success response without processing the order again.

4. Preventing Retry Storms

In a chain of services, retries can multiply. If Service A retries 3 times, and it calls Service B which also retries 3 times, a single failure at the bottom results in 9 requests.

NResilience uses a marker header to make this visible. A retrying handler adds X-NResilience-Retrying: 1 to every request it can retry - on the first attempt, not just on retries, because by the time a retry goes out the amplification has already happened.

On the server side, you add the UseResilienceNestedRetry middleware. It reads the marker and publishes the flag for the rest of the request, so any outbound calls that request makes report a NestedRetry event in telemetry: the middle hop of the chain can now see an amplification it is part of. The library reports but does not intervene - what you do with the signal (typically fewer attempts and a shorter deadline on the inner layer) is your call.

// In your ASP.NET Core Program.cs
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

// Detect incoming retries and report them for this request's outbound calls
app.UseResilienceNestedRetry();

app.MapGet("/proxy", async (HttpClient client) =>
{
    // If the caller sent X-NResilience-Retrying, every outbound call this
    // request makes reports a NestedRetry event in telemetry.
    return await client.GetStringAsync("https://api.example.com/data");
});

app.Run();

Used together with a lower attempt count on the inner layer, this turns an exponential growth of requests into a linear one, protecting your downstream services from being crushed by their own resilience policies.

5. How Failures are Classified

The library uses Classifier.Http to decide if a response is a failure that should trigger a retry or trip a breaker. Not every non-200 response is a failure.

• Transient (Verdict.Transient): 500 Internal Server Error, 408 Request Timeout, or HttpRequestException. These are retried.
• Throttled (Verdict.Throttled): 429 Too Many Requests or 503 Service Unavailable. These are retried, but the library respects the Retry-After header to avoid hammering a throttled server.
• Ok (Verdict.Ok): 404 Not Found and the other 4xx statuses. These are answers, not failures; retrying will not change the outcome.

This classification ensures that your breakers only trip on actual infrastructure failure, not on user errors.

6. The Quota the Dependency Publishes

Retry-After is the dependency telling you what to do after it has already refused you. Most large APIs also tell you before: how much quota you have, how much is left, and when the window resets - on every response, not just the 429. GitHub, Stripe, Shopify and Atlassian all send it. Nothing else in .NET reads it.

The handler does, and it is on by default. It parses two shapes - the structured fields RateLimit-Policy and RateLimit, and the older X-RateLimit-* triple those fields exist to replace - keeps the numbers per host beside that host's breaker and budget, and refuses an attempt locally once what is left is inside the reserve:

var options = new HttpResilienceOptions
{
    Quota = Quota.Reserving(reserve: 0.1) // hold a tenth of the published allowance back. The default.
};

Reserve is the only number in the feature, and it is a fraction rather than a rate, because the rate is the dependency's to publish. That is the whole difference between this and a limiter: PermitsPerSecond is a number somebody had to guess and then keep in sync with a quota they do not control.

The refusal is a local one, so it gets what every local refusal gets: the long backoff curve, the reset time honored as the pushback, no evidence against the host's breaker, and no charge against the retry budget - because the call never left the process. A RejectedByQuota event carries the time until the window resets, and the exception the caller sees is RateLimitedException.

A host that publishes nothing has no quota and the feature is invisible. A host that publishes a remaining count with no quota to take a fraction of has no denominator for the reserve, so it is refused only at zero. A malformed field is ignored rather than thrown - a header the library cannot parse is not a reason to fail a call that worked.

There is one failure mode worth stating outright. A server publishing a per-account quota while you are one of fifty pods will make every pod believe it owns the whole allowance. The reserve does not fix that; only the 429 does, and the 429 still works exactly as it always has. The honest use is a single-instance client, or a generous reserve - and Quota = null reads no headers at all.

Trade-offs and Limitations

Per-host scoping adds a small amount of memory overhead to track the host registry. While MaximumHosts prevents unbounded growth, in an environment where you call thousands of unique hosts, you may need to tune this value to avoid frequent evictions.

Nested retry detection requires the marker header to be passed through. If you have a non-NResilience proxy or a legacy service in the middle of your chain that strips unknown headers, the suppression signal is lost.

The published quota is read per process, not per fleet. It can slow you down and cannot break you - it only ever refuses an attempt, never permits one the policy would have refused - but a shared per-account quota read by several instances is a number each of them will over-count. Lower the reserve, or turn it off, when that is your shape.

Conclusion

Resilience in HTTP is about the contract between the client and the wire. By moving from global state to per-host state, resolving timeout conflicts, and using marker headers for idempotency and nested retries, you can build a system that fails gracefully without amplifying the outage.

The HTTP documentation covers the wire formats and middleware in detail. For how the same rules translate to the RPC world, gRPC in depth explains per-service scoping, the grpc-timeout math, and why blocking calls are not supported.

The "Sleep" Trap

Most resilience tests suffer from the same pattern: they use Task.Delay or Thread.Sleep to simulate a dependency outage, then await for a few seconds and hope the retry logic fired.

This is the primary source of "flaky tests." On a loaded CI server, a 100ms sleep might actually take 200ms, causing a timeout test to fail randomly. Worse, testing a 30-second circuit breaker break duration means your test suite takes 30 seconds to run.

Resilience should be tested as a state machine, not a stopwatch.

Scripting the Dependency

Instead of mocking an interface and trying to maintain an internal counter of how many times a method was called, NResilience provides Sequence<T>.

A sequence is a pre-defined script of outcomes. You define exactly what the dependency does on the first call, the second call, and so on.

using NResilience.Testing;

// 1. Script the dependency: fail twice, then succeed
var calls = Sequence.For<string>()
    .Throws(new HttpRequestException("Transient error"))
    .Throws(new HttpRequestException("Transient error"))
    .Returns("Success!");

// 2. Run the policy against the script
var result = await policy.RunAsync(
    attempt => calls.NextAsync(attempt),
    cancellationToken);

Assert.Equal("Success!", result);
Assert.Equal(3, calls.CallCount);

Because Sequence<T> is a simple list of outcomes, there is no ambiguity about what the dependency returned or when it threw.

Controlling Time

To make these scripts deterministic, you must control the clock. NResilience policies accept a TimeProvider. In tests, you use FakeTimeProvider.

When a policy decides to sleep for a backoff delay, it doesn't block the thread - it asks the TimeProvider to notify it when the time has passed. By using a fake clock, you can "warp" time forward instantaneously.

var time = new FakeTimeProvider();
var policy = TestPolicy.WithClock(time); // Instant, on the test clock

var calls = Sequence.For<string>(time: time)
    .Delays(TimeSpan.FromSeconds(10)) // Simulate a slow call
    .Returns("Done");

var task = policy.RunAsync(attempt => calls.NextAsync(attempt));

// The task is currently suspended on the 10s delay.
// Warp time forward to trigger the completion.
time.Advance(TimeSpan.FromSeconds(10));

var result = await task;

This turns a 10-second test into a 10-millisecond test.

Asserting on Intent, Not Outcome

Asserting that a method returned "Success" is not enough. You need to know how it got there. Did it succeed on the first try? Did it retry twice? Did the circuit breaker trip?

The EventRecorder is a specialized listener that captures every CallEvent raised by the policy. It allows you to assert on the exact sequence of resilience events.

var events = new EventRecorder();
var policy = TestPolicy.InstantHttp with { OnEvent = events.Record };

var calls = Sequence.For<string>()
    .Throws(new HttpRequestException("Transient"))
    .Returns("OK");

await policy.RunAsync(attempt => calls.NextAsync(attempt));

// Assert on the exact order of events
Assert.Equal(
    [CallEventKind.Attempt, CallEventKind.Retrying, CallEventKind.Attempt, CallEventKind.Succeeded],
    events.Kinds
);

By asserting on events.Kinds, you verify the internal logic of the policy (e.g., "did it actually decide to retry?") rather than just the final result.

Testing the HTTP Layer

For those using the HttpClient integration, ScriptedHttpHandler provides the same deterministic scripting but at the HttpMessageHandler level. This allows you to test how your HttpClient handles different status codes without running a local web server.

var transport = new ScriptedHttpHandler()
    .Responds(HttpStatusCode.ServiceUnavailable, times: 2)
    .Responds(HttpStatusCode.OK);

using var client = HttpResilience.CreateClient(policy, innerHandler: transport);
var response = await client.GetAsync(uri);

Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.Equal(3, transport.CallCount);

Measuring the Whole Configuration

Scripts and fake clocks prove what one call does. They say nothing about what a hundred and fifty thousand of them cost the dependency, and that is the question a reviewer actually asks.

Simulate answers it. It runs your real policy - the real executor, breaker, retry budget, classifier and estimators - against a modeled dependency on a virtual clock, and reports what the run cost:

var report = Simulate.Policy(api)
    .Against(Dependency
        .Healthy(p50: TimeSpan.FromMilliseconds(20), p99: TimeSpan.FromMilliseconds(200))
        .Brownout(after: TimeSpan.FromSeconds(30), slower: 8, lasting: TimeSpan.FromMinutes(1)))
    .Under(Load.Constant(perSecond: 500))
    .For(TimeSpan.FromMinutes(5))
    .Run(seed: 42);

Assert.True(report.LoadMultiplier <= 1.2);

Five simulated minutes take about two seconds of wall clock, because nothing sleeps. LoadMultiplier is attempts that reached the dependency per call you made; Amplification is the same ratio in the worst single second, which is the one a dependency feels. Availability, Latency(quantile), BreakerOpens and TimeToRecover round out the report.

Only three things are fake: the clock, the random source and the dependency. Nothing about the decision logic is re-implemented, because a simulator that modeled the library would be worse than no simulator. And the report reproduces to the last byte from a seed - on any operating system and any processor - so Assert.True(report.LoadMultiplier <= 1.2) is a gate rather than an anecdote.

Conclusion

Flaky tests are the result of trying to test asynchronous time with real-world clocks. By combining Sequence<T> for behavior, FakeTimeProvider for time, and EventRecorder for intent, you can write resilience tests that are deterministic and execute in milliseconds. One rule to remember: pass the same TimeProvider to the policy and the script, or a scripted delay becomes a real sleep again.

Stop sleeping in your tests. Script your failures instead.

The Testing documentation covers the full NResilience.Testing package, and Simulation covers the report in detail. For injecting realistic, messy failures into your resilience pipeline, Break things on purpose introduces the Chaos engine and how to run game days in production.

The "Zombie Request" Problem

Imagine Service A calls Service B, and Service B calls Service C.

Service A has a 10-second deadline. It calls Service B. Service B takes 9 seconds to respond. Service A's deadline expires, and it hangs up. But Service B is still working - it has already sent a request to Service C, and it's waiting for an answer.

Service B is now processing a "zombie request": the caller has already given up, but Service B is still consuming threads and memory to finish a job that no one is listening to. In a cascading failure, these zombies pile up, consuming all available resources and turning a partial outage into a total collapse.

To fix this, the deadline must travel.

Propagating the Clock

The solution is to pass the deadline as a header. When Service A calls Service B, it tells it how long this side is prepared to wait - each attempt carries min(AttemptTimeout, time left on the deadline) in whole milliseconds. Service B reads it and knows exactly how much of its own budget it can spend before the answer stops mattering.

NResilience makes this automatic in ASP.NET Core with UseResilienceDeadline().

var app = builder.Build();

// Register before anything that makes an outbound call
app.UseResilienceDeadline();

app.MapGet("/", () => "Hello World");

This middleware reads the inbound deadline header and publishes it for the rest of the request. Any policy in your app that has UseAmbientDeadline = true is bounded by the tighter of two values: its own configured deadline, or the time the caller is actually still waiting.

Propagating What Matters

A deadline is not the only fact that stops at the process edge. How much a request matters stops there too, and the consequence is sharper than a zombie: Service C sheds the checkout and serves the nightly backfill, because from where it stands the two are identical.

AmbientCriticality carries a four-level label the same way the deadline carries a number - Sheddable, SheddablePlus, Critical, CriticalPlus. The same UseResilienceDeadline() pass reads it, because both values arrive in the same headers collection and a second middleware would be a second walk over it for one lookup.

// The outbound half, on the HTTP options.
var options = new HttpResilienceOptions { PropagateCriticality = true };

// The inbound half, on the policy.
var api = Resilience.Http with { UseAmbientCriticality = true };

// And the label itself, for work that is not an inbound request.
using var scope = AmbientCriticality.Begin(Criticality.Sheddable);

What the library does with a level is deliberately narrow. It does not shed, it does not reorder, and it does not decide what Sheddable should cost your service - that depends on which endpoints are cheap and which have a fallback, and a library cannot see either. What it does is the half your own code cannot reach, because it happens inside the executor:

• A Sheddable call is never hedged. Hedging spends capacity to buy latency, and there is no latency worth buying when nobody is waiting for the answer.
• A Sheddable retry has to leave half the retry budget behind. While the dependency is healthy the bucket is full and nothing is refused; once it starts draining, the backfill stops and what remains is there for the checkout.

Two guardrails are worth stating outright. An unlabeled call is Critical, never Sheddable - a default that sheds is a default that loses requests during the first incident after an upgrade. And CriticalPlus cannot arrive from the wire: a caller that could escalate itself would, and a level everyone claims is a level that means nothing. Inbound values clamp at Critical.

Shedding itself stays with Admit, which reads AmbientCriticality.Current like any other ambient fact. The point of the feature is that Admit has a value to read rather than a guess to make.

Detecting the Retry Storm

While deadlines prevent zombies, nested retries create storms.

If Service A retries a call to Service B three times, and Service B retries its call to Service C three times, a single failure at Service C can trigger 9 requests (3x3). If you have a chain of four services, you're looking at 81 requests for one logical operation. This is how a minor glitch becomes a self-inflicted DDoS attack.

UseResilienceNestedRetry() solves this by marking the request.

var app = builder.Build();

app.UseResilienceDeadline();
app.UseResilienceNestedRetry();

app.MapGet("/", () => "Hello World");

When Service A makes a call, its handler adds a marker (e.g., X-NResilience-Retrying: 1) to the request - on the first attempt, not just on retries, because by the time a retry goes out the amplification has already happened. Service B reads this marker with UseResilienceNestedRetry(), which publishes the flag for the rest of the request. When Service B then makes its own outbound calls, its handler fires a NestedRetry event saying: "this request is already inside a retrying client."

The library reports but does not intervene: silently dropping configured retries would be surprising and hard to debug. What you do with the signal is yours - typically reducing the inner attempt count and shortening the inner deadline, so the layer closest to the failing dependency does the retrying and the layers above it pass through. That collapses the exponential growth of retries back into a linear flow.

Mapping Failures to the Wire

When a deadline expires or a retry budget is exhausted, the library throws a specific exception. You don't want these to result in a generic 500 Internal Server Error, which tells the caller nothing.

AddResilienceExceptionHandler() maps these internal exceptions to the correct HTTP responses:

• Deadline Exceeded -> 504 Gateway Timeout
• Breaker/Budget/Limiter Refusal -> 503 Service Unavailable (with a Retry-After header)

// In Program.cs
builder.Services.AddResilienceExceptionHandler();
builder.Services.AddProblemDetails(); // Required for the exception handler

var app = builder.Build();
app.UseExceptionHandler(); // The framework middleware that runs the handler

This tells the upstream caller exactly why the request failed. A 503 tells them to back off; a 504 tells them the system is too slow. This feedback loop is what allows the entire fleet to stabilize.

Conclusion

Resilience isn't something you configure in a vacuum; it's a conversation between services. By propagating deadlines and criticality and detecting nested retries, you stop your services from working on dead requests, keep a backfill from spending the capacity a checkout needs, and prevent your retry logic from becoming a weapon.

With three lines of middleware and one service registration, you move from "hoping for the best" to a system that actively coordinates its failure modes.

One more option on that middleware is worth knowing about. RejectExpired refuses a request that arrives with less time left than the Reserve says answering costs, with a 504 and a problem document, rather than running a handler whose every outbound call will fail. It is off by default and stays off: the request may well be answerable from cache, and refusing it is a decision about your service rather than about resilience.

The HTTP Deadline documentation details header configuration and nested-retry detection. For testing these propagation rules deterministically, Tests that don't flake shows how to script failures using Sequence<T> and FakeTimeProvider.

The "Log Noise" Problem

In most systems, logging is binary: it's either on or off. When you turn on Debug or Information logging for a resilience library, you often get a wall of text: "Attempt 1 started", "Attempt 1 failed", "Retrying in 100ms", "Attempt 2 started".

In a high-throughput system, this is catastrophic. A single dependency brownout can generate millions of nearly identical log lines in seconds, filling up your disks and making it impossible to find the actual root cause in your log aggregator.

NResilience takes the opposite approach: the library decides the level, and the operator decides the filter.

Profiles: Silent by Default

Instead of relying on generic log levels, NResilience uses profiles. This allows the library to encode the "meaning" of an event into its severity.

You can choose from three profiles:

• Off: Attaches no listener, so suppressed calls cost nothing.
• Default: Healthy traffic is Trace, retried-then-successful calls are Debug, incidents are Warning.
• Verbose: Raises every traffic-proportional record to Information and leaves incident records where they are. Exists for sinks that enforce a minimum level.

The key is that in the Default profile, a healthy process writes nothing above Trace. A Warning means a breaker opened, a budget was exhausted, a callback outlived its timeout, or a nested retry was detected - the library encodes the meaning of an event into its severity, so a healthy call (even one that retried) stays out of your Information feed.

Logging is on by default for policies registered in a container, and opt-in for policies you build yourself:

// A policy registered in a container logs for you. A policy in a static field does not.
var payments = (Resilience.Http with { Name = "payments" }).WithLogging(logger: logger);

Flood Control: The Repeat Window

Even when a call is failing, you don't need to see every single retry. If a call is retrying 10 times, and every attempt fails with the same 503 Service Unavailable, 10 log lines are not 10x more useful than one.

NResilience implements flood control via a repeat-window (30 seconds by default, per policy and reason). When a guard refuses a call, the first refusal writes a Warning. Refusals within the window are counted and written as Debug records, and the count is included in the next warning: "Rejections logged quietly since the previous warning: 1483."

No records are dropped, only demoted. You still see every rejection at Debug if your filter carries it - the window keeps the Warning stream legible during a total dependency collapse.

Per-Policy Filtering

The NResilience logging system handles categories well. Every policy logs under its own unique category: NResilience.<PolicyName>.

This means you can keep the entire library at Warning level, but turn on Debug logging for just one specific, problematic dependency in your appsettings.json without a redeploy:

{
  "Logging": {
    "LogLevel": {
      "NResilience": "Warning",
      "NResilience.payments": "Debug",
      "NResilience.identity": "Information"
    }
  }
}

This allows you to "zoom in" on a specific part of your distributed system while keeping the rest of the noise filtered out.

Sampling for Scale

For ultra-high-throughput services, even "silent-when-healthy" logging can be too much during an incident. NResilience supports opt-in log sampling.

LogSampling.OneIn(n) keeps one traffic record in n while the policy is healthy, and every record for a minute after an incident - so you get a consistent stream of data when it matters, and a sampled rate the rest of the time. The counting is exact rather than random: every nth record, per record and per policy.

services.AddResilienceLogging(o => o.Sampling = LogSampling.OneIn(keepOneIn: 20));

Integration with WithLogging()

If you are using the NResilience.Extensions package, logging is often handled for you. When you register policies in a DI container, the library automatically attaches a log listener.

If you need to tune the profile or levels, use WithLogging() with a ResilienceLoggingOptions callback - it takes a logger first, and a level-override function or profile in the options.

An explicit WithLogging call wins over the automatic container-based listener, so your specific tuning takes precedence. At most one log listener attaches per policy, and the first one attached wins.

Conclusion

Logs are only useful if they are actionable. By shifting the responsibility of "level" to the library and providing tools like repeat-windows and per-policy categories, NResilience ensures that your logs are a signal, not noise. You get the information you need when things break, and total silence when they don't.

The Logging documentation covers configuring profiles and sampling. For what your logs look like when requests start crossing service boundaries, Deadlines that travel covers deadline propagation and nested retry detection.

The telemetry gap

Most resilience libraries log from the perspective of the attempt. You see a "Retry 1" log, then a "Retry 2" log, and eventually a success or a failure. This is fine for debugging a single request, but it's useless for operating a fleet.

When you're looking at a dashboard during an incident, you don't care that a specific request retried twice. You care about the aggregate: is my system performing 1.1x the expected work, or 5x? If you only have "per-attempt" metrics, you have to guess the logical call volume by deduplicating trace IDs or relying on a separate high-level span that might be missing or sampled differently.

NResilience treats the logical operation as the primary unit of account. Every call, regardless of how many attempts it takes, raises exactly one terminal event.

Anatomy of a CallEvent

The CallEvent is the entire telemetry surface of the library. Instead of a family of differently-shaped event types, everything - from a successful return to a circuit breaker refusal - is a CallEvent.

It is a readonly struct (64 bytes), passed by value to listeners. This means raising an event allocates nothing. If you don't have a listener attached, the executor suppresses the event logic entirely, so the cost is zero.

The most useful part of the event for operators is the ToString() output, which follows a strict, predictable format: [PolicyName] Kind #Attempt VerdictKind ExceptionType (duration) +delay

Here is what that looks like for a call that retried once and succeeded:

[Api] Attempt #1 Transient HttpRequestException (102ms)
[Api] Retrying #1 Transient (102ms) +200ms
[Api] Attempt #2 Ok (45ms)
[Api] Succeeded #2 Ok (347ms)

In this sequence, you see two attempts, one retry, and one terminal event (Succeeded). The duration in the terminal event is the total wall-clock time the caller waited.

Listening to the executor

You can hook into these events using the OnEvent property on your policy.

var apiPolicy = Resilience.Http with
{
    OnEvent = e => 
    {
        if (e.IsTerminal)
        {
            Console.WriteLine($"Call ended: {e.Kind} in {e.Duration.TotalMilliseconds}ms");
        }
    }
};

To add a listener rather than replace one, use WithListener:

var counted = apiPolicy.WithListener(e => Metrics.Record(e.Kind));

That distinction matters more than it looks. Assigning OnEvent in a with expression replaces whatever was there - and a policy registered in a container already has telemetry and logging attached, so a with { OnEvent = mine } silently turns both off. WithListener adds to the chain, which is exactly what WithTelemetry() and WithLogging() do to each other. Listeners run in the order they were added.

Because the event is a struct, you can leave these listeners attached in production without worrying about GC pressure. The library only populates the Result property (which boxes the return value) if a listener is actually present, ensuring that the "happy path" remains allocation-free.

Automatic telemetry with WithTelemetry()

While custom listeners are great for logging, you usually want your metrics in Prometheus or Grafana. The NResilience.Extensions package provides WithTelemetry(), which attaches a pre-configured listener that pipes events into System.Diagnostics.Metrics and System.Diagnostics.Activity. A policy registered in a container is instrumented for you; a policy in a static field needs this call.

using NResilience.Extensions;

var policy = Resilience.Http
    .WithTelemetry();

// In your DI setup
builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m.AddMeter("NResilience"))
    .WithTracing(t => t.AddSource("NResilience"));

In an HTTP pipeline, AddResilience() on the client builder attaches telemetry as part of registration and starts a single logical span for the entire operation. All subsequent attempt events, retry delays, and breaker transitions are added as events to that one span. This gives you a clean, single-entry trace for every logical request.

The most important number: The Retry Fraction

The goal of this architecture is to make one specific number easy to calculate: the retry fraction.

In your metrics provider, you'll see two primary counters:

• nresilience.calls: Logical operations (one per call)
• nresilience.attempts: Wire-level attempts

The formula is simple: nresilience.attempts / nresilience.calls.

If this number is 1.0, you have no retries. If it's 1.1, you're paying a 10% tax in extra load to hide transient failures - usually an acceptable trade. If it jumps to 3.0 or 5.0, you are in a retry storm. Your clients are amplifying the load on a struggling dependency, likely making the outage worse.

By splitting these counters, NResilience turns a "feeling" about system load into a hard, observable metric.

Monitoring the Tail

Telemetry isn't just about counting failures; it's about seeing the system adapt. NResilience exposes its internal measurements through the same event system.

If you use hedging, you'll see nresilience.hedges and nresilience.hedge.threshold. The threshold is a histogram of the latency quantile that triggered the hedge.

Watching this move during an incident is how you tell the difference between a brownout and a tail:

• Tail: The threshold stays stable, but a few requests are very slow. Hedges win often.
• Brownout: The threshold starts climbing. The dependency is slowing down for everyone. Hedges start failing or taking as long as the original call.

Similarly, if you use measured bounds, the nresilience.attempt.ceiling and nresilience.backoff.base histograms tell you exactly when the library tightened a constant: each is recorded whenever the measured value moves, so a gap between the histogram and the constant you wrote is a measurement telling you your guess was off.

Four newer kinds round out the surface, each following the same one-terminal-event rule:

• SaturationDetected - raised once at the onset of a thread-pool queue episode, so a count of these is a count of local incidents, not a per-call rate.
• RejectedByQuota - the HTTP handler refused one attempt because the dependency's published allowance is spent. It refuses an attempt rather than the call, so it is neither terminal nor an IsRejection.
• StreamResumed - a checkpointed stream or stalled HTTP body picked itself back up; AttemptNumber is the restart.
• Draining - the process is shutting down and the call stopped with the failure it already had instead of retrying. It is terminal, and it is a dependency failure, not a library refusal.

The cost of observation

The core philosophy of NResilience is that you shouldn't pay for what you don't use.

If OnEvent is null, the executor performs a single null check before every event. It does not create CallEvent objects, it does not box results, and it does not allocate strings.

When you attach WithTelemetry(), the cost is a few bytes for the delegate and the overhead of the System.Diagnostics calls. Because we use readonly structs for the events, we avoid the common trap of allocating a new "Event" object for every single attempt in a high-throughput system.

Conclusion

Telemetry should be an asset, not a bottleneck. By centering the library around a single, allocation-free CallEvent and distinguishing between calls and attempts, NResilience gives you the data needed to run a fleet safely. You can see exactly what the executor is deciding, why it's deciding it, and exactly how much extra load your resilience is costing you.

The Telemetry documentation lists every instrument and tag. For the other half of the observability story, Logs that don't flood covers how the library keeps log noise under control during incidents.

The extension surface

Most resilience needs are covered by the built-in presets. You want a 30 s deadline, exponential backoff, and a circuit breaker - you use Resilience.Http and you're done. But real-world dependencies occasionally have requirements that don't fit a standard curve or a simple "is this a 503?" classifier.

Instead of opening up the executor to arbitrary "strategy" plugins - which would introduce virtual calls and layered overhead - NResilience exposes specific hooks. These are designed to be the minimum viable surface needed to handle 99% of custom cases while keeping the core loop flat.

The surface consists of:

• Admit: A guard that decides if an attempt should even start.
• BeforeAttempt: A setup hook for preparing the call.
• Backoff.Custom: A way to calculate your own retry delays.
• OnEvent: A listener for every stage of a call, from attempts to the terminal outcome.

Admit: The classified guard

The Admit hook is a Func<NextAttempt, Task<Verdict>>. It runs once per attempt, inside the same classified region as the call itself.

The key distinction here is that Admit returns a Verdict. If you return Verdict.Limited (or its alias Verdict.Refused), the executor treats the attempt as a self-imposed refusal: logged, exempt from the breaker and the retry budget, and reported in the terminal CallEvent. That exemption is deliberate - your own guard refusing a call is not evidence that the dependency is failing.

Use Admit when you have a guard whose outcome is naturally a decision rather than an exception - a distributed lock, a custom quota, anything that should refuse a call before it reaches the dependency. If throwing is more natural, the same classification works through a custom guard exception and a Classifier rule; both reach the identical code path.

There is a small performance trade-off: configuring Admit selects a separate execution path in the library to avoid paying an await tax on every single call. If you don't use it, you pay nothing. If you do, you pay one extra hoisted awaiter field in the state machine - a negligible cost for the flexibility it provides.

One thing to know if your guard answers from memory. The hook returns Task<Verdict>, not ValueTask<Verdict>, for the reason above - so a synchronous guard has to wrap its answer in a task on every attempt. Hand back Verdict.OkTask instead of Task.FromResult(Verdict.Ok):

var policy = Resilience.Default with
{
    Admit = _ => gate.Wait(0)
        ? Verdict.OkTask
        : Task.FromResult(Verdict.Limited(TimeSpan.FromMilliseconds(50))),
};

Verdict.TransientTask and Verdict.PermanentTask are the same thing for the other two constant verdicts. There is no cached task for Throttled, Limited or Refused - each takes a pushback, so there is no single value to cache, and a guard that refuses is on the slow path anyway, where the retry it causes costs far more than the task wrapping its answer.

The clearest case for it is priority-aware shedding, and the library deliberately stops one step short of doing it for you. AmbientCriticality.Current tells you how much the current call matters - a value that traveled with the request rather than one you guessed - and what Sheddable should cost your service is the part only you can answer:

var policy = Resilience.Http with
{
    Admit = _ => AmbientCriticality.Current < Criticality.Critical && Overloaded()
        ? Task.FromResult(Verdict.Refused(TimeSpan.FromSeconds(1)))
        : Verdict.OkTask,
};

The library holds back the two things Admit cannot reach - it never hedges sheddable work, and it will not spend a depleted retry budget on it - and leaves shedding to the hook. See Deadlines that travel for how the level gets there.

Admit is one slot: setting it replaces whatever was there. That is deliberate. Combining two guards needs a rule for which refusal wins, and that is a decision about your system rather than one the library should make for you. Two guards are one hook that checks both.

Backoff.Custom: Your own pacing

Standard exponential backoff is the right answer for most things, but sometimes you need a specific pacing logic - perhaps based on the specific exception thrown or a custom window of time.

Backoff.Custom takes a delegate that receives the NextAttempt context and returns a TimeSpan. This is where you implement your own pacing math.

using NResilience;

// Custom backoff that uses a linear increase for the first 3 attempts,
// then switches to a hard constant delay.
var customBackoff = Backoff.Custom(attempt => 
{
    if (attempt.Number <= 3)
    {
        return TimeSpan.FromMilliseconds(100 * attempt.Number);
    }
    
    return TimeSpan.FromSeconds(2);
});

var policy = Resilience.Default with 
{ 
    Backoff = customBackoff,
    Attempts = 5 
};

One detail: server pushback always wins. If the previous verdict carried a Retry-After value, NResilience honors it verbatim, capping it by MaximumDelay. Your custom logic is only invoked when the server is silent about when you should return.

And a custom curve does not have to start from nothing. Compute is public, so a built-in curve can be the baseline and your delegate adjusts it - which is how you add a term to exponential backoff without reimplementing it:

var baseline = Backoff.Exponential();
var curve = Backoff.Custom(next => baseline.Compute(next) + TimeSpan.FromSeconds(next.Number - 1));

BeforeAttempt: The setup hook

BeforeAttempt is a Func<NextAttempt, Task>. Unlike Admit, it doesn't return a verdict - it just runs. It is the place to build a fresh request, refresh an auth token, or reset a stateful object before the attempt begins.

Crucially, BeforeAttempt runs outside the classification region. If it throws, the exception propagates out of the call immediately. It is not retried, not logged to the attempt log, and does not raise a CallEvent. This is intentional: if your setup fails, the resilience loop cannot recover it. Use it for setup that always has to run; if your guard needs retry, backoff, or telemetry, it belongs in the callback or in Admit.

using NResilience;

var policy = Resilience.Default with
{
    BeforeAttempt = next => TokenProvider.EnsureFreshTokenAsync(next.CancellationToken)
};

// Usage: The executor runs BeforeAttempt, then the callback.
await policy.RunAsync(
    attempt => httpClient.GetAsync("https://api.example.com/data", attempt),
    cancellationToken);

If you need a guard that can be retried or reported as a failure, use Admit. If you need to ensure a prerequisite is met before the clock starts ticking on the attempt, use BeforeAttempt.

OnEvent: Telemetry without the tax

NResilience follows a "pay-for-play" telemetry model. If you don't configure a listener, the executor raises nothing and pays nothing. No events are created, and no delegates are called.

OnEvent is an Action<CallEvent>. It fires for every stage of a call: each attempt as it finishes, each retry decision, breaker transitions, and exactly one terminal event at the end. The terminal CallEvent contains the final verdict, the total number of attempts, and the cumulative delay. Every event is the same struct, so a listener can switch on Kind rather than subscribe to a family of differently-shaped types.

Because the listener is synchronous and runs on the executor's thread, you should never do I/O inside it. Log it, increment a counter, or enqueue it to a background channel. An exception thrown by the listener is swallowed; telemetry should never be able to crash the operation it is observing.

The HTTP seam

While the hooks above live on the Resilience policy, the best way to extend the library's behavior specifically for HTTP is through a DelegatingHandler.

By placing a handler in the HttpClient pipeline, you can modify requests or responses before they even hit the NResilience executor. This is the correct place for things like adding custom headers, tracing IDs, or implementing custom response-shaping that should happen before the classifier sees the result.

The Hard No: Why no "Strategy" objects?

You might wonder why NResilience doesn't allow you to implement a IResilienceStrategy interface and plug it into the loop. This is a deliberate design choice.

Layered strategy pipelines (the "onion" model) are flexible, but they are slow. Each layer adds a virtual call and a new frame to the state machine. In a high-throughput system, this adds up to significant CPU overhead and increased allocations.

NResilience replaces the pipeline with a flat loop. The executor knows exactly how to handle retries, deadlines, and breakers. By using targeted hooks like Admit and BeforeAttempt, you get the extension points you actually need without introducing the architectural bloat of a general-purpose strategy framework.

The library optimizes for the common case - the fast, allocation-free call - and provides a narrow, well-defined escape hatch for the edges.

Conclusion

Bending the library is about using the right tool for the right intent. Use Admit for guards that should trip the breaker, BeforeAttempt for required setup, Backoff.Custom for non-standard pacing, and OnEvent for lean telemetry. By keeping the extension surface small, NResilience keeps the execution path flat and the performance predictable.

The OnEvent listener is the key to understanding what the executor is doing. One event per call explains the telemetry model and how the single terminal CallEvent gives you a clean view of your system's health.

The Blast Radius Problem

In a multi-tenant application, you often use a single shared dependency (like a database or a third-party API) to serve all your customers.

If you use a single Breaker for that dependency, you have a shared blast radius. If Tenant A sends a burst of malformed requests that trigger a series of failures, they will trip the breaker. Suddenly, Tenant B - who is behaving perfectly - finds their requests being refused because the breaker is open.

This is the "noisy neighbor" problem. Tenant A's failure has effectively caused an outage for Tenant B.

Isolation via PolicyScope

To fix this, you need to scope your resilience state. You don't want one breaker; you want one breaker per tenant.

PolicyScope<TKey> allows you to define a template policy and then derive a specific instance of that policy for any given key.

// Define a template: a standard policy with a breaker
private static readonly PolicyScope<string> Tenants = new(
    Resilience.Default with { Breaker = new Breaker() });

// Use the policy for a specific tenant
await Tenants.For(tenantId).RunAsync(ct => Work(ct), cancellationToken);

When you call Tenants.For(tenantId), NResilience does something critical: it doesn't just give you the template. It creates a keyed instance of the policy. This instance has its own:

• Circuit Breaker: Tenant A's failures only trip Tenant A's breaker.
• Retry Budget: Tenant A cannot exhaust the retry tokens for Tenant B.
• Latency Estimate: A slow partition for one tenant doesn't lower the hedge threshold for everyone else.

Guarding the Memory

If you have millions of tenants, you cannot have millions of breakers in memory. An unbounded dictionary of state is a memory leak.

PolicyScope is bounded by construction - there is no unbounded mode, because every entry carries a breaker and a budget. You can set maximumKeys when you create the scope (default 1024). Eviction is approximate: keys seen since the last sweep survive the next one, so the scope can briefly exceed its cap, but no lookup ever waits on a sweep.

If a tenant is evicted and then returns, they get a fresh policy with a fresh breaker. This means the "memory" of their failure is discarded, but it also ensures your process memory stays bounded regardless of how many unique tenants you serve. Size maximumKeys above the number of tenants you expect to be active at once.

Advanced Shaping

Sometimes, "one size fits all" isn't enough. You might have a "Premium" tenant who is allowed a longer deadline or a higher retry count than "Free" tenants.

You can pass a shape function to the PolicyScope constructor to customize the policy for each key on first sight:

private static readonly PolicyScope<string> Tenants = new(
    template: Resilience.Default,
    shape: tenantId => tenantId.StartsWith("premium_")
        ? Resilience.Default with { Deadline = TimeSpan.FromSeconds(30) }
        : Resilience.Default);

Because the shaping happens once and the result is cached, this adds zero overhead to the hot path.

Summary

Resilience is about isolation. By moving from a single global policy to a keyed PolicyScope, you ensure that failures are contained. You protect your healthy tenants from your noisy ones, and you ensure that your resilience state is as granular as your architecture requires.

The built-in policies cover most cases, but real-world dependencies sometimes need custom logic. Bending the library covers the extension points - Admit, BeforeAttempt, Backoff.Custom, and OnEvent - that let you inject behavior without breaking the flat executor.

The Streaming Dilemma

Resilience is easy for a single value. You call a method, you get a result, or you get an exception. You can retry that whole cycle because the operation is atomic.

But a stream - an IAsyncEnumerable<T> - is not atomic. It's a conversation.

If you have a stream that yields 1,000 items, and the connection drops on item 500, what does "retry" even mean? Do you restart the entire stream from item 1? Do you try to resume from item 501? Unless the dependency supports a cursor or an offset, "retrying" a mid-flight failure is impossible for the library to do on your behalf - it has no way to know that offset 501 exists, let alone how to ask for it. Below is what NResilience does with the failure it can handle safely, and then the escape hatch for the one it can't guess: telling it the cursor yourself.

The "First Element" Rule

NResilience handles this by defining the boundary of an attempt as the time-to-first-element.

When you use RunAsync over a stream, the executor treats the process of obtaining the first element as the unit of work.

1. The Start: The executor calls your source and awaits the first MoveNextAsync().
2. The Protection: If the stream fails to start (e.g., a SocketException or a deadline), the executor treats it as a failed attempt. It applies the Classifier, checks the RetryBudget, and retries the entire stream start. A failed start throws from the first MoveNextAsync - the same exceptions a failed call throws, attempt log attached.
3. The Handoff: The moment the first element is successfully yielded, the attempt is marked as Ok. The executor has done its job. It now hands the remaining stream over to you.

Once that first element is in your hand, the resilience loop is closed. If the stream faults on the 10th or 100th element, that exception propagates directly to you.

Why this is the only sane way

You might ask: "Why not protect the whole stream?"

Imagine a policy with 3 retries and a 30-second deadline. If the stream yields 10 elements and then fails, and the library "retried" it, you would receive: Element 1, Element 2 ... Element 10, (Failure), Element 1, Element 2...

This is a nightmare for the consumer. You'd have to implement complex deduplication logic just to handle a retry. By limiting the protection to the start of the stream, NResilience ensures that you never receive duplicate data, and the dependency isn't hammered by repeated requests for the same data.

A fault is yours, silence is not

One thing survives the handoff, and it is worth being precise about, because "the resilience loop is closed" used to mean rather more than it should have.

If the stream faults on element 100, that is yours: the exception propagates, and the library has no honest way to retry it. But if the stream goes quiet on element 100 - connection open, server still holding it, nothing arriving - there is nothing to propagate, and there used to be nothing to stop it either. No deadline covered it, because the attempt ended at element one. You got a stream that never yielded and never threw.

AttemptTimeout now bounds the gap between two elements, and it is on by default. A pull that has been outstanding for longer than that throws an AttemptStalledException at your MoveNextAsync, carrying the bound and how many elements you had received:

await foreach (var trade in policy.RunAsync(ct => feed.SubscribeAsync(ct), token))
{
    Handle(trade); // an AttemptStalledException here means the feed went quiet, not that it broke
}

It bounds the gap rather than the total, so a stream that is slow but alive runs as long as it likes. What gets measured is how long a pending pull has been waiting on the far side, so a consumer that spends a minute processing each element is never blamed for the pause. And it is a bound, not a retry: the handoff already happened, so this ends your enumeration rather than restarting it.

That is still the "First Element" rule. It just no longer means silence is indistinguishable from success.

When you can say where it picked up

Resuming mid-stream - element 101 after a failure at 100 - needs something only you can supply: a cursor, an offset, a continuation token. When you have one, RunAsync and TryRunAsync take an overload for it, and the "First Element" rule above only ever applied to the first segment.

await foreach (var trade in policy.RunAsync(
                   (offset, ct) => feed.SubscribeAsync(fromOffset: offset, ct),
                   start: lastOffset,
                   checkpoint: static trade => trade.Offset + 1))
{
    Handle(trade);
}

A connection reset after offset 500 re-invokes the source with 501, the same way a failed start re-invokes it today - it goes through the same Attempts, the same backoff, the same classifier. What's different is that it can happen after the first element too, because now there is a way to say where "again" means. Nothing already delivered is delivered twice, because you defined both what a checkpoint is and how to read one off an element - the library never has to guess.

Two guardrails come with it. Resilience.Restarts - three by default - caps how many times one call may reconnect, separately from Attempts, so a feed that delivers one trade and then drops forever can't restart forever. And a restart is charged like a retry: it draws on the same RetryBudget, is admitted through the same Breaker, and is bounded by the deadline you set for the whole call, not a fresh window every time the feed reconnects.

The same idea covers an HTTP download you read yourself: HttpResilienceOptions.ResumeDownloads resumes a stalled body with a Range request instead of ending your read, when the server's first response made that safe to do - a strong ETag and Accept-Ranges: bytes, checked back with If-Range so a changed file is never spliced onto what you already had.

Code in Action

Using a resilient stream looks almost identical to a standard call:

var policy = Resilience.Default;

// The source is cold: each attempt re-invokes it, the way the HTTP handler
// builds a fresh request per attempt.
IAsyncEnumerable<Data> stream = policy.RunAsync(ct =>
    client.GetStreamingDataAsync(url, ct), cancellationToken);

try
{
    await foreach (var item in stream)
    {
        // Once we are here, the first element has arrived.
        // The resilience loop is closed.
        Process(item);
    }
}
catch (Exception ex)
{
    // This is either a failure to start (if 0 items were yielded)
    // or a mid-flight fault.
}

Streams without the try/catch

The example above still catches, because a stream that never starts throws from the first MoveNextAsync. If you'd rather branch than catch, TryRunAsync takes a source too, and it answers the only question a stream can be asked before you consume it: did the policy get it started?

var result = await policy.TryRunAsync(ct =>
    client.GetStreamingDataAsync(url, ct), cancellationToken);

if (!result.TryGetValue(out var stream))
{
    // The policy never got it started. Reason, Exception and the attempt log
    // are all on the result - the same failure RunAsync would have thrown.
    _logger.LogWarning("The feed never opened: {Reason}", result.Reason);
    return;
}

await foreach (var item in stream)
{
    Process(item);
}

The await ends at the first element. By the time that line returns, every attempt, every backoff and every guard has already run - which is exactly the window this post is about, expressed as a result instead of an exception. A fault at item 500 still throws from MoveNextAsync, because the result was decided before item 500 existed.

There is one rule that comes with it, and it is the only one like it in the library: a successful result owns a live enumerator. Its first element has already been pulled, so enumerate it once, or dispose it if you change your mind:

if (result.IsSuccess && !stillNeeded)
{
    await ((IAsyncDisposable)result.Value!).DisposeAsync();
}

The alternatives are worse. Reporting success before the first element would report it exactly when a streaming call actually fails, and buffering the element so the result could be safely dropped would charge every stream for the rare caller who abandons one.

Constraints: The Hedge Problem

There is one important limitation: you cannot hedge a stream.

Hedging works by starting a second copy of a call and taking whichever answer arrives first. For a value, this is easy. For a stream, it's impossible without massive buffering. If a hedge "won" on the second attempt, you'd have to discard the first attempt's progress and restart the stream, or buffer the second attempt's items until the first one was cancelled.

If you pass a hedged policy to a streaming RunAsync or TryRunAsync overload, NResilience refuses it at the call with a configuration exception rather than silently doing nothing. The same policy still runs ordinary calls.

Summary

Streaming resilience is about protecting the connection, not the data. Time-to-first-element is what gets retried by default, which keeps your data linear and duplicate-free; the gap between elements is what gets bounded, which is what actually stops a "zombie" connection from hanging your app. If you can hand the policy a checkpoint, a mid-stream fault stops being yours alone to handle - it restarts from where you left off instead. One detail worth knowing for your dashboards: what the breaker samples is time to the first element, so a stream that opens fast and dies later reads as a fast success to the breaker - a stall shows up as nresilience.stalls, and a restart as nresilience.stream_resumed.

When you have multiple tenants sharing a dependency, you need per-tenant isolation. One policy per tenant introduces PolicyScope and how to avoid the noisy neighbor problem.

The Tyranny of the Tail

If you look at a latency histogram, most of your calls are fast. But then there's the tail: those few requests that take 10x the normal time.

In a microservices architecture, the tail is a contagion. If your frontend calls five services, and each has a p99 of 1 second, your frontend's p99 is effectively the sum of those tails. One slow request at the bottom of the stack can make the entire user experience feel broken.

The traditional answer is to set a timeout. But if you time out at 1 second, you just turn a slow success into a failure. You've solved the latency, but you've broken the call.

Enter Hedging: Racing the Tail

Hedging is the a-ha moment for tail latency. Instead of waiting for a request to fail, you simply start another one.

If a call is taking longer than almost every other call to that dependency, it's probably stuck. By starting a second attempt now, you get two draws from the latency distribution. The chance that both draws land in the 99th percentile is 1 in 10,000. You've effectively traded a tiny bit of extra load for a massive reduction in p99.

The Safety Problem: Why Quantiles Matter

The biggest fear with hedging is the "Load Spike." If you set a fixed hedge delay of 100ms, and your dependency suddenly slows down so that every call takes 110ms, you've just doubled your traffic to a service that is already struggling. You've turned a brownout into an outage.

NResilience solves this by making the hedge threshold a live quantile.

When you configure Hedge.At(0.95), the library tracks the recent p95 latency. The second request only fires if the first one exceeds that p95.

Because the quantile moves with the traffic, the fraction of calls that hedge stays constant (roughly 5%) regardless of whether the dependency is responding in 10ms or 10 seconds. The load on the dependency is stable, and the protection is constant.

The Hedging Gates

To ensure hedging only happens when it's safe and useful, NResilience applies several guards:

1. The Baseline Gate: Hedging is disabled until the latency estimate has enough samples (MinimumSamples, default 20) to be reliable. A cold policy hedges nothing.
2. The Concurrency Gate: MaximumConcurrent bounds how many attempts may be in flight at once, counting the first (default 2). This prevents a single call from spawning an army of duplicates.
3. The Health Gate (SuppressAt): Hedging stops when the dependency's error rate reaches a fraction of the breaker's trip point (half, by default). Sending extra load to a failing service is a mistake.
4. The Value Gate (WinRate): Opt-in. If the second request almost never wins the race, the policy retreats and hedges less; when hedges start winning again, it adds hedging back.
5. The Criticality Gate: Opt-in, through UseAmbientCriticality. A call running at Criticality.Sheddable is never hedged, whatever the estimate says. Hedging spends capacity to buy latency, and there is no latency worth buying when nobody is waiting for the answer. See Deadlines that travel for how a level reaches the policy in the first place.

var policy = Resilience.Http with
{
    Attempts = 3,
    Hedge = Hedge.At(quantile: 0.95) with
    {
        MaximumConcurrent = 2,
        SuppressAt = 0.5 // stop hedging halfway to a breaker trip (the default)
    }
};

Watching the Threshold

Because the threshold moves, it is a number worth watching rather than a number you set once. Read it straight off the policy:

var armsAt = policy.Measured.HedgeThreshold;   // null until the estimate has MinimumSamples

That is the latency at which the library starts duplicating load, so it belongs on the same dashboard as the extra traffic it explains. In NResilience.Extensions the same figure arrives as nresilience.hedge.threshold, recorded each time a hedge fires; the property is the one that answers before a hedge has fired.

One caveat: the gates above are asked when the threshold fires, not when you read it. A non-null reading is the point at which a hedge would be considered, not a promise that one starts.

A second caveat about what the quantile is measuring. It is wall clock around your callback, so a thread pool deep enough to add 400ms to every call raises the threshold too, and hedges stop firing during exactly the incident where a second copy of the work is the wrong answer anyway. Setting Saturation = Saturation.Above(5) stops the estimate learning that - see Measured, not guessed for the mechanism.

Cost Accounting: When to Hedge

Hedging isn't free. You are spending extra CPU, memory, and network bandwidth. It's a trade-off: is a 5% increase in load worth a 90% reduction in p99?

For most "read" operations (GET requests), the answer is yes. For "write" operations (POST/PUT), you must ensure the operation is idempotent - meaning the dependency can handle the same request twice without side effects. NResilience handles this via the MarkRepeatable() flag, ensuring only safe calls are hedged.

Summary

Hedging is the most effective tool for killing the tail. By racing a duplicate request against a slow one, you route around transient glitches and zombie requests. Because NResilience uses live quantiles instead of constants, you get the benefits of a shorter tail without the risk of a retry storm.

Hedging works well for single-value calls, but streams have different constraints. Streams that fail mid-flight explains why protection is limited to the time-to-first-element and what that means for IAsyncEnumerable.

The "Magic Number" Problem

Every experienced engineer has a folder of "magic numbers" in their config files:

• PaymentApi_Timeout: 2s
• InventoryApi_RetryCount: 3
• SearchApi_BreakThreshold: 0.1

The problem is that these numbers are almost always wrong. A 2-second timeout might be perfect on Tuesday, but if the dependency's p99 latency climbs to 2.1 seconds on Wednesday, your app suddenly starts failing every single request - even though the dependency is still healthy, just slightly slower.

If you set the numbers too high, you're not protected against brownouts. If you set them too low, you're killing healthy traffic.

The Rule: Constant and Measured

NResilience solves this with a single, unifying rule: Bounds can be stated as constants or measured from the dependency, and when both exist, the tighter one wins.

The measured term never loosens what you wrote. A Deadline is pure constant - measurement never touches it, so a 10-second deadline is a 10-second deadline whatever the dependency does. What measurement tightens are the bounds that sit inside it: the per-attempt ceiling, the breaker's trip points. If the dependency suddenly slows down, the measured term kicks in and fails a call at 2 seconds, long before your "safe" 10-second constant is reached.

The Three Adaptive Pairs

The library applies this logic to the three most critical areas of resilience.

1. The Attempt Ceiling

Instead of a fixed AttemptTimeout, the library tracks the recent p95 latency of successful calls. By default, it sets the ceiling at 3x the recent p95.

If the dependency normally responds in 100ms, your attempt timeout is effectively 300ms. If it slows to 500ms, your timeout automatically shifts to 1.5s. You get the protection of a tight timeout without the fragility of a constant.

2. Adaptive Breakers

Circuit breakers usually trip on a fixed error ratio (e.g., 10%). But for a high-reliability API, a 10% error rate is a catastrophe; for a flaky legacy system, it's a Tuesday.

NResilience measures the normal error rate of the dependency. It then trips the breaker if the current rate spikes to a multiple of that baseline (e.g., 5x normal). The breaker adapts to the "noise floor" of the dependency.

3. Adaptive Concurrency (The Bulkhead)

Picking a concurrency limit (bulkhead) is a guessing game based on how many pods you have. If you have 10 pods and a dependency that handles 1,000 concurrent calls, you set a limit of 100. But what happens when you scale to 50 pods? You've just quadrupled the load on the dependency.

Limit.Adaptive discovers the limit by watching for queueing. When latency spikes while the limit is high, the limit shrinks. When latency is stable, it probes upward. The limit stays correct regardless of your pod count.

// No permit count to pick. The ceiling is the one number worth setting per dependency:
// it is what the loop cannot exceed when the measurement is wrong.
using var limiter = Limit.Adaptive(maximum: 200);

Every parameter is optional - Limit.Adaptive() on its own is a complete limiter - and the rest of the knobs (the floor, how much slower than baseline counts as queueing, how hard a congested round backs off) live on an AdaptiveLimitOptions you pass as options when you want them.

From Cold Start to Steady State

You might wonder: "If the library measures latency, what happens when the app first starts?"

A "cold" policy behaves exactly like a constants-only process. The measured terms stay invisible until they have a sufficient baseline of samples (the MinimumSamples threshold).

1. Cold Start: Only your constants (e.g., Deadline = 30s) are enforced.
2. Warming Up: The library samples successful attempts.
3. Steady State: The measured bounds kick in, tightening the ceiling around the actual behavior of the dependency.

When the Measurement Measures the Wrong Thing

Everything above rests on one assumption, and it is worth saying out loud: that the number being measured is a fact about the dependency.

It is measured with a wall clock around your callback. So it is really a fact about everything between calling the dependency and getting an answer back - and one of those things is your own thread pool.

Picture a process whose pool is starved. Work items are queuing 400ms before they reach a thread. From inside the executor that is indistinguishable from a dependency that got 400ms slower, and every measured term responds accordingly:

• The attempt ceiling rises, so a doomed attempt runs longer.
• The measured backoff base lengthens, so the retry that would have worked waits.
• The hedge threshold climbs out of reach, so hedges stop firing.

All three loosen, at the same moment, and the moment is a local incident. Meanwhile the breaker sits closed and the retry budget sits full, because both of them sample the dependency - and the dependency is fine.

That is a defect in the adaptive story rather than a missing feature, and thread-pool starvation is one of the most common causes of a .NET outage. So there is a switch for it:

var api = Resilience.Http with
{
    AttemptCeiling = AttemptCeiling.Above(multiple: 3),

    // Five times this process's own normal queue delay. Above that, the three terms above stop
    // being fed - they hold what they last learned until the queue drains.
    Saturation = Saturation.Above(multiple: 5),
};

Saturation.Above(5) is dimensionless on purpose, like every other number in this post. "Five times its own normal" survives being copied to a pod with a different core count; "200ms" does not. A Floor of 20ms sits under it, because a healthy pool queues in microseconds and five times microseconds is still microseconds - nobody wants a policy that stops learning every time the garbage collector moves one probe.

What it does is narrow, and deliberately so: it only declines to record. Nothing is refused, no bound moves, no delay is added. A policy that finds itself saturated makes the same attempts, in the same shape, with the same verdicts. The worst case is behaving exactly as it does without the switch, which is what makes it safe to turn on.

The measurement behind it is one process-wide probe: a work item stamped with a timestamp, queued into the global pool queue, measured when it runs. Four times a second, read as a couple of volatile loads, and a process that never configures this never queues one. And when the pool is deep enough that the probe itself cannot run, the probe's own overdue-ness is the reading - which is the whole trick, because anything that waited for the callback would report the microsecond it measured before the incident started.

It is off by default, and not because it could do harm. It changes what every other measured term in the policy learns, and a change of that shape earns its own deliberate rollout.

One honest failure mode, stated the way the adaptive limiter states its own: a process that is always saturated learns a high queue delay as normal and stops finding itself saturated. The baseline is relative, so all it can tell you is that this minute is unlike the last hour. It keeps a local incident from corrupting your dependency measurements. It is not a substitute for having enough threads.

Reading What It Measured

An adaptive bound you cannot see is an adaptive bound you cannot trust. Every measured term the policy holds is readable from one property:

var reading = policy.Measured;

reading.AttemptCeiling;   // what the ceiling currently measures to, before AttemptTimeout clamps it
reading.BackoffBase;      // the base delay the next transient retry would wait, after the spread clamp
reading.HedgeThreshold;   // the latency at which a hedge arms - where extra load starts
reading.QueueDelay;       // how long a work item waits for a thread in this process's pool

Each one is null until its feature is configured and its estimate is warm, which is the cold-start behaviour above, stated as a value you can check. They are readings, never configuration: the knobs that produce them are AttemptCeiling, Backoff.MeasuredBase, Hedge and Saturation, and those keep the plain nouns.

For the whole picture at once, policy.Explain() prints them beside the configuration that produces them, and labels each one cold or warm:

  Measured now:    attempt ceiling   1.97s  3x p95 over 5m, 20 samples minimum
                   backoff base      -      not configured
                   hedge threshold   -      cold - p95 over 30s, 20 samples minimum
                   pool queue delay  -      not configured

Three states, told apart: a warm reading, a term nobody configured, and a configured term that has not seen enough traffic to say anything yet. That last distinction is the one every support question turns on. "Configured" and "in effect" are different facts, and a policy that reports only the first one is a policy you are reading half of.

Measured.HedgeThreshold is the one worth putting on a dashboard first. It is the latency at which the library starts duplicating requests, so it explains the extra traffic beside it - and because it is a live quantile, watching it move during an incident tells a brownout from a tail.

Put Measured.QueueDelay next to it. It is the only reading here that describes your process rather than the dependency, and when it moves with the other two, the dependency did not get slower and this host did. The nresilience.pool.delay histogram records the same number at the onset of each episode, so a count of samples is a count of local incidents.

The estimates are private to the policy instance. The HTTP handler derives one policy per host, so each host is measured on its own and a slow host never tightens a fast one's ceiling. QueueDelay is the exception: there is one thread pool, so every policy in the process reports the same number for it.

Opting Out

Some scenarios require absolute determinism, like a strict performance benchmark. You can disable all adaptive behavior with a single switch:

var deterministicPolicy = Resilience.Default with
{
    Adaptive = false
};

Setting Adaptive = false on the policy turns off the measured attempt ceiling and every other measured term the library would supply. You are left with exactly the constants you wrote - no more, no less. Saturation is refused alongside it rather than ignored: it decides when to stop measuring, so a policy that measures nothing has nothing for it to do, and the library says so instead of picking a winner.

One boundary on the switch: it deliberately stops at the breaker. A breaker is a live object two policies may share, and a policy switch that silently muted a shared breaker would surprise whoever holds the other reference. Give the breaker's settings their own Adaptive = false if you want it constants-only too. In configuration there is no such sharing, and a single "Adaptive": false JSON key covers both.

Summary

Resilience shouldn't be a game of "guess the number." By composing constants with real-time measurements, NResilience gives you the safety of a hard ceiling and the efficiency of an adaptive one. You define the absolute limits of what you'll tolerate, and the library handles the nuance of the dependency's actual performance.

The most visible benefit of measured bounds is shorter tail latency. Race the tail explains how hedging uses live quantiles to send a duplicate request when the first one is slow, cutting p99 without the risk of a retry storm.