The core problem: LLMs don't have a type system, your code does

A REST API with a defined contract fails loudly and predictably when something's wrong, wrong status code, malformed JSON, an error you can catch. An LLM call can return something that's valid JSON, matches most of your expected shape, and is still subtly wrong: a field that's usually a number coming back as a string, an enum value that's close but not quite one of your defined options, a field silently missing because the model decided it wasn't relevant this time.

typescript

interface FraudAssessment {
  riskScore: number;
  flags: string[];
  recommendation: "approve" | "review" | "decline";
}

const response = await llmClient.assess(transaction);
const assessment: FraudAssessment = JSON.parse(response); // trusting blindly

That type annotation is a lie the moment the actual JSON doesn't match. TypeScript's type system checks your code at compile time. It has no way to verify that a string parsed from an LLM response actually conforms to the shape you've declared.

Zod: validation that matches your types instead of trusting them

typescript

import { z } from "zod";

const FraudAssessmentSchema = z.object({
  riskScore: z.number().min(0).max(1),
  flags: z.array(z.string()),
  recommendation: z.enum(["approve", "review", "decline"]),
});

type FraudAssessment = z.infer<typeof FraudAssessmentSchema>;

const parsed = JSON.parse(response);
const result = FraudAssessmentSchema.safeParse(parsed);

if (!result.success) {
  logger.error("LLM response failed validation", { errors: result.error.issues });
  return fallbackAssessment();
}

const assessment: FraudAssessment = result.data;

Deriving the TypeScript type from the Zod schema with z.infer means the compile-time type and the runtime validation can never silently drift apart, because there's only one source of truth instead of two. This single change turned "the model probably returned what we expected" into "we know exactly what we got, and if it's wrong, we know precisely how."

Fallback behavior matters more than perfect prompting

You can spend real effort tightening a prompt to reduce malformed responses, and you should. But no amount of prompt engineering gets you to zero malformed responses, because the model is fundamentally probabilistic, not deterministic. What actually matters more than prompt perfection is having a defined, tested fallback for validation failure: a conservative default decision, a retry with a stricter prompt, an escalation to human review. On a fraud detection integration, the fallback for a failed validation was always the more conservative outcome, flagging for review rather than silently approving, because the cost of that failure mode was asymmetric.

Partial validation for streaming responses

Streaming LLM output complicates this further, since you're validating an incomplete JSON structure as it arrives. For streaming cases, I validate at natural completion boundaries, a completed field, a completed array element, rather than attempting to validate a JSON fragment that isn't syntactically complete yet.

typescript

const partialSchema = FraudAssessmentSchema.partial();
const partialResult = partialSchema.safeParse(accumulatedSoFar);

Logging the raw response, always

When validation fails, I always log the raw, unparsed response alongside the validation errors. Prompt behavior drifts over time, model versions change, and having the actual raw output is what lets you distinguish "the model got worse at this task" from "our schema was too strict" when you're debugging a spike in validation failures three weeks later.

The real discipline

Treating an LLM response with the same skepticism you'd apply to unvalidated user input, not the trust you'd extend to your own internal service, is the entire lesson here. The model is powerful and useful. It is not your type system, and pretending otherwise is how a "mostly correct" response becomes a production incident.

Coverage measures execution, not correctness

A test that calls a method and asserts nothing meaningful, or asserts something trivially true, inflates coverage without verifying behavior. Coverage tells you code ran during a test. It tells you nothing about whether the test would have caught a real bug.

Arrange, Act, Assert, and why the shape matters

csharp public class LoanApprovalTests { [Fact] public void Approve_WhenPending_SetsStatusToApproved() { // Arrange var application = new LoanApplication(amount: 5000m); // Act application.Approve(approverId: Guid.NewGuid()); // Assert Assert.Equal(LoanStatus.Approved, application.Status); } }

Keeping the three phases visually distinct pays for itself the first time someone other than you has to debug a failing test six months later.

Mocking with Moq: isolate the thing you're actually testing

csharp [Fact] public async Task ProcessPayment_WhenGatewayFails_ReturnsFailureResult() { var mockGateway = new Mock(); mockGateway.Setup(g => g.ChargeAsync(It.IsAny())) .ThrowsAsync(new PaymentGatewayException());

var service = new PaymentService(mockGateway.Object);
var result = await service.ProcessAsync(100m);

Assert.False(result.IsSuccess);

}

Mocking the gateway verifies your service's error handling specifically, without a real network call. The failure mode I've seen teams fall into is over-mocking, replacing so much of the real code path that the test ends up verifying the mock's configuration instead of your code's actual behavior.

Testing behavior, not implementation details

Tests coupled to internal implementation details break every time you refactor, even when behavior hasn't changed, which trains people to see failing tests as noise. Test through the public contract, what the class promises callers, not its private internals.

The tests that actually catch bugs are the edge cases

The happy path is usually the easiest part to get right. The genuinely valuable tests check a null input, an empty collection, a boundary value, a concurrent call. The bug that would have shipped on the loan portal wasn't in the standard approval flow, it was in approving an application that had already been rejected.

What I actually optimize for

Not a coverage percentage. Whether the test suite gives genuine confidence to change code without manually re-verifying every affected path. A smaller number of tests that check real behavior gives more of that confidence than a large number that mostly verify the code ran.

REST: still the right default for most things

For a public-facing or partner-integrated API, REST remains my default. It's understood by nearly every developer without explanation, works naturally with HTTP caching and load balancers, and its constraints, resources and verbs, are simple enough that a new consumer can guess their way through your API from the URL alone.

csharp

app.MapGet("/api/policies/{id}", async (Guid id, IPolicyService service) =>
{
    var policy = await service.GetByIdAsync(id);
    return policy is not null ? Results.Ok(policy) : Results.NotFound();
});

Where REST strains is exactly where its resource-shaped model doesn't match what the consumer needs: a mobile client wanting three fields from what would be five different REST resources.

GraphQL: for consumers with genuinely different data needs

I reach for GraphQL when I have multiple consumers with meaningfully different shapes of the same data, and REST would otherwise force heavy over-fetching or an ever-growing set of specialized endpoints.

graphql

query { customer(id: "123") { name policies { status premium } } }

The cost is real: you lose simple URL-based caching, rate limiting has to reason about query complexity, and N+1 doesn't disappear, it just moves into resolvers, needing DataLoader-style batching.

gRPC: for service-to-service calls where performance actually matters

This one gets underused, specifically because it's less familiar. For internal, high-throughput paths where you control both ends, gRPC's binary protocol buffers and HTTP/2 multiplexing give real latency advantages over JSON-over-HTTP.

protobuf

service PaymentService {
  rpc ProcessPayment (PaymentRequest) returns (PaymentResponse);
}

On a fraud detection integration where every millisecond mattered, moving that internal call to gRPC produced a real, noticeable improvement. I wouldn't make that same call for a public-facing API, where gRPC's tooling friction outweighs the benefit for most consumers.

The question I actually ask

Not "which is best," because that has no stable answer. I ask: who is calling this, do I control both ends, and does this specific interaction have a performance or fetching-shape problem the default, REST, doesn't already solve well enough.

Mixing protocols is normal, not messy

A platform I worked on used REST for partner integrations, GraphQL for the mobile app's flexible needs, and gRPC for internal transaction-processing calls. That's matching each interaction to what it needs, not architectural indecision.

The real takeaway

The protocol debate gets treated like a religious argument online. In practice it's an engineering decision: understand your consumers, understand your constraints, pick the tool that fits. None of the three are wrong. They're answers to different questions.

The core shift: models that know their own rules

Most codebases I've inherited have anemic domain models, classes that are really just data bags, with all the actual logic living in a separate service layer.

csharp

public class LoanApplication
{
    public decimal Amount { get; set; }
    public LoanStatus Status { get; set; }
}

DDD pushes validation into the model itself, so it's impossible to represent an invalid state in the first place:

csharp

public class LoanApplication
{
    public decimal Amount { get; private set; }
    public LoanStatus Status { get; private set; }

    public void Approve(Guid approverId)
    {
        if (Status != LoanStatus.Pending)
            throw new DomainException("Only pending applications can be approved.");
        Status = LoanStatus.Approved;
    }
}

The private setter matters more than it looks. Nobody outside this class can silently flip the status without going through Approve() and its validation. The rule lives in exactly one place instead of being re-implemented, inconsistently, everywhere the status gets changed.

Ubiquitous language

One underrated part of DDD is insisting code, conversation, and documentation all use the same vocabulary. On the loan portal, stakeholders said "disbursement," not "payout." Once the code matched, translation errors between what was asked for and what got built dropped, because there was no mental mapping step in between.

Bounded contexts

A "member" in the loan approval context has a credit history. A "member" in the notifications context just has an email address. Forcing one shared class to serve both either bloats it with irrelevant fields or creates hidden coupling. Bounded contexts give each part of the system its own accurate model, with an explicit translation layer where they need to talk.

Aggregates

An aggregate is a cluster of objects that must stay consistent as a unit, with one entity as the entry point for change. A LoanApplication and its ApprovalHistory entries formed one aggregate: you can't add a history entry without going through the application itself, which is what keeps the two from drifting out of sync.

csharp

public class LoanApplication
{
    private readonly List<ApprovalHistoryEntry> _history = new();
    public IReadOnlyList<ApprovalHistoryEntry> History => _history;

    public void Approve(Guid approverId)
    {
        Status = LoanStatus.Approved;
        _history.Add(new ApprovalHistoryEntry(approverId, DateTime.UtcNow));
    }
}

Getting aggregate boundaries right is genuinely hard, and I've drawn them wrong more than once. Too large, you get contention. Too small, you lose the consistency guarantee that was the whole point.

Where I don't apply this

A simple CRUD screen for product categories doesn't need rich domain modeling. DDD earns its complexity where business rules are genuinely complex and getting them wrong has real consequences, not uniformly across a codebase.

The real payoff

The loan portal's approval logic became something you could read and trust, because the rules lived in one place, in the business's own language, impossible to bypass accidentally. That's the value proposition: not a purity test, a way of making complex business logic legible and safe to change.

Pods aren't your application, they're disposable

The mental shift that matters most: a pod running your .NET service is not a stable, long-lived server. It can be killed and rescheduled at any time, for reasons that have nothing to do with your code, a node failing, a deployment rolling out, the scheduler rebalancing load. Your application needs to assume it will be terminated and restarted regularly, and behave correctly through that.

This changes how you think about state. Anything held only in memory on a single pod, an in-process cache, a background job's progress, session state, disappears the moment that pod is replaced. Externalize what needs to survive: Redis for cache, a database for job state, a distributed session store if you genuinely need server-side sessions.

Health checks aren't optional, they're how Kubernetes knows what to do with you

csharp

builder.Services.AddHealthChecks()
    .AddSqlServer(connectionString, name: "database")
    .AddCheck<DownstreamServiceHealthCheck>("payment-gateway");

app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = check => check.Tags.Contains("ready") });

Liveness and readiness probes serve genuinely different purposes, and conflating them causes real problems. Liveness answers "should this pod be restarted because it's stuck." Readiness answers "should this pod currently receive traffic." A pod that's alive but not ready, say, still warming up a cache, or temporarily unable to reach its database, should be pulled from the load balancer without being killed and restarted. Mixing the two up means Kubernetes either restarts a healthy pod that's momentarily busy, or keeps sending traffic to one that genuinely can't serve it.

Resource requests and limits change how your app actually behaves

yaml

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Requests are what Kubernetes guarantees your pod when scheduling it. Limits are the hard ceiling. Set memory limits too low for a .NET service and you'll see the runtime get OOMKilled during garbage collection spikes that would have been completely normal with more headroom. .NET's server garbage collector, in particular, can be memory-hungry under load, and I've had to explicitly tune DOTNET_gcServer and memory limits together to avoid pods dying under perfectly normal traffic.

Graceful shutdown is your responsibility, not the platform's

When Kubernetes terminates a pod, it sends a SIGTERM and gives a grace period, typically 30 seconds by default, before force-killing it. ASP.NET Core listens for this and gives you IHostApplicationLifetime to hook into it:

csharp

app.Lifetime.ApplicationStopping.Register(() =>
{
    // stop accepting new work, finish in-flight requests, close connections cleanly
});

Ignore this and in-flight requests get cut off mid-execution when the grace period expires, exactly the kind of failure that's hard to reproduce locally and painful to debug from logs alone.

ConfigMaps and Secrets, and knowing which is which

Configuration that changes per environment but isn't sensitive belongs in a ConfigMap. Credentials, connection strings, API keys belong in a Secret, and even then, a Kubernetes Secret is base64-encoded, not encrypted, by default. On systems handling financial data, I've paired Kubernetes Secrets with an external secrets manager rather than trusting base64 encoding as if it were real protection.

csharp

builder.Configuration.AddEnvironmentVariables();

Both ConfigMaps and Secrets typically surface as environment variables or mounted files, and IConfiguration picks them up the same way it picks up anything else, which keeps the application code itself blissfully unaware of where its configuration actually came from.

Namespaces and resource quotas: know your neighborhood

On a shared cluster, your service isn't running in isolation. Resource quotas at the namespace level prevent one team's runaway deployment from starving every other service on the same cluster. I've been on both sides of this, the team that got starved, and later, the team that learned to set sane resource requests so we wouldn't be that neighbor.

The real value of understanding this layer

None of this makes you a platform engineer. What it does is close the gap between "my service works on my machine" and "my service behaves correctly when Kubernetes inevitably kills, reschedules, or scales it without asking permission." That gap is exactly where a lot of production incidents live, and it's a gap application developers can close themselves, without needing to own the cluster.

What idempotency actually means

An idempotent operation produces the same result no matter how many times it's executed. GET requests are naturally idempotent, you can call them a thousand times and nothing changes. POST requests that create a resource are not, by default, and a payment endpoint is almost always a POST. That mismatch is exactly where duplicate charges come from.

Idempotency keys: the standard pattern

csharp

app.MapPost("/api/transfers", async (
    TransferRequest request,
    [FromHeader(Name = "Idempotency-Key")] string idempotencyKey,
    ITransferService service) =>
{
    var existing = await service.GetByIdempotencyKeyAsync(idempotencyKey);
    if (existing is not null)
        return Results.Ok(existing);

    var result = await service.ProcessAsync(request, idempotencyKey);
    return Results.Ok(result);
});

The client generates a unique key per logical operation, usually a GUID, and sends it with every attempt at that same operation, including retries. The server checks whether it's already seen that key before doing anything else. If it has, it returns the original result instead of processing the request again. The client doesn't need to know whether their first attempt actually succeeded before timing out. They just retry safely with the same key.

Where the naive version breaks

A common mistake is checking for the idempotency key and processing the request as two separate, non-atomic steps. Under concurrent requests, arriving within milliseconds of each other, both can pass the check before either has recorded the key, and both proceed to charge the customer. The check-and-record needs to happen atomically, typically with a unique database constraint on the idempotency key that causes the second concurrent insert to fail cleanly rather than silently succeed twice.

csharp

public async Task<TransferResult> ProcessAsync(TransferRequest request, string idempotencyKey)
{
    try
    {
        await _repository.InsertIdempotencyRecordAsync(idempotencyKey);
    }
    catch (UniqueConstraintViolationException)
    {
        return await _repository.GetResultByIdempotencyKeyAsync(idempotencyKey);
    }

    return await ExecuteTransferAsync(request);
}

The unique constraint at the database level is what actually makes this safe under concurrency, not the application-level check that runs before it.

What happens while the first request is still processing

There's a subtle window where a retry can arrive while the original request is still being processed, not yet finished, not yet failed. A naive implementation might not find a completed result yet and process the request again. Handling this properly means recording the idempotency key as "in progress" before starting the actual work, and having a retry that hits an in-progress record wait for or poll the original result, rather than starting a second execution.

Idempotency keys need to expire, deliberately

Keeping every idempotency key forever isn't practical, and it's also not usually necessary. A reasonable retention window, often 24 hours, covers realistic retry scenarios like a client retrying after a timeout or a brief outage. After that window, a repeated key can be treated as a genuinely new request. Where I draw that line depends on the specific failure modes I'm protecting against, not a default I copy from project to project.

This applies beyond the payment call itself

It's tempting to think idempotency is only about the initial charge, but the same problem shows up anywhere a retry could duplicate an effect: sending a confirmation email twice, reserving inventory twice, triggering a webhook twice. Any operation triggered by an event that might be delivered more than once needs to ask the same question a payment endpoint does: what happens if this runs twice?

The real takeaway

Idempotency isn't a defensive afterthought bolted onto a payment API. It's a basic acknowledgment that networks are unreliable, clients retry, and a system handling real money has to behave correctly under those conditions by design, not by luck. Every payment-adjacent endpoint I build starts with this question before a single line of business logic gets written.

The promise you're actually making

Every API response shape, every field, every status code behavior becomes a contract the instant someone builds against it. Change it without warning and you've broken that contract, regardless of how justified the change was on your end. Versioning is how you change your mind without breaking that promise.

URL versioning: the one I default to

csharp

app.MapGroup("/api/v1/policies").MapPolicyEndpointsV1();
app.MapGroup("/api/v2/policies").MapPolicyEndpointsV2();

Header-based versioning is arguably more "correct" from a REST purist's standpoint, keeping the URL stable while the version travels in an Accept header. In practice, on systems where third parties integrate against you, URL versioning wins because it's impossible to get wrong by accident. A consumer can see the version in the URL, log it, debug against it, and never wonder if they forgot to set a header correctly.

Additive changes don't need a new version at all

Not every change is breaking. Adding a new optional field to a response, adding a new endpoint, adding a new optional query parameter, none of these break an existing consumer who ignores fields they don't recognize. I reserve new versions specifically for breaking changes: removing a field, changing a field's type, changing required behavior. Bumping a version for every minor addition trains consumers to distrust your versioning scheme entirely, because now they can't tell which version bumps actually matter.

Deprecation needs a timeline, not just a warning

csharp

app.MapGroup("/api/v1/policies")
   .MapPolicyEndpointsV1()
   .WithMetadata(new ApiDeprecatedAttribute(sunset: "2026-12-01"));

Announcing a version is deprecated without a concrete sunset date is close to meaningless. Consumers deprioritize open-ended warnings behind whatever's actually on fire this week. A specific date, communicated well in advance and included in response headers, gives them something to actually plan around.

Sunset: Tue, 01 Dec 2026 00:00:00 GMT
Deprecation: true
Link: <https://api.example.com/docs/migration-v1-v2>; rel="deprecation"

Running two versions in parallel is real operational cost

This is the part that gets underestimated. Maintaining v1 and v2 side by side means double the test surface, double the documentation, and double the places a bug can hide. I don't take on that cost casually. Before shipping a breaking change, I ask whether it's genuinely necessary or whether it could be delivered as an additive change instead. Most of the time, with enough thought, it can.

Internal APIs get a different calculus

For APIs with a small number of known internal consumers, I've sometimes skipped formal versioning entirely in favor of coordinated deployments, where the API and its consumers ship together and breaking changes are a conversation, not a migration. That only works when you actually control both sides. The moment an API has consumers outside your direct coordination, formal versioning stops being optional.

The real discipline

Versioning done well is invisible to consumers, they simply never get surprised. Versioning done poorly announces itself loudly, usually through a support ticket at 2am from someone whose integration just broke. The difference isn't the versioning scheme you pick. It's whether you treat every public-facing change as a promise you're responsible for keeping.

The developers who grow fastest aren't usually the most naturally talented ones. They're the ones who ask why something works, not just how to make it work, and who have someone around willing to answer honestly instead of just fixing it for them. I learned this the hard way as a mentor too. Early on, when someone brought me a bug, I'd just look at the code and tell them the fix. It felt helpful in the moment. It taught them almost nothing. Now I ask what they've already tried and what they expected to happen instead. More often than not, they find the bug themselves halfway through explaining it to me, and they walk away having actually practiced debugging instead of watching me do it for them. If you're mentoring someone, that shift alone is worth making. If you're the one being mentored, ask for that kind of space instead of just the answer.

AI has changed how I work, and how almost every developer I talk to works. I'm not going to pretend otherwise. Writing a first version of something, generating tests, working through a rough draft of an implementation, all of that moves faster than it did a few years ago. But the developers who are actually thriving with these tools aren't the ones who've handed over their thinking. They're the ones who've gotten sharper at the parts AI still can't do for them. Knowing which approach actually fits the problem in front of you. Catching when a generated solution looks right but will quietly break under real load or edge cases. Being able to explain a tradeoff clearly to someone who isn't technical. AI has made it faster to produce code. It hasn't made it faster to develop judgment, and judgment is still what separates someone who ships working software from someone who ships software that survives contact with real users. If you're early in your career, that's the skill worth building deliberately, because it's the one that doesn't get replaced by a better model next year.

I get asked a lot whether .NET is still worth learning, usually by developers weighing it against something newer. I always give the same answer. The platforms that last aren't the ones that chase trends, they're the ones that keep solving real problems as those problems change. .NET has done that for two decades now, and it's still doing it. Minimal APIs made it easier to get started without giving up the maturity the platform is known for. Performance work has made it genuinely competitive in places it wasn't before. The newer tooling around cloud-native development is aimed at the unglamorous, everyday friction of running services locally and in production, which is exactly the kind of problem that actually matters once you're past the tutorial stage. A platform that keeps solving real problems tends to keep the developers who build on it.

If there's one thing I would want a developer earlier in their career to take from all of this, it's this: the mechanical part of writing code is getting easier and faster for everyone, so it stops being what makes you valuable. What doesn't get automated is knowing why you're building something a certain way, being able to teach that reasoning to someone else, and being honest about tradeoffs instead of hiding behind a tool's output. Build that, and you'll be fine no matter what changes next.

Fire-and-forget for work that shouldn't block the response

csharp

app.MapPost("/api/orders", async (Order order, IOrderService service) =>
{
    await service.CreateAsync(order);
    BackgroundJob.Enqueue<INotificationService>(n => n.SendOrderConfirmationAsync(order.Id));
    return Results.Ok(order);
});

The API returns immediately once the order is actually saved. The confirmation email goes out on its own timeline, on its own thread, without the customer waiting on an email provider's response time before they see their order confirmed.

Delayed jobs for anything that needs to happen later

csharp

BackgroundJob.Schedule<ISubscriptionService>(
    s => s.SendRenewalReminderAsync(subscriptionId),
    TimeSpan.FromDays(subscription.DaysUntilExpiry - 7));

This replaces what used to require a separate scheduled task or a homegrown polling mechanism, with a single line that Hangfire persists and guarantees will run, even across application restarts.

Recurring jobs, declared once

csharp

RecurringJob.AddOrUpdate<IReportService>(
    "daily-transaction-summary",
    r => r.GenerateDailySummaryAsync(),
    Cron.Daily);

I've replaced more than one fragile cron-job-plus-console-app setup with this pattern. It's persisted, visible in the dashboard, and doesn't depend on a separate deployment or a server-level scheduled task that nobody remembers exists until it silently stops running.

Retries need the same thought as any other resilience concern

By default, Hangfire retries a failed job automatically, which is useful but not automatically safe. A job that partially succeeded before failing, say, it charged a customer but failed to send the confirmation, will retry the entire job, including the part that already succeeded, unless you've made the job idempotent.

csharp

public async Task ProcessPaymentAsync(Guid paymentId)
{
    if (await _repository.IsAlreadyProcessedAsync(paymentId)) return;

    await _paymentGateway.ChargeAsync(paymentId);
    await _repository.MarkProcessedAsync(paymentId);
}

This is the same idempotency discipline that matters for message consumers in an event-driven system, and it applies here for exactly the same reason: at-least-once execution is the guarantee you actually get, not exactly-once.

The dashboard is more useful than it looks at first

Hangfire's built-in dashboard shows queued, processing, succeeded, and failed jobs, and it's saved me real debugging time on more than one occasion, being able to see exactly which job failed, how many times, and with what exception, without digging through log files first.

csharp

app.UseHangfireDashboard("/hangfire", new DashboardOptions
{
    Authorization = new[] { new HangfireAuthorizationFilter() }
});

That authorization filter isn't optional. An unsecured Hangfire dashboard exposes internal job details and lets anyone trigger jobs manually, which is exactly as risky as it sounds.

Job priorities and queues for mixed workloads

Not every job deserves equal footing. A time-sensitive payment retry shouldn't sit behind a low-priority batch report in the same queue.

csharp

[Queue("critical")]
public async Task ProcessRefundAsync(Guid refundId) { }

[Queue("low-priority")]
public async Task GenerateMonthlyReportAsync() { }

Separate queues, processed by separate workers if needed, keep a backlog of low-priority work from delaying something time-sensitive.

Where I draw the line

Hangfire is excellent for jobs measured in seconds to minutes. For genuinely long-running batch processing, or workloads that need horizontal scaling well beyond what a single server's worker pool handles comfortably, I look at dedicated tools built for that scale instead of stretching Hangfire past what it's suited for. Knowing that boundary matters as much as knowing how to use the tool itself.

The real value

Background jobs let your API stay fast and responsive while the actual work happens on its own schedule. The part that's easy to get wrong isn't the setup, Hangfire makes that simple, it's the idempotency and monitoring discipline around it that determines whether background processing quietly does its job or becomes its own source of confusing, hard-to-reproduce bugs.

The gate mentality is the root problem

When code review is treated purely as approval-seeking, something a developer has to survive to merge their work, the incentives get twisted. Reviewers rubber-stamp things to avoid conflict, or developers game the process by picking the reviewer least likely to push back. Neither produces better code. The review stops being about the code and starts being about clearing a hurdle.

I try to frame reviews, explicitly, as a second set of eyes on a shared problem, not a checkpoint one person has to pass and another has to guard.

Every comment should carry a reason

"This is wrong" tells someone nothing they can use next time. "This will fire a separate query per item in this loop, worth wrapping it in .Include() to batch it" tells them exactly what's wrong and why, and they'll likely catch it themselves next time without needing the comment repeated.

// Less useful:
Change this.

// More useful:
This creates a new HttpClient per request, which can exhaust socket 
connections under load. Worth injecting IHttpClientFactory instead.

The extra sentence costs almost nothing to write and changes whether the feedback teaches something or just gets grudgingly applied and forgotten.

Separate "this will break" from "I would have done it differently"

Not every comment carries the same weight, and treating them all the same in tone flattens that distinction in a way that erodes trust over time. A correctness issue, a missing null check, an unhandled exception path, a genuine security gap, deserves a firm, clear comment. A stylistic preference deserves a much softer one, or sometimes no comment at all. I've seen reviewers who apply the same tone to both, and it trains people to either ignore all feedback as noise or become defensive about all of it, neither of which helps.

I try to be explicit about which category a comment falls into: "this will break under concurrent load" versus "small nit, feel free to ignore."

Ask questions instead of issuing verdicts, when you're not certain

If I don't fully understand why a decision was made, I ask rather than assume it's wrong. "What led you to this approach over X?" often surfaces a constraint I didn't know about, and it keeps the conversation collaborative instead of adversarial. Sometimes the answer reveals a genuine problem. Sometimes it reveals I was missing context. Either way, the question got there faster and with less friction than a confident, wrong correction would have.

Review the design before the syntax

Catching a missing semicolon or an inconsistent variable name matters, but it matters far less than catching a fundamentally wrong approach before it's fully built out. I try to review structure and approach first, ideally before a PR is even opened, in a quick design conversation, rather than only encountering a bad architectural decision after someone has spent two days implementing it in full.

Respond to reviews as quickly as you'd want your own reviewed

A PR that sits unreviewed for three days doesn't just delay one developer, it often blocks whatever depended on that work, and it quietly teaches people that code review is a bottleneck to route around rather than a normal part of shipping. I treat review turnaround as seriously as I treat my own deadlines, because from the other side of the review, it's exactly that.

The measure of a healthy review culture

The best signal I've found isn't review speed or comment count, it's whether people feel comfortable submitting work they're genuinely unsure about, without fear of the review turning into a personal critique. That comfort is what actually produces honest, catchable-early mistakes instead of code that's been polished defensively to survive review rather than to be correct.

Cache-aside: the pattern I reach for most

csharp

public async Task<Customer> GetCustomerAsync(Guid id)
{
    var cached = await _cache.GetStringAsync($"customer:{id}");
    if (cached is not null)
        return JsonSerializer.Deserialize<Customer>(cached);

    var customer = await _repository.GetByIdAsync(id);
    await _cache.SetStringAsync($"customer:{id}", JsonSerializer.Serialize(customer),
        new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10) });

    return customer;
}

Check the cache first, fall back to the source of truth on a miss, populate the cache for next time. It's simple, and simple is a feature when a dozen developers will touch this code over the life of a project.

Expiration is a business decision, not a technical afterthought

The single most common caching mistake I see isn't a missing cache, it's a cache with no thought put into how long the data should live. Reference data that rarely changes, product categories, country lists, can safely live in cache for hours. A customer's account balance cannot, and treating both the same way, either both cached aggressively or both cached cautiously, gets one of them wrong every time.

csharp

var expiration = category switch
{
    CacheCategory.ReferenceData => TimeSpan.FromHours(6),
    CacheCategory.UserProfile => TimeSpan.FromMinutes(15),
    CacheCategory.FinancialBalance => TimeSpan.FromSeconds(30),
    _ => TimeSpan.FromMinutes(5)
};

I ask a specific question for every cached value: what's the actual cost of this being stale for the expiration window I've chosen? For financial data, that cost is high and the window should be short or nonexistent. For a list of insurance product categories, the cost is close to zero.

Cache invalidation on write, not just expiration

Waiting for a TTL to expire is fine for data where brief staleness is acceptable. It's not fine when a user just updated their own profile and immediately sees the old version reflected back at them. Explicit invalidation on write closes that gap:

csharp

public async Task UpdateCustomerAsync(Customer customer)
{
    await _repository.SaveAsync(customer);
    await _cache.RemoveAsync($"customer:{customer.Id}");
}

This is a small addition that prevents one of the more common and more visible caching bugs: a user makes a change, refreshes, and sees their old data staring back at them.

The thundering herd problem

When a popular cache key expires, a burst of concurrent requests can all miss the cache simultaneously and all hammer the database at once to repopulate it, briefly recreating exactly the load problem the cache exists to prevent. On a high-traffic ticketing-adjacent platform, this was a real issue during popular event releases. Locking around cache population, so only one request repopulates the cache while others wait briefly for that result, solves it:

csharp

public async Task<Customer> GetCustomerAsync(Guid id)
{
    var cached = await _cache.GetStringAsync($"customer:{id}");
    if (cached is not null) return JsonSerializer.Deserialize<Customer>(cached);

    using var lockHandle = await _lockProvider.AcquireLockAsync($"lock:customer:{id}", TimeSpan.FromSeconds(5));
    // re-check cache after acquiring lock, another request may have already populated it
    cached = await _cache.GetStringAsync($"customer:{id}");
    if (cached is not null) return JsonSerializer.Deserialize<Customer>(cached);

    var customer = await _repository.GetByIdAsync(id);
    await _cache.SetStringAsync($"customer:{id}", JsonSerializer.Serialize(customer));
    return customer;
}

Distributed cache, not per-instance memory cache, in a multi-instance deployment

IMemoryCache is fast because it lives in-process, but that's also its limitation: in a load-balanced deployment with multiple instances, each instance has its own separate cache, and invalidating on one instance does nothing for the others. Redis, as a shared distributed cache, keeps every instance seeing the same data. I use IMemoryCache only for genuinely instance-local concerns, and Redis for anything that needs to be consistent across a fleet of running instances.

What I don't cache

Not everything benefits from caching. Data that changes on nearly every read, or data where correctness matters more than the milliseconds saved, sometimes just isn't worth the invalidation complexity. Caching is a tool for a specific problem, expensive or frequent reads of relatively stable data, not a default applied everywhere out of habit.

The real discipline

Redis makes caching technically easy. What actually keeps a cache trustworthy is being deliberate about expiration per data type, invalidating on write where staleness is visible to users, and thinking through concurrent access patterns before they become a production incident. The technology is the easy part.

What actually changes

In a request-response world, Service A calls Service B directly, waits for a response, and Service B has to be up and responsive for that to work. In an event-driven model, Service A publishes "OrderPlaced" and moves on. Whoever cares about that event, inventory, notifications, billing, subscribes independently, and Service A never knows or cares who's listening.

csharp

public async Task PlaceOrderAsync(Order order)
{
    await _repository.SaveAsync(order);
    await _publisher.PublishAsync(new OrderPlacedEvent(order.Id, order.CustomerId, order.Total));
}

The publishing service is now decoupled from every downstream consumer. You can add a new consumer, say, a fraud check that runs on every new order, without touching the order service at all. That's the real value: change isolation, not just async processing for its own sake.

Setting up RabbitMQ in .NET

MassTransit has become my default abstraction over raw RabbitMQ client code, because it handles a lot of the operational plumbing, retries, dead-lettering, serialization, that you'd otherwise hand-roll and get subtly wrong.

csharp

builder.Services.AddMassTransit(x =>
{
    x.AddConsumer<OrderPlacedConsumer>();
    x.UsingRabbitMq((context, cfg) =>
    {
        cfg.Host("rabbitmq://localhost");
        cfg.ConfigureEndpoints(context);
    });
});

csharp

public class OrderPlacedConsumer : IConsumer<OrderPlacedEvent>
{
    public async Task Consume(ConsumeContext<OrderPlacedEvent> context)
    {
        var order = context.Message;
        await _inventoryService.ReserveStockAsync(order.OrderId);
    }
}

The question everyone underestimates: what happens when a message fails

This is where event-driven systems separate teams that have done it before from teams doing it for the first time. A message that fails processing shouldn't just vanish, and it shouldn't infinitely retry and block the queue either. Dead-letter queues, where a message that's failed a defined number of times gets routed somewhere for inspection rather than lost or endlessly retried, are not optional in production.

csharp

cfg.ReceiveEndpoint("order-placed-queue", e =>
{
    e.UseMessageRetry(r => r.Interval(3, TimeSpan.FromSeconds(5)));
    e.ConfigureConsumer<OrderPlacedConsumer>(context);
});

I treat the dead-letter queue as something a human actually monitors, not a place messages go to be forgotten. A growing dead-letter queue is an early warning sign of a bug or a downstream outage, and it's easy to miss if nobody's watching it.

Idempotent consumers, always

Message brokers generally guarantee at-least-once delivery, not exactly-once. That means your consumer will, eventually, receive the same message twice, whether from a network blip, a retry, or a redelivery after a crash before acknowledgment. If processing that event twice reserves stock twice or charges a customer twice, that's a bug in your consumer, not a broker malfunction.

csharp

public async Task Consume(ConsumeContext<OrderPlacedEvent> context)
{
    if (await _repository.AlreadyProcessedAsync(context.MessageId)) return;

    await _inventoryService.ReserveStockAsync(context.Message.OrderId);
    await _repository.MarkProcessedAsync(context.MessageId);
}

Eventual consistency is a real tradeoff, not a footnote

The moment you go event-driven, you're accepting that different parts of your system will briefly disagree about the state of the world. Inventory might show a reservation a few hundred milliseconds before billing has processed the corresponding charge. For most business processes, that's completely fine. For anything where a customer could act on stale information in that gap, this needs to be a deliberate design decision, not something discovered after a customer complaint.

When I don't reach for this pattern

Not every interaction needs to be an event. If Service A genuinely needs a response before it can proceed, and that response needs to be synchronous, forcing that into an event-driven shape adds complexity without adding value. I reach for events specifically where the interaction is naturally "notify and move on," not for every service-to-service call by default.

The real payoff

The decoupling event-driven architecture buys you is real, but it comes with genuine operational complexity: monitoring dead-letter queues, designing idempotent consumers, and accepting eventual consistency where it doesn't hurt the business. Teams that adopt the pattern without budgeting for that complexity usually end up with a system that's harder to debug than the tightly coupled one it replaced.