The hardest incidents I've dealt with weren't the ones with obvious causes. They were the ones where a request slowed down somewhere across four or five services, and nobody could say exactly where, because each service only had visibility into itself. Logs told you something happened. They didn't tell you why, or where in the chain the time actually went.
That gap is what observability closes, and in the .NET ecosystem, OpenTelemetry has become the standard way to close it.
It's worth being precise about this, because teams often reach for logging when what they actually need is tracing.
Logs tell you discrete events happened: "payment processed," "validation failed."
Metrics tell you aggregate numbers over time: request rate, error rate, latency percentiles.
Traces tell you the full journey of a single request across every service it touched, with timing for each step.
On a distributed BNPL platform I helped design, integrating fraud detection with Kubernetes-orchestrated services, traces were what actually let us pinpoint that a specific external fraud-check call was adding unacceptable latency, something logs alone never made obvious because each service logged in isolation with no way to connect the dots.
Setting up OpenTelemetry in ASP.NET Core
The instrumentation itself is fairly mechanical once you understand the pieces:
builder.Services.AddOpenTelemetry()
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddEntityFrameworkCoreInstrumentation()
.AddOtlpExporter())
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddOtlpExporter());
This alone gives you automatic instrumentation for incoming HTTP requests, outgoing HTTP calls, and database queries, exported to whatever backend you're using, whether that's Jaeger, Grafana Tempo, or a managed APM tool.
Propagating context across service boundaries
Automatic instrumentation only gets you so far. The real value of tracing comes from context propagation, where a trace ID generated at the entry point of a request flows through every downstream service call, so you can see the entire journey as one connected timeline instead of disconnected fragments.
With AddHttpClientInstrumentation(), this mostly happens automatically for outbound HTTP calls in ASP.NET Core. Where it gets trickier is with message queues. If you're publishing to RabbitMQ or a similar broker, you need to explicitly propagate trace context into message headers and extract it on the consuming side, or the trace breaks the moment a request crosses an asynchronous boundary.
var propagator = Propagators.DefaultTextMapPropagator;
propagator.Inject(new PropagationContext(activity.Context, Baggage.Current), message,
(msg, key, value) => msg.Headers.Add(key, value));
Skipping this step is the single most common reason teams end up with tracing that looks complete for synchronous HTTP calls but goes dark the moment a message hits a queue.
Custom spans for the parts that matter
Automatic instrumentation covers the framework-level work. It won't tell you how long your actual business logic took inside a method. For that, I add custom activities around the operations that matter most:
private static readonly ActivitySource ActivitySource = new("PaymentService");
public async Task ProcessPaymentAsync(Payment payment)
{
using var activity = ActivitySource.StartActivity("ProcessPayment");
activity?.SetTag("payment.amount", payment.Amount);
activity?.SetTag("payment.currency", payment.Currency);
// actual processing logic
}
I'm deliberate about what gets a custom span. Instrumenting everything creates noise that buries the signal. I focus on operations with real business significance or a history of being slow.
Structured logging that actually correlates with traces
Plain text logs are close to useless once you're debugging across services. Structured logging, where each log entry carries the trace ID and relevant context as queryable fields rather than a formatted string, is what lets you jump from a trace directly to the exact log lines for that request.
logger.LogInformation("Payment processed for {CustomerId} with amount {Amount}",
customerId, amount);
Paired with OpenTelemetry's logging integration, this log automatically carries the active trace ID, so when you're staring at a slow trace in your tracing backend, you can pull the exact log lines for that specific request instead of grepping through everything in a time window.
Health checks are observability too
They get treated as a separate concern, but health checks are part of the same picture. MapHealthChecks() combined with checks for your database, message broker, and critical downstream dependencies gives your orchestrator, and your on-call engineer, an honest answer to "is this service actually okay right now," not just "is the process running."
builder.Services.AddHealthChecks()
.AddSqlServer(connectionString)
.AddRabbitMQ(rabbitConnectionString);
Why this isn't optional at scale
Every service you add to an architecture is another place a request can slow down or fail, and another place where local reasoning stops working. You cannot debug a five-service request path by mentally reconstructing what each service probably did. You need the actual timeline, in front of you, with the exact point where things went wrong highlighted.
Setting this up before you need it is the difference between a ten-minute investigation and a multi-hour one during an actual incident. I treat observability the same way I treat security: something designed in from day one, not bolted on after the first outage makes the gap painfully obvious.
