I had an interesting conversation with a colleague at work recently while walking through the observability setup of an appliance i instrumented. We looked at different signals coming from logs, application telemetry, dependencies, and distributed traces.
At some point, while looking at database spans, an interesting question came up: “When Otel creates database spans, do those spans expose customer information (PII) or sensitive customer data?”
Its a good question, and it points to something bigger about how we think about instrumentation in general
What should actually go inside a span?
And maybe more importantly:
How much context do we really need to understand what our systems are doing?
A span is a unit of work
At the simplest level, a span represents an operation that happened during the execution of a request, or some other piece of work.
Say an API receives a request to create an order. In code, that might look like this:
async function createOrder(req: Request, res: Response) {
return tracer.startActiveSpan("POST /orders", async (span) => {
try {
await validateRequest(req);
await checkInventory(req);
await insertOrder(req);
await callPaymentService(req);
res.status(201).json({ ok: true });
} finally {
span.end();
}
});
}
The whole request can be one span. The important steps underneath it can become child spans.
In your trace backend, timing might show up like this:
{
"name": "POST /orders",
"duration_ms": 620,
"children": [
{ "name": "inventory-service", "duration_ms": 80 },
{ "name": "INSERT orders", "duration_ms": 35 },
{ "name": "POST payment-service", "duration_ms": 470 }
]
}
Right away, you know something useful. The database probably isnt the bottleneck. The payment service is where the time went.
Thats the value of spans. They turn:
“The API is slow.”
into:
“This particular operation in this particular dependency consumed most of the request latency.”
That difference saves a lot of debugging time.
The real power comes from trace context
Individual spans are useful. Distributed tracing gets interesting when those spans connect.
Every trace has a trace context the thing that lets different components know theyre part of the same distributed operation.
Your order service might call inventory and payment in sequence:
await tracer.startActiveSpan("POST /orders", async (orderSpan) => {
await tracer.startActiveSpan("inventory-service", async () => {
await inventoryClient.reserveItems(order);
});
await tracer.startActiveSpan("payment-service", async () => {
await paymentClient.charge(order);
});
orderSpan.end();
});
Those systems might run in different processes, containers, pods, or even clusters. From an observability perspective, you still want to answer one question:
What happened to this request?
Thats where trace propagation comes in.
When the order service calls inventory, trace context travels with the request. Inventory creates its own span but stays linked to the original trace. Same for payment.
I wrote about this in more detail last year in Lifting the Hood on Trace Propagation in OpenTelemetry — traceparent, tracestate, inject/extract, and how context actually moves between services. This post is more about what belongs on spans and why; that one goes deeper on keeping traces connected across a distributed system.
Without propagation you get disconnected telemetry three separate traces that happen to occur around the same time. With it, you get one story:
{
"trace_id": "7f83...",
"root": "POST /orders",
"spans": [
"Order Service",
"Inventory Service",
"Database",
"Payment Service",
"Payment Provider"
]
}
Otel usually propagates this via W3C Trace Context headers like traceparent carrying the trace and parent span IDs. Propagation is easy to overlook when you’re getting started. Without it, you might have plenty of spans and still no coherent narrative.
Outgoing calls need to inject context; incoming handlers need to extract it
// order-service → inventory-service
const headers: Record<string, string> = {};
propagation.inject(context.active(), headers);
await fetch(`${inventoryUrl}/reserve`, {
method: "POST",
headers,
body: JSON.stringify(payload),
});
// inventory-service receives the request
const parentCtx = propagation.extract(context.active(), req.headers);
await context.with(parentCtx, async () => {
await tracer.startActiveSpan("inventory-service", async (span) => {
// this span belongs to the same trace as POST /orders
span.end();
});
});
Then what goes inside a span?
This is where semantic conventions matter.
If every team invents its own attribute names, telemetry becomes painful to query. One service sends request_method=GET, another httpMethod=GET, another just method=GET. Same idea, three dialects.
OpenTelemetry Semantic Conventions give you shared vocabs for HTTP, databases, messaging, RPC, and more. instead of debating naming in every repo, you use attributes like:
span.setAttributes({
"http.request.method": "GET",
"http.response.status_code": 200,
"service.name": "order-service",
"db.system.name": "postgresql",
"server.address": "payments.internal",
"error.type": "TimeoutError",
});
At small scale, inconsistency is annoying. At hundreds of services, its the difference between having telemetry data and being able to use it.
What about database spans and PII?
Back to the question that started the conversation.
Seeing a database span does not mean my guy (Otel) suddenly exposes everything in that database. A database span describes the operation and its behaviour how long it took, whether it failed, which system was queried.
await tracer.startActiveSpan("db.query", async (span) => {
span.setAttributes({
"db.system.name": "postgresql",
"db.operation.name": "INSERT",
"db.collection.name": "orders",
});
try {
await db.query("INSERT INTO orders (...) VALUES (...)", params);
} finally {
span.end();
}
});
Useful questions: which database, how long, success or failure, how much of the request latency came from here. Not: whats the customer’s password.
Credentials, tokens, secrets, none of that should land on a span just because the application has access to it. We can attach arbitrary attributes. That does not mean we should.
Business observability doesnt require exposing the customer
Theres another side to instrumentation that gets overlooked. Observability isnt only CPU, HTTP latency, and slow queries. Sometimes the business has questions too.
How many signups today? How many checkouts succeeded? How many payments failed? Whats the transaction volume?
Those are legitimate telemetry needs. You can instrument a signup flow without putting customer secrets in the pipeline:
await tracer.startActiveSpan("signup", async (span) => {
await validateInput(req);
await createAccount(req);
await sendWelcomeEmail(req);
span.addEvent("signup.completed");
span.setAttributes({
"workflow.name": "signup",
"workflow.outcome": "success",
});
});
From spans and the metrics you derive from them, you might track signup attempts, successes, failures, and duration. You dont need customer.password, customer.access_token, or customer.card_number to know whether the workflow is healthy.
Same for payments. You might record that a transaction happened, whether it succeeded, how long it took, and carefully scoped business measurements for volume or revenue. Thats business context. Its not the same as stuffing login credentials into a trace.
Traces and metrics answer different questions
This is why traces and metrics work well together.
A metric might tell you:
signup_success_total = 18,421
payment_failure_rate = 2.4%
Thats what is happening at scale. A trace tells you what happened on one path through the system.
Say payment success rate drops from 99.8% to 91.2%. The metric flags the problem. Traces help you find why. Affected requests might look like:
{
"name": "POST /checkout",
"duration_ms": 5800,
"children": [
{ "name": "validate-cart", "duration_ms": 20 },
{ "name": "create-order", "duration_ms": 45 },
{ "name": "payment-service", "duration_ms": 5600 },
{ "name": "external-provider", "duration_ms": 5500 },
{ "name": "response", "duration_ms": 15 }
]
}
Metrics tell you something is happening. Traces help explain where.
Spans are what make those traces readable.
Not every function needs a span
Early on, there’s a temptation to instrument everything. I have seen traces that look like a stack dump parseRequest, validateString, getCurrentTime, formatCurrency each with its own span. Technically possible. Not useful.
I instrument meaningful boundaries:
- HTTP requests
- database operations
- service-to-service calls
- external APIs
- message publish/consume
- background jobs
- business workflows that actually matter
Simple rule: if this operation fails or gets slow, would a span here help me understand why? If yes, consider it. If no, skip it.
// worth a span
await tracer.startActiveSpan("checkout", async (span) => {
await createOrder();
await processPayment();
span.end();
});
// probably not worth individual spans
function formatCurrency(amount: number) {
return new Intl.NumberFormat("en-NG", {
style: "currency",
currency: "NGN",
}).format(amount);
}
closing thoughts
I hope this was useful to anyone who made it this far. A lot of this came from a simple conversation at work, but it got me thinking about how we approach instrumentation and what we actually need from our telemetry.
I also hope I can stay consistent with writing. I have a lot of thoughts from the things I build, break, debug, and learn along the way and I had like to start putting more of them out there and sharing them with the community.
More to come, hopefully.