i had an interesting conversation with an engineering team today around Otel and application observability
One of the questions that came up was simple, but important:
“When OpenTelemetry creates database spans, do those spans expose customer information (PII) or sensitive customer data?”
The short answer is: not automatically.
And I think the question highlights something that engineers sometimes misunderstand about observability.
OpenTelemetry is not a tool that magically opens an application and exposes everything happening inside it. What you see depends heavily on what you instrument, what telemetry your application already produces, and what attributes you deliberately attach to that telemetry.
During the discussion, I walked through an appliance I had instrumented. There were several layers of telemetry available.
At the edge, we had NGINX access logs.
We also had SSL-related access logs.
Then, deeper inside the appliance, we had application instrumentation producing traces and spans.
These are very different signals.
An NGINX access log might tell me that a request came in, which endpoint was called, the response status, how long the request took, and perhaps information about the client:
192.168.1.10 - - [31/Aug/2026:14:02:11 +0000] "POST /api/orders HTTP/1.1" 200 482 4.102
A trace tells a different story.
It can tell me what happened after that request entered the application.
In code, that journey often looks less like a black box and more like nested work you can name:
async function createOrder(req: Request, res: Response) {
return tracer.startActiveSpan("POST /api/orders", async (span) => {
try {
await validate(req); // span: Validate request
await authenticate(req); // span: Check authentication
const order = await db.createOrder(/* ... */);
// span: Query customer database / SELECT ...
await payments.charge(order);
// span: Call payment service / POST /payments
res.status(201).json(order);
} finally {
span.end();
}
});
}
Each meaningful operation can become a span.
Together, those spans form a trace.
This is where distributed tracing becomes extremely useful.
Imagine an API request takes four seconds to complete.
From the outside, all we know is:
POST /api/orders → 200 → 4.1 seconds
The API technically worked.
But four seconds is terrible.
With tracing, we might discover timing that looks like this:
{
"name": "POST /api/orders",
"duration_ms": 4100,
"children": [
{ "name": "authentication", "duration_ms": 30 },
{ "name": "business logic", "duration_ms": 40 },
{ "name": "database.query", "duration_ms": 3700 },
{ "name": "response", "duration_ms": 20 }
]
}
Now we have somewhere to investigate.
The application isn’t simply “slow.”
A particular database operation is slow.
That distinction matters.
Database spans are especially interesting
One of the things I demonstrated during the conversation was how database spans are created and how they appear as part of a distributed trace.
Suppose an application receives a request and eventually talks to PostgreSQL, MySQL, MongoDB or another datastore.
With database instrumentation, a span often looks like this:
await tracer.startActiveSpan("db.query", async (span) => {
span.setAttributes({
"db.system.name": "postgresql",
"db.operation.name": "SELECT",
"db.collection.name": "customers",
"db.query.text": "SELECT * FROM customers WHERE id = $1",
});
try {
return await db.query("SELECT * FROM customers WHERE id = $1", [customerId]);
} finally {
span.end();
}
});
In the backend UI, that shows up as a child of the HTTP span — for example a GET /customers/32904 request where the DB query alone takes 840ms when it normally takes 20ms.
That span has given us a very useful clue.
This becomes even more powerful when several services are involved. Without distributed tracing, troubleshooting a cross-service request becomes a familiar exercise:
Check the gateway logs.
Check the order service logs.
Ask another team to check the inventory service.
Search the database logs.
Ask the payment team.
Compare timestamps.
Try to reconstruct what happened.
With properly propagated trace context, those operations can belong to the same trace. The outgoing call carries the parent context:
const headers: Record<string, string> = {
"content-type": "application/json",
};
propagation.inject(context.active(), headers);
const response = await fetch(inventoryUrl, {
method: "POST",
headers,
body: JSON.stringify(payload),
});
Instead of investigating five disconnected systems, you’re following one request through the system.
So where should spans actually be used?
Not every function needs a span.
That would create noise rather than observability.
I find spans most valuable around meaningful boundaries and operations.
A few examples:
Database operations
If your API depends on a database, understanding how much time is spent querying that database can immediately narrow down performance problems.
External API calls
Your service might be healthy while a third-party API is taking three seconds to respond. A client span makes that dependency visible:
await tracer.startActiveSpan(
"HTTP POST",
{
kind: SpanKind.CLIENT,
attributes: {
"server.address": "payments.example.com",
"http.request.method": "POST",
"url.path": "/v1/charges",
},
},
async (span) => {
try {
return await fetch("https://payments.example.com/v1/charges", {
method: "POST",
body: JSON.stringify(charge),
});
} finally {
span.end();
}
},
);
Service-to-service communication
In microservice environments, spans allow you to understand how requests move between services and where latency is introduced.
Queues and background jobs
A request may finish quickly but trigger asynchronous work. Tracing producers and consumers helps answer questions such as: when was the message published, when was it consumed, and where did processing fail?
// producer
const headers: Record<string, string> = {};
propagation.inject(context.active(), headers);
await producer.send({
topic: "orders.created",
messages: [{ value: payload, headers }],
});
// consumer
const parentCtx = propagation.extract(context.active(), message.headers);
await context.with(parentCtx, async () => {
await tracer.startActiveSpan("orders.created process", async (span) => {
try {
await processOrder(message.value);
} finally {
span.end();
}
});
});
Critical business operations
Some operations deserve explicit instrumentation because they matter to the system:
await tracer.startActiveSpan("checkout", async (span) => {
try {
// payment-processing, order-creation, account-validation,
// report-generation, file-processing — same idea
} finally {
span.end();
}
});
These are much more useful observability boundaries than creating spans around every small helper function.
But what about usernames and passwords?
This is where observability needs engineering discipline.
Telemetry should provide enough context to understand system behaviour without unnecessarily collecting sensitive information.
A span might reasonably contain things like:
span.setAttributes({
"http.request.method": "POST",
"http.response.status_code": 200,
"server.address": "api.internal",
"service.name": "order-service",
"db.system.name": "postgresql",
"error.type": "TimeoutError",
});
That doesn’t mean we should start attaching things like:
// don't do this
span.setAttributes({
"user.password": password,
"authorization.header": authHeader,
"credit_card.number": card,
"access_token": token,
});
Observability is not an excuse to collect everything.
In fact, one of the most important parts of building an observability platform is deciding what should never enter the telemetry pipeline in the first place.
Passwords, authentication tokens, session secrets and similar credentials should not become telemetry attributes.
Even seemingly harmless attributes deserve thought. A username, email address or customer ID may help with debugging, but it can also introduce privacy and data-governance concerns.
The question shouldn’t simply be:
“Can OpenTelemetry capture this?”
It should also be:
“Do we actually need this information to understand the behaviour of the system?”
Those are very different questions.
Observability starts with the engineer writing the system
This was probably my biggest takeaway from the conversation.
I think every backend engineer should have observability in mind while designing a system.
When you’re implementing an endpoint, don’t only think:
Does this code work?
Also think:
If this fails in production at 2 AM, what information will tell me why?
When you’re adding a database call:
Will I know if this query suddenly becomes slow?
When you’re calling another service:
Will I know whether the latency came from my application or the downstream service?
When you’re publishing a message:
Can I follow that operation when another service consumes it?
When you’re handling an error:
Will the telemetry preserve enough context to understand what happened without exposing sensitive data?
Those questions influence how observable a system becomes.
OpenTelemetry gives us the standards and instrumentation primitives.
Logs give us events.
Metrics tell us how the system is behaving over time.
Traces show us how work moves through the system.
Spans give us the individual pieces of that journey.
But none of those signals replace thoughtful engineering.
The best observability doesn’t start with a dashboard.
It starts while the system is being built.