When to add custom OpenTelemetry instrumentation

WHAT THIS TESTS Whether you understand the gap between technical telemetry and business observability. Auto-instrumentation gives you spans for HTTP handlers, database queries, and outbound calls, but it has no idea what a span means to the business.

A GOOD ANSWER COVERS You add custom spans around domain operations that have no clean framework boundary: a multi-step pricing calculation, an inventory reservation, a fraud-scoring step, or a feature-flag-gated code path. On those spans you attach semantic attributes such as tenant.id, plan.tier, order.value_bucket, or payment.provider. This lets you answer questions auto-instrumentation cannot: which plan tier experiences the worst p99 checkout latency, whether one payment provider drives most retries, or how long the discount engine takes independent of the surrounding HTTP request. You should also create custom metrics and events for things like cart-abandonment steps.

COMMON WRONG ANSWERS Instrumenting every function indiscriminately, which adds noise and overhead. Putting raw user IDs, emails, or request bodies into attributes, creating high cardinality and PII risk. Treating custom instrumentation as a replacement for auto-instrumentation rather than a complement.

LIKELY FOLLOW-UPS How do you avoid cardinality explosions in attributes? How do you keep instrumentation maintainable as code changes? How do you propagate business context across async boundaries?

ONE CONCRETE EXAMPLE A SaaS billing service auto-instruments the HTTP and DB layers but cannot explain why some invoices are slow. Engineers wrap the tax-calculation and currency-conversion steps in custom spans with attributes country and line_item_count. Traces immediately reveal that invoices with many line items in high-tax jurisdictions spend most of their time in tax calculation, pointing the optimization effort precisely where auto-instrumentation showed only a single opaque slow handler.