OpenTelemetry Best Practices

A solid foundation for observability starts with a uniform SDK bootstrap across all services.

A solid foundation for observability starts with a uniform SDK bootstrap across all services.
The Resource object is the metadata that the backend uses to locate, aggregate, and route telemetry.
Missing or inconsistent resource attributes quickly render data useless.

Key attributes to set

  • service.name – a globally unique, lowercase, hyphen‑separated identifier (e.g., order-service).
  • service.version – the semantic version of the binary, enabling performance regression checks.
  • deployment.environment – distinguishes production, staging, test, etc.
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "order-service",
    "service.version": "1.2.0",
    "deployment.environment": "production",
})

provider = TracerProvider(
    resource=resource,
    sampler=TraceIdRatioBased(0.1),  # 10 % sampling
)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
)
trace.set_tracer_provider(provider)

Package this initialization into a shared library or inject it via a Java Agent / Node.js auto‑instrumentation to avoid divergent setups across teams.

Prioritizing Instrumentation Strategies

Choose the least invasive method that still covers your traffic:

  1. Zero‑Code Instrumentation – leverage official auto‑instrumentation for HTTP, gRPC, database drivers, etc.
  2. Framework‑Level Integration – wrap custom RPC or messaging frameworks in a reusable middleware that emits spans.
  3. Manual Instrumentation – add spans only for critical business steps (e.g., “order split”, “risk check”).

Example of a manual span:

tracer = trace.get_tracer("order-service")

with tracer.start_as_current_span("create_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.item_count", len(items))
    result = do_create_order(order_id, items)

Crafting a Practical Sampling Policy

Full‑trace collection is rarely sustainable in production.
A layered approach balances observability with cost:

  • Header‑Based Sampling – TraceIdRatioBased (or ParentBased wrapper) offers a simple, low‑overhead solution.
  • Tail‑Sampling – configure in the OpenTelemetry Collector to retain all error paths, slow requests, and a representative sample per service instance.
  • Dynamic Adjustment – expose the sampling rate via a configuration service so it can be increased during incidents or traffic spikes.

Typical configuration:

  • Development: 100 % sampling.
  • Production: 10 % header sampling + tail‑sampling for anomalies.

Adhering to Semantic Conventions and Controlling Attribute Cardinality

OpenTelemetry’s semantic conventions (e.g., http.method, db.system, messaging.destination) enable out‑of‑the‑box dashboards and reduce migration friction.
However, high‑cardinality attributes can explode storage and degrade query performance.

What to avoid

  • User IDs, order numbers, full URLs with query strings.
  • Metric labels that vary per request or user.
  • Sensitive data such as phone numbers, IDs, or tokens.

What to sanitize

  • Strip query parameters from URLs before setting http.url.
  • Replace user identifiers with hashed or bucketed values.
  • Mask or drop PII entirely at the collection point.

Deploying the Collector as a Unified Data Gateway

In production, never let applications talk directly to back‑ends.
Deploy an OpenTelemetry Collector per cluster to act as a single entry point:

  • Protocol Decoupling – all services send OTLP; the Collector handles conversion to Jaeger, Prometheus, or log stores.
  • Sampling & Filtering – perform tail‑sampling, attribute pruning, and PII cleansing in the Collector.
  • Resilience – built‑in buffering and retry logic protects services from downstream outages.
  • Multi‑Export – a single telemetry stream can be routed to multiple observability back‑ends simultaneously.

Ensuring Correct Context Propagation

The value of distributed tracing hinges on seamless context flow:

  • Standardize on W3C Trace Context (traceparent / tracestate). Avoid mixing B3 or other legacy formats.
  • For message‑queue scenarios, guarantee that producers embed the context in message headers and consumers extract it before creating child spans.
  • In asynchronous or concurrent code (thread pools, async/await), explicitly pass the current context to avoid broken links.
  • Periodically run end‑to‑end traces to validate connectivity and include link‑coverage checks in regression suites.

Continuous Validation and Performance Tuning

Observability is an ongoing effort. Implement the following checks:

  1. Telemetry Pipeline Health – monitor export failure rates, queue back‑pressure, and dropped span counts.
  2. Span & Attribute Audits – regularly review the number of spans and the size of attributes; prune redundant instrumentation.
  3. SDK Overhead Testing – load‑test the application with the chosen sampling and batching settings to keep CPU and memory usage in check.
  4. Code‑Review Enforcement – embed instrumentation guidelines in the team handbook and verify new spans against the standards during pull requests.

By following these best practices, OpenTelemetry becomes a reliable, cost‑effective backbone for troubleshooting and understanding system behavior, rather than an expensive, under‑utilized data sink.