A Detailed Explanation of OpenTelemetry Metrics

In an observability stack, metrics are one of the primary signals that describe how a system is behaving at any moment.

In an observability stack, metrics are one of the primary signals that describe how a system is behaving at any moment. OpenTelemetry (OTel) provides a vendor‑agnostic framework for defining, collecting, aggregating, and exporting metrics through a unified API and SDK specification. This allows developers to build a consistent monitoring layer that can be swapped between back‑ends without touching application code.

The OTel metric system is organized into three layers:

  • MeterProvider – the global entry point that manages resources and the metric pipeline. It is usually created once during application startup.
  • Meter – a logical namespace for a component or library. A MeterProvider creates Meters, typically named after the component.
  • Instrument – the concrete object used to record a single measurement.

Instrument Types

OpenTelemetry defines six instrument types, grouped into synchronous and asynchronous categories.

Synchronous Instruments

These are invoked directly by application code.

Instrument Typical Use Example
Counter Monotonically increasing counts (e.g., request totals, error counts). order_counter.add(1)
UpDownCounter Counts that can increase or decrease (e.g., active connections, queue length). conn_counter.add(-1)
Histogram Distribution of numeric values (e.g., request latency, response size). latency_hist.record(120)

Asynchronous Instruments

These read values via callbacks at collection time.

Instrument Typical Use Example
ObservableCounter Monotonically increasing observed values (e.g., CPU time since start). cpu_time_observable.observe(lambda: get_cpu_time())
ObservableUpDownCounter Observed values that can go up or down (e.g., cache size). cache_size_observable.observe(lambda: get_cache_size())
ObservableGauge Instantaneous snapshots (e.g., current memory usage, temperature). mem_gauge.observe(lambda: get_memory_usage())

Code Example (Python)

Below is a minimal Python snippet that demonstrates how to set up a provider, create a meter, and record metrics with attributes (labels).

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
    PeriodicExportingMetricReader,
    ConsoleMetricExporter,
)

# Initialize the MeterProvider with a periodic exporter
provider = MeterProvider(
    metric_readers=[PeriodicExportingMetricReader(ConsoleMetricExporter())]
)
metrics.set_meter_provider(provider)

# Create a meter for the "shop.orders" component
meter = metrics.get_meter("shop.orders")

# Synchronous counter for total orders
order_counter = meter.create_counter(
    "shop.orders.total",
    unit="1",
    description="Total number of orders processed",
)

# Histogram for order latency
latency_hist = meter.create_histogram(
    "shop.order.latency",
    unit="ms",
    description="Latency of order creation",
)

def handle_order(channel: str):
    order_counter.add(1, {"channel": channel})
    latency_hist.record(120, {"channel": channel})

The attribute dictionary ({"channel": channel}) provides dimensionality, similar to Prometheus labels.

Data Model & Aggregation

OTel’s metric data model is built around data points. Each data point contains:

  • Timestamp – when the measurement was taken.
  • Value – the numeric measurement.
  • Attributes – key/value pairs for dimensionality.

Metrics can be reported in two temporal modes:

  • Cumulative – the total value since the start of the measurement period.
  • Delta – the change since the last collection.

Aggregation is performed by the SDK. By default:

  • Counter values are summed.
  • Histogram values are bucketed into predefined ranges.

Developers can customize aggregation through Views, allowing changes to bucket boundaries, renaming metrics, or altering aggregation strategies.

Exporting & Backend Integration

Metrics are sent to a backend via an Exporter. Common exporters include:

  • OTLP Exporter – sends data over gRPC or HTTP to an OpenTelemetry Collector. This is the most flexible, generic choice.
  • Prometheus Exporter – exposes a /metrics endpoint that Prometheus scrapes, enabling seamless integration with existing Prometheus setups.
  • Console Exporter – outputs metrics to stdout, useful for local debugging.

A typical production pipeline looks like:

Application → OTLP Exporter → OpenTelemetry Collector → Prometheus / Grafana

The Collector decouples the application from the storage or visualization layer, so switching back‑ends only requires a Collector configuration change.

Best Practices

  1. Naming Convention
    Use the pattern <namespace>.<metric_name>.<unit>. Keep names concise, descriptive, and consistent across services.

  2. Attribute Cardinality
    Avoid high‑cardinality values such as user IDs or request IDs as attributes. They can explode the number of unique time series and overwhelm back‑ends.

  3. Instrument Selection
    Record business events with synchronous instruments. Use asynchronous instruments to expose the current state of the system (e.g., memory usage).

  4. Export Interval
    A 30‑second to 1‑minute export window balances freshness with overhead for most monitoring scenarios.

  5. View Configuration
    Tailor aggregation (e.g., custom histogram buckets) to the specific metrics you care about. This reduces noise and improves query performance.

  6. Combine Signals
    Pair metrics with traces and logs to gain a holistic view of distributed systems. OTel’s unified model makes this integration straightforward.

Conclusion

Mastering OpenTelemetry metrics involves understanding the instrument hierarchy, selecting the right instrument type, designing low‑cardinality attributes, and configuring efficient export pipelines. When combined with traces and logs, OTel delivers a complete observability solution that is portable, scalable, and vendor‑agnostic.