OpenTelemetry basic

OpenTelemetry (OTel) is the open standard for collecting and exporting telemetry data - traces, metrics, and logs - from distributed systems. "OpenTelemetry basic" means instrumenting your services with the OTel SDK to produce distributed traces, configuring an OTel Collector to receive and forward that data, and visualizing the results in a tracing backend like Jaeger or Grafana Tempo. At this level, you have your first end-to-end view of how a request travels through multiple services - which services it touched, how long each took, and where errors occurred.

Distributed tracing solves a problem that structured logging alone cannot: following a single request across service boundaries. When a user action triggers calls across an API gateway, an auth service, a business logic service, and a database, each service may log the relevant steps. But without a shared trace context, those logs are four unrelated streams. A trace links them: every log line, every database query, every external call that occurred during that request is tagged with the same trace ID and organized into a parent-child span hierarchy. You can see in a single view that the API gateway received the request (10ms), forwarded to the auth service (45ms), which queried the user database (38ms of that), then returned to the business logic service (120ms), which made two downstream calls before responding.

The OTel SDK provides auto-instrumentation for most popular frameworks. In Node.js, the @opentelemetry/auto-instrumentations-node package automatically instruments Express, HTTP clients, database drivers, and more without modifying application code. In Python, opentelemetry-instrumentation covers Flask, Django, SQLAlchemy, Redis clients, and gRPC. In Java, the OTel Java agent instruments Spring Boot, JDBC, and most common libraries via a JVM agent flag. Auto-instrumentation means you can get distributed traces running across your entire application in a day, without modifying a single line of business logic.

The OTel Collector is the recommended deployment pattern at even basic scale. Rather than shipping telemetry directly from each service to the backend, services send to the Collector (running as a sidecar or cluster-level agent), which batches, filters, samples, and forwards to the configured backend. This decouples your services from the choice of tracing backend and allows you to change or add backends (Jaeger, Tempo, Datadog, Honeycomb) without modifying service code. The Collector also provides sampling: in a high-traffic service, you do not need to record every trace; sampling 10% of requests while keeping 100% of error traces is a common and sensible configuration.

The shift from structured logging to distributed tracing changes how you understand your system:

• End-to-end request visibility - see exactly how long each service took for a specific user request, not just aggregated averages; identify the specific hop where latency is added
• Error attribution in distributed systems - when a request fails, the trace shows exactly which service threw the exception and what the call chain looked like at that moment
• Performance regression detection - traces make it obvious when a deployment added latency to a specific service; the P99 latency graph shows the change, and individual traces show which span got slower
• Database and external call visibility - auto-instrumentation captures database queries, HTTP calls, queue operations, and cache interactions without any manual instrumentation; these are often the root cause of performance problems
• Agent-queryable request context - agents investigating incidents can query the tracing backend for traces matching specific criteria (high latency, specific error type, specific user) and reconstruct the exact sequence of events during an incident

1. Start with auto-instrumentation on one service - Pick your most trafficked service and add the OTel auto-instrumentation package for its language. For Node.js: npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node. For Python: pip install opentelemetry-sdk opentelemetry-instrumentation. Configure the OTLP exporter endpoint and run. You should see traces within minutes.
2. Deploy the OTel Collector - Run the otel/opentelemetry-collector-contrib Docker image. Configure a minimal pipeline: OTLP receiver → batch processor → Jaeger/Tempo exporter. Start all services pointing at the Collector on port 4317 (gRPC) or 4318 (HTTP). The Collector handles buffering and retry so your services are not blocked by backend unavailability.
3. Set up Jaeger or Grafana Tempo as your tracing backend - Jaeger (jaegertracing/all-in-one) runs in Docker with a single command and provides a UI for trace search and visualization. Grafana Tempo integrates with the broader Grafana stack and scales better for production. Start with Jaeger for speed; migrate to Tempo when you need the Grafana integration.
4. Propagate trace context across service calls - Auto-instrumentation handles context propagation automatically for HTTP calls, gRPC, and most message queues. Verify that traces actually span multiple services by creating a test request that touches at least two services and confirming that the trace shows both services as spans. If spans are disconnected, trace context is not being propagated.
5. Add custom spans for business-critical operations - Auto-instrumentation captures framework-level operations but not your business logic. Add manual spans for operations that matter: tracer.startSpan("process_payment"), tracer.startSpan("validate_inventory"). These custom spans appear in the trace alongside the auto-instrumented framework operations and give you business-level visibility.
6. Configure tail-based sampling for production - Recording 100% of traces in a high-traffic production service is expensive. Configure the OTel Collector's tail sampling processor to keep 100% of error traces, 100% of slow traces (P99+), and 10% of normal traces. This captures all the interesting data while controlling cost.

Instrumenting only one service and declaring success. A trace that shows only one service is just a structured log with extra steps. The value of distributed tracing emerges when traces span multiple services. Instrument all services that participate in user-facing requests before drawing conclusions about your tracing setup.

Ignoring sampling from the start. A high-traffic service with 100% trace sampling will generate enormous data volumes and incur significant storage costs. Configure sampling before going to production. Head-based sampling (decide at the entry point) is simple but loses error traces. Tail-based sampling in the OTel Collector (decide after the trace completes) is more sophisticated and keeps all error traces.

Confusing metrics with traces. OTel supports both. Traces answer "what happened during this specific request?" Metrics answer "what is the overall behavior of this service over time?" Both are necessary and complementary. Many teams start OTel for tracing and forget to also configure metrics export, leaving gaps in their observability.

Not adding trace IDs to structured logs. Traces and logs are most powerful when correlated: given a trace, find the logs; given a log line, find the trace. This requires the same trace ID to appear in both. Configure your structured logging library to read the current OTel trace ID from the context and include it as a log field. Most OTel libraries provide a bridge for this; use it from day one.

Over-instrumenting and creating span noise. Not every function call needs a span. A service with thousands of spans per request creates traces that are harder to read than a trace with 20 meaningful spans. Instrument at the boundary level (external calls, database queries, significant business operations) rather than at the function call level.