Structured logging

Structured logging replaces free-form text log output with machine-parseable records - typically JSON - where every field has a defined name and type. Instead of writing "Error processing payment for user 42: timeout after 5000ms", you write {"level":"error","service":"payment","event":"payment_timeout","user_id":42,"duration_ms":5000,"timestamp":"2024-01-15T14:32:01Z"}. The information content is identical, but the format is queryable: you can filter by level=error, aggregate by service, histogram by duration_ms, and alert when event=payment_timeout exceeds a threshold.

Structured logging is the first observability practice that genuinely enables machine consumption of log data. Unstructured text logs require human pattern recognition to extract meaning - a human reads "Error processing payment" and understands what happened. Structured logs require schema definition upfront but return that investment as queryability, alertability, and eventually agent-accessibility. When every log line is a JSON object with consistent fields, the entire log stream becomes a queryable database rather than a text file.

The standard approach is to adopt a structured logging library in each language your services use - structlog in Python, zerolog or zap in Go, winston with JSON format in Node.js, slf4j with Logback JSON encoder in Java - and configure it to emit JSON to stdout. The output flows through your container orchestrator to a log aggregation system: Datadog, the ELK stack (Elasticsearch, Logstash, Kibana), Grafana Loki, or AWS CloudWatch Logs Insights. Once in the aggregation layer, every field in every log line is indexed and queryable.

The key fields that every structured log line should include are: timestamp (ISO 8601, UTC), log level (DEBUG/INFO/WARN/ERROR), service name, trace ID (discussed below), and the specific event that occurred. Beyond these mandatory fields, add domain-specific fields for every log line: user IDs, request IDs, operation names, durations, result codes. The richness of these fields determines the richness of the queries you can run against your logs later. A structured log line with 15 well-chosen fields is dramatically more valuable than 15 unstructured log lines covering the same events.

The shift from unstructured to structured logging unlocks a cascade of downstream capabilities:

• Queryability transforms investigation speed - diagnosing a production issue by running service:payment level:error event:payment_timeout in Datadog takes 10 seconds; grepping log files takes 10 minutes and requires infrastructure access
• Aggregation enables anomaly detection - when logs have consistent numeric fields (duration_ms, retry_count, queue_depth), you can build dashboards and alerts on their distributions; unstructured logs have no numeric fields to aggregate
• Consistent schema enables cross-service correlation - when every service uses the same field names for common concepts (trace_id, user_id, service), you can query across service boundaries and find the path a request took through your system
• Log data becomes agent-accessible - Datadog, Elasticsearch, and Loki all expose query APIs; an AI agent can call these APIs to retrieve relevant log data for investigation, something impossible with unstructured log files
• Audit trails become reliable - structured authentication, authorization, and data-mutation logs can serve as compliance audit trails; unstructured logs cannot, because there is no guarantee they contain the required fields

1. Choose a structured logging library for each service - Pick the idiomatic choice for your language: structlog (Python), zap or zerolog (Go), winston (Node.js), logback with logstash-logback-encoder (Java/Kotlin). Configure it to output JSON to stdout. This is a one-time setup per service.
2. Define your mandatory field schema - Before touching application code, define the fields every log line must include: timestamp, level, service, version, trace_id. Write a logging wrapper or middleware that adds these fields automatically. No application code should ever have to set service or timestamp manually.
3. Add a request-scoped context carrier - Use your language's context mechanism (Go context.Context, Python contextvars, Java MDC) to carry trace ID and other request-scoped fields through your call stack. Any log line emitted during request processing automatically inherits these fields without the application code passing them explicitly.
4. Connect to a log aggregation platform - Configure your container runtime or log shipper (Fluentd, Filebeat, the Datadog Agent) to forward stdout JSON logs to your chosen aggregation system. Datadog and Grafana Loki parse JSON automatically; ELK requires a Logstash or Beats pipeline. Validate that logs are appearing in the UI before proceeding.
5. Migrate unstructured logs incrementally - Do not attempt to refactor all logging at once. Start with the highest-traffic code paths: HTTP request handlers, database query wrappers, external API calls. Structured logs for these critical paths give you immediate value even while the rest of the codebase is still unstructured.
6. Define and document your log schema - Write a document listing all standard field names and their types. When a new service is created or a new event type is logged, it follows the schema. Consistency across services is the entire value proposition of structured logging; inconsistency recreates the fragmentation problem you were trying to solve.

Inconsistent field names across services. Service A logs userId, service B logs user_id, service C logs uid. All three contain user identifiers but no query can retrieve them together. Enforce a field naming standard before teams start instrumenting. The opentelemetry-semantic-conventions package defines standard names for common fields - use it as your baseline rather than inventing your own.

Over-logging at DEBUG level in production. Structured logging is cheap compared to unstructured logging because it is machine-parseable, but it is not free. Debug-level logs that run in production hot paths can add meaningful latency and cost. Set production log levels to INFO by default, with the ability to dynamically enable DEBUG for specific services or request IDs when investigating an incident.

Treating JSON as sufficient for correlation. Structured logs from multiple services are still siloed unless they share a common correlation ID (trace ID). Without trace IDs, you can query within a service but cannot follow a request across service boundaries. The trace ID is what connects structured logging to distributed tracing, and it needs to be threaded through every service call from day one.

No log retention policy. Centralized structured logs at scale can be expensive. A service emitting 10,000 log lines per second generates terabytes per day. Without a retention policy (keep ERROR logs for 90 days, DEBUG logs for 7 days), costs grow unchecked. Define retention tiers by log level as part of your initial aggregation setup.

Logging the same information at multiple levels. Logging a request at entry, logging each step, logging the result, and logging a summary creates redundant data that inflates storage costs without adding analytical value. Log the entry with full context, log errors if they occur, log the result with duration. Omit intermediate steps unless they represent distinct events worth tracking independently.