Production-Grade Logging and Observability: A Practical Guide

How does a real production system handle logging? Where do logs actually live, what format are they stored in, and how do logs differ from metrics and traces? This guide walks the full picture — from structured logging and storage lifecycle to the Grafana stack, the three pillars of observability, a worked "is this a log or a trace?" example, capturing request duration with Promtail, and finally wiring logs into Claude Code over MCP in a token-efficient way.

Logging Is a First-Class Concern

In production-grade applications, logging isn't an afterthought — it's designed in from the start.

Structured logging. Logs are written as structured data (usually JSON), not plain text, so they can be parsed, filtered, and queried programmatically.

{"timestamp":"2026-06-24T10:15:00Z","level":"ERROR","service":"payment","trace_id":"abc123","message":"Payment failed","user_id":"u789"}

Log levels. Standard severity levels are used consistently: TRACE, DEBUG, INFO, WARN, ERROR, FATAL. Production typically runs at INFO or WARN; DEBUG is enabled temporarily for troubleshooting.

A few non-negotiables:

• Never log secrets or PII — mask/redact tokens, passwords, card numbers before they hit the log.
• Use async/non-blocking logging so logging doesn't become a performance bottleneck.
• Always include context: service name, version, environment, host.
• Sample high-volume logs to control cost.
• Encrypt logs in transit and at rest; control and audit access.

Anatomy of a Production-Grade Log Structure

A good log line is a flat-ish JSON object with a consistent, predictable schema across every service, so it parses the same way everywhere. Group fields into clear concerns:

A canonical structured log following that schema:

{
  "timestamp": "2026-06-24T08:50:34.763Z",
  "level": "info",
  "action": "request-response",
  "appName": "res-crm",
  "environment": "production",
  "version": "1.4.2",
  "host": "res-crm-7d9f-abc",
  "traceId": "abc123",
  "spanId": "def456",
  "request":  { "method": "GET", "url": "/orders", "ip": "10.20.28.62" },
  "response": { "statusCode": 200 },
  "metrics":  { "duration": 0.77 },
  "userId": "u789",
  "orgId":  "o42",
  "error": {},
  "businessContext": { "orderId": "ord_001" }
}

Schema rules that keep it production-grade:

• Stable field names across all services — duration always means the same thing, always a number.
• Always emit the correlation fields (even if empty) so logs are trace-ready from day one — a populated traceId is what later links a log line to its span.
• IDs, not PII — userId: "u789", never names/emails.
• Bounded payloads — don't dump full request/response bodies; truncate or omit.
• Type consistency — duration as a number, not the string "0.77", so stores can aggregate it without casting.

Where Logs Live: The Lifecycle

Logs move through stages, and where they're stored depends on the stage.

1. Locally (short-term / transient). Files on disk (/var/log/, rotated by logrotate), or stdout/stderr in containerized apps. Containers are ephemeral, so this is never the final destination.

2. Centralized hot stores (primary destination). Where logs actually "live" for querying:

• Elasticsearch / OpenSearch — indexed for fast full-text search (the ELK stack).
• Grafana Loki — cheap; uses object storage as its backend.
• Splunk — proprietary indexed storage, enterprise.
• Cloud-native — AWS CloudWatch Logs, GCP Cloud Logging, Azure Monitor / Log Analytics.
• SaaS — Datadog, New Relic, Sumo Logic.

3. Object storage (long-term / archival). Cheap, durable retention for compliance: AWS S3 (Glacier for cold archive), GCP Cloud Storage, Azure Blob. Logs age out of hot storage into here.

4. Specialized cases. Time-series DBs for metric-like logs; data warehouses (Snowflake, BigQuery) when logs feed analytics.

Typical flow:

App (stdout/file)
      │
   Collector (Fluent Bit / Vector / OTel Collector)
      │
Hot store (Elasticsearch / Loki / CloudWatch)   ← searchable for days/weeks
      │
Cold store (S3 / Glacier)                        ← archived for months/years

In short: logs are written locally, shipped to a centralized searchable store for active use, then archived to cheap object storage for retention.

What Format Are Logs Stored In?

Logs are generally not stored in traditional relational tables. The format depends on the store.

The key distinction: Elasticsearch indexes every field (rich but heavy), Loki indexes only labels (cheap but coarser search). Tables only come into play when logs are loaded into analytics/warehouse systems for SQL querying.

The Grafana Stack, Demystified

These tools are constantly confused because they share an ecosystem but do very different jobs. The fundamental split is logs vs metrics:

• Logs = text records of events ("Payment failed for user X").
• Metrics = numeric measurements over time (CPU = 80%, requests = 1200/sec).

How they fit together:

LOGS path:
  App logs → Promtail (collect) → Loki (store) ─┐
                                                 ├→ Grafana (visualize)
METRICS path:                                    │
  App /metrics → Prometheus (scrape + store) ───┘

Related tools you'll hear about:

Note: Promtail is being deprecated in favor of Grafana Alloy, the newer unified collector.

Corrected Mental Model

The full pipeline is instrument (create) → collect (ship) → store → visualize, split by signal type. The "create" step matters: something has to produce the signal before a collector can ship it — and for traces that producer is mandatory.

Logs vs Metrics vs Traces — The Three Pillars

Logs and traces are both observability signals, but they answer different questions:

• Logs answer "What happened?" — discrete events at a point in time.
• Traces answer "Where did it happen and how long did it take?" — the journey of a single request across services.

A log is a timestamped record of one discrete event. Standalone, rich in detail (error messages, stack traces). No inherent connection between entries unless you add a trace_id.

A trace represents the end-to-end path of a single request across multiple services. A trace is made of spans — each span is one unit of work (a service call, a DB query) with a start time and duration.

Trace: checkout request (trace_id: abc123)  ── total: 450ms
├─ span: API Gateway          [ 20ms]
├─ span: Auth Service         [ 50ms]
├─ span: Payment Service      [300ms]  ← bottleneck
│   └─ span: DB query         [280ms]  ← root cause
└─ span: Notification Service [ 80ms]

They're complementary, linked by a shared trace_id:

1. Trace shows: Payment Service span failed, took 300ms.
2. Grab the trace_id from that span.
3. Search logs filtered by that trace_id.
4. Logs reveal: ERROR: card declined - insufficient funds.

The trace tells you where to look; the logs tell you what actually happened there. Together with metrics (how much / how often), they form the three pillars of observability:

Worked Example: Is This a Log or a Trace?

Here's a real line from an app called res-crm:

{"timestamp":"2026-06-24T08:50:34.763Z","level":"info","message":{"action":"request-response","request":{"method":"GET","url":"/","ip":"::ffff:10.20.28.62","traceId":""},"response":{"code":200},"metrics":{"duration":"0.77"},"appName":"res-crm","environment":"development"}}

This is a log — specifically a structured (JSON) log. It's a single, self-contained record of one event (one HTTP request/response). That's the defining characteristic of a log. A trace would instead be a collection of connected spans with parent-child relationships and per-span durations.

What gives it away:

The interesting part: it's trace-ready. It carries traceId (currently empty) and metrics.duration (span-like timing). Populate that traceId via OpenTelemetry and the same request could appear in a trace and link straight to this log line. The trace version would look like:

Trace (traceId: abc123) — GET /  — 0.77ms
└─ span: res-crm HTTP handler [0.77ms]
   ├─ http.method: GET
   ├─ http.url: /
   └─ http.status_code: 200

On its own, though, it's one event, one record, no spans — a log.

Capturing Request Duration with Promtail

A common question: "Can I capture log + duration with only Promtail?"

Promtail doesn't create the log — your app does. Promtail is a shipper/collector, not a generator. The duration, traceId, method, etc. were written by your application (here, a logging middleware/interceptor). In a Node.js (NestJS/Express) app:

// Simplified middleware logic
const start = performance.now();
res.on('finish', () => {
  const duration = (performance.now() - start).toFixed(2);
  logger.info({
    action: 'request-response',
    request: { method, url, ip, traceId },
    response: { code: res.statusCode },
    metrics: { duration },   // ← duration created HERE, by your app
    appName: 'res-crm',
  });
});

So yes, Promtail captures duration — because it's already a field in the JSON. Promtail just ships the whole line to Loki, duration included.

By default Loki only indexes labels, so to query/aggregate on duration, parse it in Promtail's pipeline stages:

scrape_configs:
  - job_name: res-crm
    static_configs:
      - targets: [localhost]
        labels:
          job: res-crm
          __path__: /var/log/res-crm/*.log

    pipeline_stages:
      # 1. Parse the JSON log line
      - json:
          expressions:
            level: level
            duration: 'message.metrics.duration'
            traceId: 'message.request.traceId'
            statusCode: 'message.response.code'

      # 2. Promote useful fields to labels (use sparingly!)
      - labels:
          level:

      # 3. (Optional) Turn duration into a metric
      - metrics:
          request_duration:
            type: Histogram
            source: duration
            config:
              buckets: [0.1, 0.5, 1, 2, 5]

• json extracts nested fields (note the dot path message.metrics.duration).
• labels makes fields filterable in LogQL — only low-cardinality fields like level; never traceId or userId, high cardinality kills Loki.
• metrics exposes duration as a Prometheus-style metric.

Query it in LogQL even without making it a label:

# Filter slow requests
{job="res-crm"} | json | message_metrics_duration > 1

# Average duration over 5 minutes
avg_over_time({job="res-crm"} | json | unwrap message_metrics_duration [5m])

Caveat: this is still a log. Even with duration extracted, you have a log with a duration field — not a trace. Promtail/Loki cannot create traces. Real traces (spans across services, parent-child timing) require an OpenTelemetry SDK in your app exporting spans to Grafana Tempo (or Jaeger). That's what populates the empty traceId and builds the span tree.

Connecting Logs to Claude Code via MCP (Token-Optimized)

If you want Claude Code to query your logs over MCP, response size = token cost. The store and query language matter a lot.

ClickHouse deserves a special mention. It's columnar, compresses logs 10x+, and its official mcp-clickhouse server (run read-only) lets Claude Code issue precise SQL. SQL gives tight control over output, which is exactly what keeps token counts low:

-- Project only needed columns, filter precisely, cap rows
SELECT timestamp, level, message
FROM logs
WHERE service = 'payment' AND level = 'error'
  AND timestamp > now() - INTERVAL 1 HOUR
LIMIT 50;

-- Aggregate server-side → returns a tiny summary, not raw logs
SELECT level, count() FROM logs GROUP BY level;

Token-optimization tips, regardless of backend:

• Filter at the source — constrain by time range, service, and level in the query, not after.
• Limit result count — request 10–50 lines, not thousands.
• Project only needed fields — message + timestamp often suffices; skip full documents.
• Aggregate first — ask for counts/patterns before pulling raw logs.

Bottom line: for lean log tailing use Loki via the official Grafana MCP; for analytical investigation use ClickHouse via mcp-clickhouse — server-side aggregation and column projection keep Claude Code's context small either way.

Takeaways

• Structure your logs (JSON), use consistent levels, and never log secrets.
• Logs live locally → centralized hot store → archived object storage; they're stored as documents/indexes, not relational tables (tables appear only in warehouses).
• The Grafana stack splits by signal: Promtail/Alloy → Loki (logs), exporters → Prometheus (metrics), OTel → Tempo (traces), all visualized in Grafana.
• Logs, metrics, and traces are the three pillars — linked by trace_id, they take you from "something's wrong" to "here's exactly why".
• Promtail ships and parses logs (including a duration field), but cannot create traces — that needs OpenTelemetry + Tempo.
• For Claude Code over MCP, choose Loki for lean tailing or ClickHouse for token-cheap analytical queries, and always filter/aggregate at the source.