Observability for Node.js with OpenTelemetry, Grafana, Prometheus, Loki and Jaeger

"It works on my machine" stops being funny the first time production is slow and nobody can say why. Logs alone rarely answer the real questions: which request was slow, where the time went, and whether it's getting worse.

That's what observability is for. It rests on three signals:

• Traces - the path of a single request through your services, with timings for every step.
• Metrics - numbers over time: request rate, error rate, latency percentiles, memory.
• Logs - detailed events, ideally connected to the trace they belong to.

OpenTelemetry (OTel) is the open standard for producing all three. You instrument your code once, and you can send the data to any backend - so you never get locked into one vendor.

The stack

Apps send everything to the Collector over OTLP; the Collector routes each signal to its backend

• OTel SDK in each Node.js service produces traces and metrics.
• OTel Collector receives everything over OTLP, batches it, and forwards it.
• Jaeger stores traces, Prometheus stores metrics, Loki stores logs.
• Grafana shows all three in one place, linked together.

The Collector in the middle is the important design choice. Apps only know one address. Swapping Jaeger for Grafana Tempo, or adding a hosted backend later, becomes a Collector config change - no redeploy of your services.

1. Instrument the Node.js service

Install the SDK, the auto-instrumentations and the OTLP exporters:

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/sdk-metrics

Create instrumentation.mjs:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
 
const sdk = new NodeSDK({
  serviceName: 'orders-api',
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
    exportIntervalMillis: 15_000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
  ],
});
 
sdk.start();
 
process.on('SIGTERM', async () => {
  await sdk.shutdown();
  process.exit(0);
});

Load it before your app code, so the instrumentations can patch modules as they're imported:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
node --import ./instrumentation.mjs dist/server.js

That's it. Without touching your handlers, you now get spans for incoming HTTP requests, outgoing fetch calls, and queries to PostgreSQL, MongoDB, Redis and more. I disable the fs instrumentation because it creates a lot of noise for little value.

2. Add the spans that matter

Auto-instrumentation shows the plumbing. For business logic, add your own spans:

import { trace, SpanStatusCode } from '@opentelemetry/api';
 
const tracer = trace.getTracer('orders');
 
export async function checkout(cartId: string) {
  return tracer.startActiveSpan('checkout', async (span) => {
    span.setAttribute('cart.id', cartId);
    try {
      const order = await createOrder(cartId);
      span.setAttribute('order.total', order.total);
      return order;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Keep attribute values low-cardinality where you can. cart.id on a span is fine, but never use user IDs or URLs with IDs as metric labels - every unique value creates a new time series and Prometheus memory explodes.

3. Configure the Collector

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  batch: {}
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
 
exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]

Jaeger accepts OTLP natively, Loki 3 has a native OTLP endpoint, and Prometheus scrapes the Collector on port 8889. Run the Collector (otel/opentelemetry-collector-contrib), Jaeger, Prometheus, Loki and Grafana with Docker Compose, and add the three backends as data sources in Grafana.

4. Connect logs to traces

The real power shows up when signals link to each other. If every log line carries the trace_id of the request, you can jump from a slow trace straight to its logs - and back.

With Pino, @opentelemetry/instrumentation-pino (included in the auto-instrumentations) injects trace_id and span_id into every log record automatically. In Grafana, add a derived field on the Loki data source that turns trace_id into a link to Jaeger.

Now debugging looks like this: an alert fires on p99 latency → open the slow traces → see the span that took 2 seconds → click through to the exact logs for that request.

5. What to measure first

Don't build fifty dashboards. Start with the RED method for every service:

• Rate - requests per second.
• Errors - failed requests per second.
• Duration - latency, as p50, p95 and p99 (averages hide the slow tail).

The HTTP instrumentation already records http.server.request.duration as a histogram, so these come almost for free. Alert on error rate and p99 latency, not on CPU - users feel slowness and errors, not CPU.

Sampling: keep the bill sane

Tracing every request is fine at low traffic. At scale, it's expensive. Two options:

• Head sampling in the SDK - keep, say, 10% of traces: OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1.
• Tail sampling in the Collector - decide after the trace completes, so you can keep 100% of errors and slow requests while dropping most of the boring, fast ones.

Tail sampling is what you want eventually, but it needs all spans of a trace to reach the same Collector, so start with head sampling.

Takeaways

• Instrument once with OpenTelemetry; choose backends later.
• Put a Collector between apps and backends.
• Auto-instrumentation gets you 80% of the value in ten minutes.
• Link logs to traces with trace_id - it changes how you debug.
• Start with RED metrics and alert on what users feel.