Lesson 1 of 3

Observability & Log Analysis

An error spike shows up in three different dashboards during an incident. Correlating them fast — not staring at each one separately — is the actual skill.

The tools

Four platforms cover most of what you'll be handed access to on day one.

Datadog

Unified logs, metrics, and traces with strong out-of-box dashboards — common at product-led companies.

Splunk

The original enterprise log-search platform — a powerful query language (SPL), common in larger and regulated environments.

Prometheus

Pull-based metrics collection with its own query language (PromQL) — the default in Kubernetes-native stacks, usually paired with Grafana.

CloudWatch / Azure Monitor

The cloud provider's native, first-party observability tool — the default when you haven't set up anything else yet.

Correlating a spike across dashboards

A timestamp alone can't tell two concurrent requests apart. A trace ID can.

A trace or correlation ID is generated once per request and threaded through every log line, metric, and span it touches, across every service it passes through. When an error log, a latency alert, and an error-rate alert all reference the same service and time window, they are very likely the same underlying incident — the trace ID is how you confirm it instead of guessing from timing alone.

Exercises

Look at the sanitized log line and alerts below, then work through both exercises.

{"level":"error","ts":"2026-07-22T14:03:11Z","trace_id":"9f2a1c...","service":"checkout","msg":"payment provider timeout","latency_ms":30021}

[Alert] p99 latency for checkout > 5000ms (14:02–14:05 UTC)
[Alert] error rate for checkout > 2% (14:02–14:06 UTC)

Concept check

Answer in your head first, then click a question to reveal the answer.

Scenario checklist

Work through the sample log line and alerts and check each item off. Progress is saved in your browser.

Go deeper

These primers are a starting point. Read the primary sources to build real fluency.

Training Notes

What building this project taught me.

This module opens the trail, but isn't itself the flagged gap

Modern data stack fluency (the next two lessons) was the named gap in the job-hunting scan — this lesson is the prerequisite vocabulary: correlating a log line, a metric, and a trace by a shared ID.

Header and field names vary by tool

The example below uses generic field names (level, trace_id) — real tools use their own conventions (Datadog's dd.trace_id, Splunk's indexed fields), but the underlying correlation concept is identical.