Lesson 5 of 10

AI as a TSE

Used well, AI tools compress the first draft of triage, summarization and RCA writing from minutes to seconds. Used carelessly, they hand a customer a confident, wrong answer.

Force multiplier, not force of truth

The value is speed on the first draft. The risk is treating that draft as verified fact.

A drafting assistant, not an oracle

AI tools are strong at summarizing a noisy ticket thread, drafting a first-pass explanation, or generating diagnostic questions. They are not a substitute for checking a live system, a spec, or your own runbook.

Hallucination is confident, fluent, and wrong

A model can state a made-up parameter or the wrong meaning of a status code with the same confidence as a correct answer. Fluency is not evidence — verify claims the same way you would a junior teammate’s draft.

Ground it in real evidence

Paste the actual log line, request, or error — not a description of the problem from memory. The output is only as accurate as the context you provide.

Data handling still applies

Anything that identifies a customer or exposes a secret — account numbers, tokens, contact details, payment data — gets scrubbed before it goes into a general-purpose tool, or handled only in approved, enterprise-grade tooling.

You own the verification step

A drafted RCA, customer reply, or triage summary is a starting point. Checking it against the actual evidence before it ships is still your job, not the model’s.

Exercises

Look at the sanitized ticket, log, and AI-drafted summary below, then work through both exercises.

Ticket: "Messages started failing around 2pm, we didn't change anything."

Log excerpt:
14:02:11 POST /v1/messages -> 400 {"error":{"code":"invalid_recipient"}}
14:02:14 POST /v1/messages -> 400 {"error":{"code":"invalid_recipient"}}
14:02:19 POST /v1/messages -> 400 {"error":{"code":"invalid_recipient"}}

AI-drafted summary:
"The customer is hitting a 500 Internal Server Error caused by a platform
outage. Recommend escalating to the on-call engineer immediately."

Concept check

Answer in your head first, then click a question to reveal the answer.

Scenario checklist

Work through the sample AI-drafted triage summary and check each item off. Progress is saved in your browser.

Compare every factual claim in the draft against the actual log/evidenceConfirm the status code and error the draft cites match what’s in the logNote the customer’s own assumption without accepting it as factDecide whether the drafted recommendation is actually the fastest path hereRedact or remove anything sensitive before this text leaves an internal tool

Go deeper

These primers are a starting point. Read the primary sources to build real fluency.

Anthropic: Prompt engineering overviewHow to write clear, structured prompts for tasks like log summarization Anthropic: Reducing hallucinationsConcrete techniques for grounding a model’s output in real evidence OWASP Top 10 for LLM ApplicationsRisks like hallucination and sensitive information disclosure to watch for NIST AI Risk Management FrameworkA neutral, government framework for trustworthy, accountable AI use

Training Notes

What building this project taught me.

The fifth topic is a tool, not a protocol — the lesson shape still held

HTTP, Auth, Events, and RCA are all "how the system behaves" topics. This one is "how you work," but the same primer → resources → concept-check → scenario-checklist structure fit without forcing it — a sign the pattern generalizes rather than being HTTP-specific.

The scenario intentionally contains a wrong answer

The sample AI-drafted summary in this lesson's exercise cites the wrong status code on purpose. The point isn't to distrust AI output by default — it's to build the habit of checking a specific, falsifiable claim against the log before accepting it.