How it was built
From one model call to a model comparison
Five stages, each one working and runnable on its own. No stage counted as done because the types compiled. Each one had to pass a real test outside the code first.
v1
Baseline with a scored eval
- What it added
- One model call that returns typed JSON: category, severity, summary, first response, escalate. API errors retry with backoff. Malformed output gets re-prompted with a correction. The pipeline never throws.
- How it was proven
- A scorer against 20 hand-labeled tickets. Pinning temperature to 0 made the scores drop, which showed the earlier numbers were partly luck. After fixing the rubric and two wrong labels: 90% category, 90% severity exact.
v2
Grounded in a knowledge base (RAG)
- What it added
- 15 support articles embedded with Voyage. Each ticket retrieves its top 3 by cosine similarity, and the model returns the ids of the articles it actually used.
- How it was proven
- The same 5 tickets with and without retrieval, read side by side. Retrieval helped on 3, including fixing 2 confident factual errors (an SSO detail and the wrong product's error codes). It was neutral on 2 and never made an answer worse.
v3
A tool-using agent
- What it added
- The model decides which tools to call: KB search, account lookup, ticket history, escalation. A loop runs the calls and feeds results back, capped at 6 turns. Every call is logged.
- How it was proven
- The exit paths were tested, not just the happy path. Forcing the cap to 1 produced a clean max_iterations failure with the tool log intact: no hang, no crash.
v4
Tools behind a real MCP server
- What it added
- The account, history and escalation tools moved behind a Model Context Protocol server over stdio. The agent discovers them at runtime instead of hardcoding them.
- How it was proven
- The server was verified alone in the MCP Inspector before the agent touched it. The first live run through MCP filed the project's first real escalation.
v5
Anthropic vs OpenAI, side by side
- What it added
- An interface in front of the model with one adapter per vendor, so the pipeline does not know which model it calls. Both get a byte-identical prompt. Every scored run is saved and browsable.
- How it was proven
- Saved runs on 22 labeled tickets, compared ticket by ticket in a static dashboard, with a prompt hash so runs that used different prompts are never compared by accident.
Where it stands
Latest category accuracy on 22 labeled tickets, same prompt for both models:
- openai · gpt-5.5
- 45%
- anthropic · claude-sonnet-4-6
- 68%
The honest caveat: the prompt was tuned on Claude, so part of the gap is the prompt, not the model. And when both models give the same wrong answer, the gold label is the first suspect.