Support tooling
Paste a support ticket and Claude will classify it — category, severity, a one-line summary, a suggested first response, and whether it needs engineering escalation.
Every prompt change is scored against 22 hand-labeled tickets. The same prompt runs on two providers, so the results compare models directly. Latest category accuracy:
What building this project taught me.
The first scores looked good at the default temperature. Pinning temperature to 0 made them drop, which showed the earlier numbers were partly luck. A lower score you can reproduce beats a higher one you cannot, because only a stable score can tell you whether a prompt change helped.
Twice in the first version, the model was right and the hand-written gold label was wrong. Later, when Claude and GPT gave the same wrong answer on the same ticket, the label became the first suspect. An eval checks your data as much as your model.
Retrieval was tested on the same tickets with and without the knowledge base. It helped on 3 of 5, and its real wins were fixing 2 confident factual errors: a wrong claim about SSO and error codes from the wrong product. It never made an answer worse, and when no article fit, the model cited none.
An agent decides which tools to call, so some code paths only run when the model chooses them. Forcing the iteration cap to 1 confirmed the loop fails cleanly with its tool log intact. The tool log is the main debugging artifact: it shows whether a wrong answer came from a skipped tool or a bad conclusion.
With the same prompt, Claude scored 68% on category and GPT scored 45%. But the rubric was refined against Claude's mistakes, so part of that gap is the prompt, not the model. A fair comparison names what it was tuned on.
The triage package talks to a small TriageProvider interface instead of a specific SDK. Anthropic and OpenAI each get an adapter with their own retry rules, and both receive a byte-identical prompt, so the comparison measures the model and not the plumbing.
The runs dashboard reads saved JSON run files during next build and ships static HTML. Nothing reads the file system at request time, which avoids monorepo file-tracing problems on Vercel and makes the pages instant.
The page wrapper (page.tsx) stays a Server Component for metadata and layout. The form itself becomes a Client Component (TriageForm.tsx) because it needs useState and event handlers. Only the leaf that needs interactivity opts in to 'use client'.
The browser never calls Claude. It posts the ticket to the Next.js route at /api/triage, which runs the shared triage package in a Node.js serverless function. The API keys live in Vercel's environment settings for that project and never reach the client. Those settings are read when a deployment is created, so a new or changed key needs a redeploy.
For triage classification, a complete JSON object is easier to parse than a stream. Streaming makes more sense for long-form responses where showing partial output improves perceived speed.
