supportdesk/triage.py:12

classify_category

found by asteasyexacttextmax_tokens 5temperature 0

Classify a support ticket into exactly one of six predefined categories.

Output contract

Exactly one lowercase word from the set {billing, shipping, technical, account, refund, other}; no punctuation, no extra words; max_tokens is 5.

keepstays on qwen2.5:7b

no cheaper model passed the checks

$140.40

Before / month

$140.40

After / month

$0.00

Savings (0.0%)

20,000

Calls / day

Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.

Models

How each model did

Model	Pass rate	Tokens in / out	Latency	Cost / call	vs baseline	Check
qwen2.5:7bbaseline premium tier	84.0% (21/25)	86 / 2	0.33 s	$0.000234	n/a	n/a
qwen2.5:3b standard tier	64.0% (16/25)	86 / 2	0.17 s	$0.000037	76.2%	fail
qwen2.5:1.5b budget tier	72.0% (18/25)	86 / 2	0.12 s	$0.000014	85.7%	fail
qwen2.5:0.5b nano tier	48.0% (12/25)	86 / 2	0.05 s	$0.000009	57.1%	fail

3b keeps 76% of baseline quality, needs 95%
1.5b keeps 86% of baseline quality, needs 95%
0.5b keeps 57% of baseline quality, needs 95%

Prompt

What the call sends

Model in code: qwen2.5:7b

system

You are a support ticket classifier.

user

Classify this support ticket into exactly one category: billing, shipping, technical, account, refund, other. Reply with the category name only. Ticket: {ticket_text}

Eval examples

3 of 25 cases, every model side by side

›cat-017b3b1.5b0.5b

Inputs

ticket_text: Refund request for ORD-10421 Hi, I ordered a pair of wireless headphones on 2026-09-02 (order ORD-10421). They're unused and still in the box, but I changed my mind. Can I get a refund?

Expected

refund

change-of-mind refund request, unused item (T001)

Outputs

qwen2.5:7bpass

refund

match

qwen2.5:3bpass

refund

match

qwen2.5:1.5bpass

refund

match

qwen2.5:0.5bpass

refund

match

›cat-027b3b1.5b0.5b

Inputs

ticket_text: Where is my package?? Order ORD-10588 placed on 2026-08-28 still hasn't arrived. Tracking hasn't updated in 10 days. This is ridiculous. I want my money back.

Expected

refund

non-delivery with explicit money-back ask overrides shipping label (T002)

Outputs

qwen2.5:7bfail

shipping

expected 'refund', got 'shipping'

qwen2.5:3bfail

billing

expected 'refund', got 'billing'

qwen2.5:1.5bpass

refund

match

qwen2.5:0.5bpass

refund

match

›cat-037b3b1.5b0.5b

Inputs

ticket_text: App crashes on login Every time I open the SupportDesk Store app on my phone and tap Log in, it crashes. I've reinstalled it twice. App version 4.2.1.

Expected

technical

app crash bug (T003)

Outputs

qwen2.5:7bpass

technical

match

qwen2.5:3bpass

Technical

match

qwen2.5:1.5bpass

technical

match

qwen2.5:0.5bpass

technical

match

All cases

Pass and fail across all 25 cases

qwen2.5:7b

21/25

qwen2.5:3b

16/25

qwen2.5:1.5b

18/25

qwen2.5:0.5b

12/25

pass fail no resultHover a square for the case id.