supportdesk/triage.py:54

tag_urgency

found by Bobvia supportdesk/llm.py::askeasyexacttextmax_tokens 3temperature 0

Rate the urgency of a support ticket as low, medium, or high based on its category and content.

Output contract

Exactly one lowercase word from the set {low, medium, high}; caller strips trailing period and lowercases; max_tokens is 3.

keepstays on qwen2.5:7b

no cheaper model passed the checks

$201.07

Before / month

$201.07

After / month

$0.00

Savings (0.0%)

20,000

Calls / day

Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.

Models

How each model did

Model	Pass rate	Tokens in / out	Latency	Cost / call	vs baseline	Check
qwen2.5:7bbaseline premium tier	72.7% (16/22)	126 / 2	0.51 s	$0.000335	n/a	n/a
qwen2.5:3b standard tier	50.0% (11/22)	126 / 2	0.21 s	$0.000054	68.8%	fail
qwen2.5:1.5b budget tier	40.9% (9/22)	126 / 2	0.12 s	$0.000020	56.3%	fail
qwen2.5:0.5b nano tier	27.3% (6/22)	126 / 2	0.05 s	$0.000013	37.5%	fail

3b keeps 69% of baseline quality, needs 95%
1.5b keeps 56% of baseline quality, needs 95%
0.5b keeps 37% of baseline quality, needs 95%

Prompt

What the call sends

Model in code: qwen2.5:7b

system

You are a support operations lead.

user

This is a '{category}' support ticket. Rate its urgency as low, medium, or high. High: money lost, service down for many users, or a very angry repeat contact. Medium: an order or account problem that blocks the customer. Low: questions, feedback, and requests with no time pressure. Reply with one word. Ticket: {ticket_text}

Eval examples

3 of 22 cases, every model side by side

›urg-017b3b1.5b0.5b

Inputs

category: technical
ticket_text: App crashes on login Every time I open the SupportDesk Store app on my phone and tap Log in, it crashes. I've reinstalled it twice. App version 4.2.1.

Expected

medium

technical bug blocking one user — medium (T003)

Outputs

qwen2.5:7bpass

Medium

match

qwen2.5:3bfail

Low

expected 'medium', got 'low'

qwen2.5:1.5bfail

High

expected 'medium', got 'high'

qwen2.5:0.5bfail

High

expected 'medium', got 'high'

›urg-027b3b1.5b0.5b

Inputs

category: technical
ticket_text: Checkout page down Nobody on our team can check out right now, the payment page returns error 500. We're a business account and need to place a large order today.

Expected

high

service down for many users, large order impact (T020)

Outputs

qwen2.5:7bpass

High

match

qwen2.5:3bpass

High

match

qwen2.5:1.5bpass

High

match

qwen2.5:0.5bfail

Medium

expected 'high', got 'medium'

›urg-037b3b1.5b0.5b

Inputs

category: billing
ticket_text: Charged twice I was charged twice for order ORD-10733 on 2026-09-15. Please remove the duplicate charge.

Expected

high

money lost (duplicate charge) — high (T004)

Outputs

qwen2.5:7bfail

Medium

expected 'high', got 'medium'

qwen2.5:3bfail

Medium

expected 'high', got 'medium'

qwen2.5:1.5bpass

High

match

qwen2.5:0.5bpass

High

match

All cases

Pass and fail across all 22 cases

qwen2.5:7b

16/22

qwen2.5:3b

11/22

qwen2.5:1.5b

9/22

qwen2.5:0.5b

6/22

pass fail no resultHover a square for the case id.