supportdesk/agent_assist.py:23
summarize_for_agent
Produce a two-sentence English summary of a support ticket for a human agent.
Output contract
Exactly 2 short sentences in English. Sentence 1: what the customer wants. Sentence 2: the key details (order ID, dates, amounts) when the ticket has them. Must not invent details that are not in the ticket.
no cheaper model passed the checks
Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.
Models
How each model did
| Model | Pass rate | Judge avg | Tokens in / out | Latency | Cost / call | vs baseline | Check |
|---|---|---|---|---|---|---|---|
qwen2.5:7bbaseline premium tier | 59.1% (13/22) | 3.64 / 5 | 115 / 38 | 2.52 s | $0.000668 | n/a | n/a |
qwen2.5:3b standard tier | 22.7% (5/22) | 2.41 / 5 | 115 / 41 | 1.27 s | $0.000111 | 38.5% | fail |
qwen2.5:1.5b budget tier | 0.0% (0/22) | 1.14 / 5 | 115 / 66 | 1.38 s | $0.000057 | 0.0% | fail |
qwen2.5:0.5b nano tier | 4.5% (1/22) | 1.41 / 5 | 115 / 40 | 0.41 s | $0.000028 | 7.7% | fail |
- 3b keeps 38% of baseline quality, needs 95%
- 1.5b keeps 0% of baseline quality, needs 95%
- 0.5b keeps 8% of baseline quality, needs 95%
Prompt
What the call sends
Model in code: qwen2.5:7b
Eval examples
3 of 22 cases, every model side by side
›sum-017b3b1.5b0.5b
Inputs
- subject
- Refund request for ORD-10421
- body
- Hi, I ordered a pair of wireless headphones on 2026-09-02 (order ORD-10421). They're unused and still in the box, but I changed my mind. Can I get a refund?
Expected
standard change-of-mind refund with order ID and date (T001)
Outputs
judge 1/5: Output contains only one sentence instead of the required two.
judge 2/5: Sentences are present but the first omits the order ID and the second omits the unused/in-original-packaging detail.
judge 1/5: The response is not exactly two short sentences; it uses headings and bullet points, violating the required format.
judge 1/5: Output is only one sentence, missing order ID and item condition details required by the rubric.
›sum-027b3b1.5b0.5b
Inputs
- subject
- Where is my package??
- body
- Order ORD-10588 placed on 2026-08-28 still hasn't arrived. Tracking hasn't updated in 10 days. This is ridiculous. I want my money back.
Expected
non-delivery with refund demand (T002)
Outputs
judge 5/5: Meets all rubric criteria with exactly two correct sentences and no extra details.
judge 1/5: Sentence 1 does not state the refund request as required.
judge 1/5: The response is not exactly two short sentences and includes bullet points and extra details, violating the rubric.
judge 1/5: The output does not mention the refund request or the required order details, violating the rubric.
›sum-037b3b1.5b0.5b
Inputs
- subject
- App crashes on login
- body
- Every time I open the SupportDesk Store app on my phone and tap Log in, it crashes. I've reinstalled it twice. App version 4.2.1.
Expected
technical bug, no order details (T003)
Outputs
judge 5/5: Meets all rubric criteria: exactly two short sentences, first states desire to fix login crash, second includes version 4.2.1 and two reinstalls, no extra info.
judge 1/5: The response has three sentences, includes irrelevant details, and omits mention of the two reinstalls, violating the rubric.
judge 1/5: Output is not two short sentences; it uses bullet points and extra formatting, not meeting the rubric.
judge 2/5: Only two sentences, but second sentence omits the required app version and first sentence doesn't explicitly state the desire for the login crash to be fixed.
All cases