supportdesk/agent_assist.py:34

draft_reply

found by asthardjudgetextasyncmax_tokens 250temperature 0

Draft a customer-facing reply to a support ticket in the customer's detected language.

Output contract

Plain-text reply under 120 words, in the language given by the ISO 639-1 code in {language}. Must not promise refunds, credits, delivery dates or specific timeframes (for example '24-48 hours'). Must say that a teammate will confirm next steps.

keepstays on qwen2.5:7b

no cheaper model passed the checks

$531.75

Before / month

$531.75

After / month

$0.00

Savings (0.0%)

20,000

Calls / day

Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.

Models

How each model did

Model	Pass rate	Judge avg	Tokens in / out	Latency	Cost / call	vs baseline	Check
qwen2.5:7bbaseline premium tier	13.6% (3/22)	2.77 / 5	153 / 50	3.26 s	$0.000886	n/a	n/a
qwen2.5:3b standard tier	9.1% (2/22)	2.05 / 5	153 / 24	0.87 s	$0.000100	66.7%	fail
qwen2.5:1.5b budget tier	4.5% (1/22)	2.14 / 5	153 / 45	1.13 s	$0.000050	33.3%	fail
qwen2.5:0.5b nano tier	0.0% (0/22)	1.59 / 5	153 / 39	0.43 s	$0.000031	0.0%	fail

3b keeps 67% of baseline quality, needs 95%
1.5b keeps 33% of baseline quality, needs 95%
0.5b keeps 0% of baseline quality, needs 95%

Prompt

What the call sends

Model in code: qwen2.5:7b

system

You are a friendly, concise customer support agent for SupportDesk Store. Never promise refunds, credits, or delivery dates. Say a teammate will confirm next steps. Keep replies under 120 words.

user

Reply language (ISO 639-1): {language} Agent summary: {summary} Customer subject: {subject} Customer message: {body}

Eval examples

3 of 22 cases, every model side by side

›dft-017b3b1.5b0.5b

Inputs

language: en
summary: The customer wants a refund for order ORD-10421. The order was placed on 2026-09-02 for unused wireless headphones still in the original box.
subject: Refund request for ORD-10421
body: Hi, I ordered a pair of wireless headphones on 2026-09-02 (order ORD-10421). They're unused and still in the box, but I changed my mind. Can I get a refund?

Expected

Written in English. Under 120 words. Addresses the change-of-mind refund request for ORD-10421. Says a teammate will confirm next steps. Does not promise a refund, credit, delivery date or any timeframe.

English change-of-mind refund — must not promise refund (T001)

Outputs

qwen2.5:7bfailjudge 2/5

Thank you for reaching out. We can certainly assist with your refund request. A teammate will review your order and get back to you with next steps. Please expect a response within 24-48 hours.

judge 2/5: It omits the specific order number and provides a response timeframe, violating the rubric's no‑timeframe rule.

qwen2.5:3bpassjudge 5/5

I understand your request for refund. Our team will confirm next steps for ORD-10421.

judge 5/5: Meets all rubric criteria: English, concise, addresses request, mentions teammate, no promises.

qwen2.5:1.5bfailjudge 2/5

Hi there! Thank you for reaching out. We'd be happy to assist with your refund request. Please provide your order number and any additional details you might have, such as the reason for your change of mind. Once we have this information, we can proceed with the refund process.

judge 2/5: It omits mentioning ORD-10421 and does not state a teammate will confirm next steps, though it is in English, under 120 words, and avoids promises.

qwen2.5:0.5bfailjudge 2/5

The customer wants a refund for order ORD-10421. The order was placed on 2026-09-02 for unused wireless headphones still in the original box.

judge 2/5: The reply only repeats the request and omits a response or mention of a teammate confirming next steps.

›dft-027b3b1.5b0.5b

Inputs

language: en
summary: The customer wants a refund for a missing package. The order ORD-10588 was placed on 2026-08-28 and tracking has not updated in 10 days.
subject: Where is my package??
body: Order ORD-10588 placed on 2026-08-28 still hasn't arrived. Tracking hasn't updated in 10 days. This is ridiculous. I want my money back.

Expected

Written in English. Under 120 words. Addresses the missing package and refund request for ORD-10588. Says a teammate will confirm next steps. Does not promise a refund, credit, delivery date or any timeframe.

English non-delivery refund demand — must not promise refund or delivery date (T002)

Outputs

qwen2.5:7bfailjudge 3/5

I understand your frustration. I'll forward your order details to our logistics team for a status update. They'll get back to you with more information. Thank you for your patience.

judge 3/5: Meets language, length, and no promises criteria, but fails to explicitly address the customer's refund request

qwen2.5:3bfailjudge 2/5

I understand your concern. Your order is being reviewed. A team member will contact you shortly to discuss next steps.

judge 2/5: The response is in English, short, and mentions a teammate, but it fails to reference the missing package, the refund request, or the specific order number.

qwen2.5:1.5bfailjudge 3/5

I'm sorry to hear that your package hasn't arrived yet. We'll need to confirm the next steps with a teammate to ensure your refund request is processed correctly. Please provide your order number and any additional details you might have, and we'll take it from there.

judge 3/5: The response is in English, under 120 words, mentions a teammate and avoids promises, but fails to reference the specific order ORD-10588.

qwen2.5:0.5bfailjudge 2/5

The customer is experiencing a delay in tracking updates for their order, which is causing confusion and frustration. They are seeking a refund for the missing package.

judge 2/5: It omits the order number and does not mention a teammate confirming next steps, though it is English, brief, and avoids promises.

›dft-037b3b1.5b0.5b

Inputs

language: es
summary: The customer wants a refund for a damaged lamp. The order ORD-10812 was placed on 2026-07-30.
subject: Solicitud de reembolso
body: Hola, compré una lámpara (pedido ORD-10812) el 2026-07-30 y llegó rota. ¿Pueden devolverme el dinero?

Expected

Written in Spanish. Under 120 words. Addresses the refund request for the damaged lamp in order ORD-10812. Says a teammate will confirm next steps. Does not promise a refund, credit, delivery date or any timeframe.

Spanish ticket — reply must be in Spanish (T006)

Outputs

qwen2.5:7bfailjudge 2/5

¡Entendido! Lamento que tu lámpara llegara dañada. Un equipo de soporte revisará tu caso. Pronto recibirás un correo con más detalles. Gracias por tu paciencia.

judge 2/5: The reply is in Spanish and short, but it omits the order number, does not explicitly say a teammate will confirm next steps, and gives a vague timeframe.

qwen2.5:3bfailjudge 2/5

Entendido, un equipo de servicio al cliente se pondrá en contacto con usted para resolver esto.

judge 2/5: Does not mention the damaged lamp or order number, so does not address the refund request.

qwen2.5:1.5bfailjudge 2/5

Hola, gracias por contactar con nosotros. Confirmaremos la solicitud de reembolso para el pedido ORD-10812. Un equipo técnico se encargará de revisar la situación y procederá con la devolución. Una confirmación de la próxima etapa se proporcionará en breve.

judge 2/5: Meets language, length, and order reference but incorrectly promises a refund and does not explicitly state a teammate will confirm next steps.

qwen2.5:0.5bfailjudge 2/5

El pedido de lámpara ORD-10812 fue devuelto por el cliente Solicitud de reembolso.

judge 2/5: Meets language and length but does not address the refund request properly nor mention a teammate will confirm next steps.

All cases

Pass and fail across all 22 cases

qwen2.5:7b

3/22

qwen2.5:3b

2/22

qwen2.5:1.5b

1/22

qwen2.5:0.5b

0/22

pass fail no resultHover a square for the case id.