supportdesk/policy.py:22
decide_refund
found by asthardjson_fieldsjsonmax_tokens 220temperature 0
Apply the store refund policy to a ticket and return a structured eligibility decision.
Output contract
JSON object with exactly four keys in order: reasoning (string, 1-3 sentences), is_refund_request (boolean), eligible (one of "yes", "no", "store_credit", "need_info"), policy_section (string like "2.1", or null). When is_refund_request is false, eligible must be "no" and policy_section must be null. Graded fields: is_refund_request, eligible, policy_section, which must match the refund policy applied to the order date and today's date. reasoning is not graded.
keepstays on qwen2.5:7b
no cheaper model passed the checks
$1,212.72
Before / month
$1,212.72
After / month
$0.00
Savings (0.0%)
20,000
Calls / day
Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.
Models
How each model did
| Model | Pass rate | Tokens in / out | Latency | Cost / call | vs baseline | Check |
|---|---|---|---|---|---|---|
qwen2.5:7bbaseline premium tier | 40.0% (10/25) | 477 / 83 | 6.82 s | $0.002021 | n/a | n/a |
qwen2.5:3b standard tier | 40.0% (10/25) | 477 / 80 | 2.64 s | $0.000318 | 100.0% | fail |
qwen2.5:1.5b budget tier | 28.0% (7/25) | 477 / 82 | 1.95 s | $0.000121 | 70.0% | fail |
qwen2.5:0.5b nano tier | 0.0% (0/25) | 477 / 70 | 0.75 s | $0.000076 | 0.0% | fail |
- 3b pass rate 40% is below the floor 80%
- 1.5b keeps 70% of baseline quality, needs 95%
- 0.5b keeps 0% of baseline quality, needs 95%
Prompt
What the call sends
Model in code: qwen2.5:7b
system
You are a refund policy assistant. Apply the policy exactly as written. Count days carefully between the order date and today's date.
user
Refund policy:
{policy}
Today's date: {today}
Extracted order info: {order_info_json}
Ticket subject: {subject}
Ticket message:
{body}
Return JSON with these keys, in this order:
"reasoning": 1-3 sentences applying the policy,
"is_refund_request": true or false,
"eligible": "yes", "no", "store_credit", or "need_info",
"policy_section": the section number you applied, e.g. "2.1".
If it is not a refund request, set eligible to "no" and policy_section to null.
Eval examples
3 of 25 cases, every model side by side
›pol-017b3b1.5b0.5b
Inputs
- policy
- # SupportDesk Store Refund Policy (v3) This is a fictional policy for a fictional store, used only as demo data. ## 1. Standard returns 1.1 Physical items can be returned for a full refund within 30 days of the order date. 1.2 Returned items must be unused and in their original packaging. ## 2. Damaged or defective items 2.1 Items that arrive damaged or defective are eligible for a full refund or replacement within 60 days of the order date. 2.2 Damage reported more than 60 days after the order date is not eligible. ## 3. Exceptions 3.1 Digital products (e-books, software licenses, gift cards) are non-refundable once downloaded or activated. 3.2 Items bought on clearance sale are eligible for store credit only, not a cash refund. 3.3 If an order has not been delivered 14 days after the order date, the customer is eligible for a full refund. ## 4. Missing information 4.1 If the order ID or order date is missing, the refund cannot be decided. Ask the customer for the missing details.
- today
- 2026-09-20
- order_info_json
- { "order_id": "ORD-10421", "order_date": "2026-09-02" }
- subject
- Refund request for ORD-10421
- body
- Hi, I ordered a pair of wireless headphones on 2026-09-02 (order ORD-10421). They're unused and still in the box, but I changed my mind. Can I get a refund?
Expected
{
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1.1"
}
change-of-mind, unused in original packaging, 18 days — within 30-day limit (T001)
Outputs
qwen2.5:7bpass
{
"reasoning": "The order was placed on 2026-09-02, and the item is unused and in its original packaging. According to section 1.1 of the refund policy, physical items can be returned for a full refund within 30 days of the order date.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1.1"
}
all fields match
qwen2.5:3bpass
{
"reasoning": "The order was placed on September 2, 2026, and the customer is requesting a refund for unused items within the 30-day return period.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1.1"
}
all fields match
qwen2.5:1.5bpass
{
"reasoning": "The order was placed on 2026-09-02, which is within the 30-day return period. The items are unused and in their original packaging, so they can be returned for a full refund.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1.1"
}
all fields match
qwen2.5:0.5bfail
{
"reasoning": "The customer ordered a pair of wireless headphones on 2026-09-02 (order ORD-10421) and they are in the box, but they changed their mind and want a refund.",
"is_refund_request": true,
"eligible": "no",
"policy_section": 2
}
eligible: expected 'yes', got 'no'; policy_section: expected '1.1', got 2
›pol-027b3b1.5b0.5b
Inputs
- policy
- # SupportDesk Store Refund Policy (v3) This is a fictional policy for a fictional store, used only as demo data. ## 1. Standard returns 1.1 Physical items can be returned for a full refund within 30 days of the order date. 1.2 Returned items must be unused and in their original packaging. ## 2. Damaged or defective items 2.1 Items that arrive damaged or defective are eligible for a full refund or replacement within 60 days of the order date. 2.2 Damage reported more than 60 days after the order date is not eligible. ## 3. Exceptions 3.1 Digital products (e-books, software licenses, gift cards) are non-refundable once downloaded or activated. 3.2 Items bought on clearance sale are eligible for store credit only, not a cash refund. 3.3 If an order has not been delivered 14 days after the order date, the customer is eligible for a full refund. ## 4. Missing information 4.1 If the order ID or order date is missing, the refund cannot be decided. Ask the customer for the missing details.
- today
- 2026-10-05
- order_info_json
- { "order_id": "ORD-09877", "order_date": "2026-06-12" }
- subject
- Refund please
- body
- Order ORD-09877 from 2026-06-12, a blender. I never used it. I want a refund.
Expected
{
"is_refund_request": true,
"eligible": "no",
"policy_section": "1.1"
}
change-of-mind, unused, but 115 days — far past 30-day window (T012)
Outputs
qwen2.5:7bfail
{
"reasoning": "The blender can be returned for a full refund since it is an unused physical item and the return is within 30 days of the order date.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1.1"
}
eligible: expected 'no', got 'yes'
qwen2.5:3bfail
{
"reasoning": "The order date is June 12, 2026, and the current date is October 5, 2026. The customer is requesting a refund for a blender that was never used, which falls under the 'Standard returns' section of the refund policy.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1.1"
}
eligible: expected 'no', got 'yes'
qwen2.5:1.5bfail
{
"reasoning": "The order was placed on June 12, 2026, and the customer wants a refund for a blender that they never used. This falls under the 'Physical items can be returned for a full refund within 30 days of the order date' section of the policy. The customer has not exceeded the 30-day return period, so they are eligible for a full refund.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "1"
}
eligible: expected 'no', got 'yes'; policy_section: expected '1.1', got '1'
qwen2.5:0.5bfail
{
"reasoning": "The customer never used the blender and wants a refund.",
"is_refund_request": true,
"eligible": "no",
"policy_section": 2
}
policy_section: expected '1.1', got 2
›pol-037b3b1.5b0.5b
Inputs
- policy
- # SupportDesk Store Refund Policy (v3) This is a fictional policy for a fictional store, used only as demo data. ## 1. Standard returns 1.1 Physical items can be returned for a full refund within 30 days of the order date. 1.2 Returned items must be unused and in their original packaging. ## 2. Damaged or defective items 2.1 Items that arrive damaged or defective are eligible for a full refund or replacement within 60 days of the order date. 2.2 Damage reported more than 60 days after the order date is not eligible. ## 3. Exceptions 3.1 Digital products (e-books, software licenses, gift cards) are non-refundable once downloaded or activated. 3.2 Items bought on clearance sale are eligible for store credit only, not a cash refund. 3.3 If an order has not been delivered 14 days after the order date, the customer is eligible for a full refund. ## 4. Missing information 4.1 If the order ID or order date is missing, the refund cannot be decided. Ask the customer for the missing details.
- today
- 2026-09-19
- order_info_json
- { "order_id": "ORD-10588", "order_date": "2026-08-28" }
- subject
- Where is my package??
- body
- Order ORD-10588 placed on 2026-08-28 still hasn't arrived. Tracking hasn't updated in 10 days. This is ridiculous. I want my money back.
Expected
{
"is_refund_request": true,
"eligible": "yes",
"policy_section": "3.3"
}
not delivered, 22 days after order — eligible under section 3.3 (T002)
Outputs
qwen2.5:7bpass
{
"reasoning": "Since your order was placed on 2026-08-28 and it has not been delivered 14 days later, you are eligible for a full refund according to our policy.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "3.3"
}
all fields match
qwen2.5:3bfail
{
"reasoning": "The order has not been delivered for 14 days after the order date, which qualifies the customer for a full refund under section 3.3 of the refund policy.",
"is_refund_request": true,
"eligible": "full refund",
"policy_section": "3.3"
}
eligible: expected 'yes', got 'full refund'
qwen2.5:1.5bfail
{
"reasoning": "The order was placed on August 28, 2026, and the tracking hasn't updated in 10 days. This is more than 14 days after the order date, so the customer is eligible for a full refund.",
"is_refund_request": true,
"eligible": "yes",
"policy_section": "3.1"
}
policy_section: expected '3.3', got '3.1'
qwen2.5:0.5bfail
{"reasoning": "The order has not arrived and the tracking hasn't updated in 10 days, which is considered a significant delay. This is a clear indication that the customer's package is missing or damaged.", "is_refund_request": true, "eligible": "store_credit", "policy_section": 2}
eligible: expected 'yes', got 'store_credit'; policy_section: expected '3.3', got 2
All cases
Pass and fail across all 25 cases
qwen2.5:7b
10/25
qwen2.5:3b
10/25
qwen2.5:1.5b
7/25
qwen2.5:0.5b
0/25
pass fail no resultHover a square for the case id.