Which LLM calls are you overpaying for?
Downshift finds every LLM call site in a Python repo, writes evals for each one, tests cheaper models against the one you use today, and shows the cost impact of every pull request.
Downshift is the toolkit, IBM Bob is the brain.
Projected monthly cost · supportdesk
Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.
01 · Results · supportdesk
Three of eight call sites can run on a smaller model
| Call site | Decision | Model | Pass rate | Savings/mo |
|---|---|---|---|---|
| extract_order_info supportdesk/extract.py | downgrade | qwen2.5:7b → qwen2.5:3b | 95.2% → 100.0% | $282.04 |
| lang_of supportdesk/misc_utils.py | downgrade | qwen2.5:7b → qwen2.5:1.5b | 90.9% → 90.9% | $163.88 |
| detect_sentiment supportdesk/triage.py | downgrade | qwen2.5:7b → qwen2.5:3b | 90.9% → 90.9% | $118.73 |
| draft_reply supportdesk/agent_assist.py | keep | qwen2.5:7b | 13.6% | $0.00 |
| summarize_for_agent supportdesk/agent_assist.py | keep | qwen2.5:7b | 59.1% | $0.00 |
| decide_refund supportdesk/policy.py | keep | qwen2.5:7b | 40.0% | $0.00 |
| classify_category supportdesk/triage.py | keep | qwen2.5:7b | 84.0% | $0.00 |
| tag_urgency supportdesk/triage.py | keep | qwen2.5:7b | 72.7% | $0.00 |
- The floor did its job: on decide_refund the 3B model tied the 7B at 40%, and the 80% floor blocks both from counting as safe.
- Bigger is not always better: the 3B model scored below the 1.5B on classify_category and lang_of.
- Adding one mid-tier model (qwen2.5:3b) took projected savings from 5% to 18%.
- A self-graded 7B judge said draft_reply passed 100% of cases. An independent 120B judge found 14%.
02 · Static analysis vs Bob
Config-driven code blinds static analysis
The ast scanner found 7 call sites and resolved 6 of their models. It cannot see through shared helpers or config lookups.
The Bob auditor started from the scan, read only the unresolved and shared call sites, and returned 8 logical call sites with every model resolved. It split the shared llm.py::ask helper into the two features that use it and found the hidden model in lang_of.
| Metric | ast | Bob | ast after refactor |
|---|---|---|---|
| Call sites | 7 | 8 | 7 |
| Models resolved | 6 | 8 | 0 |
| Prompts resolved | 5 | 8 | 5 |
| Enriched | 0 | 8 | 0 |
| Found by Bob | 0 | 3 | 0 |
| Split from helpers | 0 | 2 | 0 |
03 · CI guardrail
Every PR gets a cost diff
A demo PR added one new LLM call with a hard-coded 7B model and max_tokens 1024, run on every ticket. The Downshift Action commented the projected monthly increase and failed the check. Bob then reviewed the PR: route the call through models.yaml, cut max_tokens, and evaluate the 3B model first.