Downshift · Cut LLM costs per PR

Pipeline

From call site to cost diff

01ScanDownshift
downshift scan
Static analysis finds every OpenAI-compatible call site, its model and prompt.
02AuditBob
Downshift Auditor mode
Bob resolves hidden models, splits shared helpers into features, and adds purpose, output contract, difficulty and grading.
03Write evalsBob
Eval Writer mode or downshift evalgen
20 to 25 cases per call site, including edge cases, grounded in the refund policy.
04RunDownshift
downshift run
Every eval on the baseline and each cheaper model, with tokens and latency recorded.
05GradeDownshift
downshift rescore
Exact match, JSON fields, or an independent LLM judge for free text.
06Decide and reportDownshift
downshift report
Cheapest model that keeps quality wins, with a projected monthly cost.
07ApplyBob
Agent mode
Bob moves model names into config, applies the decisions and opens the PR with the report.
08Guard every PRDownshift
downshift diff (GitHub Action)
Comments the projected cost change on every pull request and can fail the check.

Data flow

your repo | v downshift scan ....... call sites found by static analysis | v Bob auditor .......... hidden models resolved, shared helpers split | v evals ................ Bob Eval Writer, or downshift evalgen with any model | v downshift run ........ baseline + cheaper models, graded by a judge | v downshift report ..... cheapest safe model and projected cost per call site | -> Bob applies the decisions and opens a PR v downshift diff ....... GitHub Action comments the cost change on every PR

Where Bob fits

Downshift is the toolkit, Bob is the brain

Bob does the parts that need real code understanding: auditing call sites, writing evals, applying the decisions and reviewing pull requests. Scanning, running, grading, costing and the CI diff are deterministic Python that runs without Bob.

Task	Bob feature	What it did	Bobcoins
Project setup	/init, AGENTS.md	Generated project rules for every mode	0.95
Architecture doc	Plan mode	Wrote docs/architecture.md with a Mermaid diagram	0.23
Audit SupportDesk	Custom mode + skill	7 ast sites to 8 logical call sites, all models resolved	0.32
Write evals	Custom mode + skill, document understanding	181 eval cases across 8 call sites	0.92
Build the report	Plan mode, then Agent mode	report.py, CLI command and snapshot tests from a spec	8.34
Apply decisions	Agent mode, commit + PR	models.yaml refactor, downgrades, opened PR #8	1.38
Apply mid tier	Agent mode, commit + PR	Switched two call sites to the 3B model, opened PR #11	0.27
Review a bad PR	Ask mode, context mentions	Found the hard-coded model and the max_tokens cost driver	0.07
Build the demo app	Agent mode, spec in context	Scaffolded the Next.js app from web/SPEC.md	5.68
Audit OrchestrAI	Custom mode + skill, context mentions	Real repo: 1 shared helper to 5 features, all models resolved	0.46
Audit mem0	Custom mode + skill, context mentions, steering	Real repo: 14 ast call sites to the 4 features that spend money	3.32

Decision rule

How a model gets picked

A candidate passes when its eval pass rate is at least the quality threshold times the baseline's pass rate.
It must also clear an absolute floor, so two equally bad models never count as safe.
Among passing candidates the cheapest one wins. Otherwise the call site keeps its current model.
Judge-graded cases pass at 4 of 5 or higher. Each model is priced with its own measured tokens.

threshold 95% · floor 80% · judge openai/gpt-oss-120b

Pricing

Illustrative prices

Model	Tier	Input / 1M tokens	Output / 1M tokens
qwen2.5:7b	premium	$2.50	$10.00
qwen2.5:3b	standard	$0.40	$1.60
qwen2.5:1.5b	budget	$0.15	$0.60
qwen2.5:0.5b	nano	$0.10	$0.40

Tier prices that map the local models to realistic API costs. Replace them with your provider's prices in downshift.yaml.

Dollar figures are projections: measured token counts x illustrative per-model prices x an assumed call volume, all set in downshift.yaml. They are not a real bill.