Downshift

Docs

Install, configure, run, and wire Downshift into CI

Install

pip install downshift

Python 3.10 or newer.

Quickstart

# 1. find every LLM call site (writes path/to/repo/.downshift/callsites.json) downshift scan path/to/repo # 2. write evals with any configured model (or the Bob Eval Writer mode) downshift evalgen --callsites path/to/repo/.downshift/callsites.json --out path/to/repo/.downshift/evals # 3. run the baseline and every cheaper model on the evals downshift run --callsites path/to/repo/.downshift/callsites.json # 4. cost and quality report with a decision per call site downshift report --callsites path/to/repo/.downshift/callsites.json --out report.md # 5. projected cost change of your branch vs main downshift diff path/to/repo --base main

Configuration

Put a downshift.yaml next to the code you scan. Every command also takes -c to point at another file.

# downshift.yaml, next to the code you scan version: 1 provider: base_url: "http://localhost:11434/v1" # any OpenAI-compatible endpoint api_key_env: null # env var holding the key, or null models: baseline: "qwen2.5:7b" # the model your code uses today candidates: ["qwen2.5:3b", "qwen2.5:1.5b", "qwen2.5:0.5b"] judge: null # grades free text; defaults to the baseline quality_threshold: 0.95 # fraction of baseline quality a candidate must keep min_pass_rate: 0.80 # floor: a candidate must also pass this share of cases pricing: # USD per 1M tokens, replace with your provider's "qwen2.5:7b": { input: 2.50, output: 10.00, tier: premium } "qwen2.5:3b": { input: 0.40, output: 1.60, tier: standard } "qwen2.5:1.5b": { input: 0.15, output: 0.60, tier: budget } "qwen2.5:0.5b": { input: 0.10, output: 0.40, tier: nano } volume: default_per_day: 20000 # calls per day, used for monthly projections per_call_site: {} # e.g. "app/triage.py::classify": 50000 scan: include: ["*.py"] exclude: ["tests/*"]

Commands

  • scanFind every LLM call site in a repository.
  • validateCheck a callsites or audit file against the schema.
  • compareCompare the ast scan with a Bob audit, side by side.
  • check-evalsCheck eval files against the call sites they test.
  • evalgenGenerate an eval set for each call site with a configured model.
  • runRun each call site's evals on the baseline and candidate models, and score them.
  • rescoreRe-grade saved outputs of judge-graded call sites with the configured judge.
  • reportRender the cost and quality report.
  • estimateProject monthly LLM cost from a scan or audit file. No evals needed.
  • diffShow the projected LLM cost change between two git refs.
  • exportExport JSON and the report for the demo web app.

Run downshift COMMAND --help for every option.

GitHub Action

Add this workflow to get a projected cost diff comment on every pull request. Remove fail-above to comment without ever failing the check.

.github/workflows/cost-diff.yml

name: Cost diff on: pull_request permissions: contents: read pull-requests: write issues: write jobs: cost-diff: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: ana-lan/downshift@v0.1.0 with: path: . fail-above: "500" github-token: ${{ secrets.GITHUB_TOKEN }}

Using Bob

  • The .bob/ folder ships two custom modes (Downshift Auditor, Downshift Eval Writer) and their skills (downshift-audit, downshift-evals). See docs/bob.md to use them in Bob IDE.
  • The auditor starts from downshift scan output and only reads unresolved or shared call sites, which keeps it cheap.
  • Everything except the audit and eval writing runs without Bob.

Methodology

  • Evals: 20 to 25 cases per call site, graded by exact match, JSON fields, or an LLM judge (pass at 4 of 5).
  • Models run locally with Ollama. Dollar figures use real token counts, illustrative tier prices and an assumed volume, all from downshift.yaml.
  • downshift diff is a static projection for per-PR deltas: unresolved models are priced as the baseline, missing max_tokens assume 256 output tokens, and prompts are counted from the resolved text. It is not an absolute spend estimate. downshift estimate uses the same projection on a single scan or audit file.

Limitations

  • Python only; detects the OpenAI v1 SDK and the Anthropic messages API.
  • Prices are illustrative and volumes are assumed; plug in your own.
  • Local Qwen 2.5 models on a laptop, not a production API benchmark.
  • Eval inputs are synthetic (SupportDesk demo data), not sampled from real traffic.

Future work

  • GitHub Marketplace listing for the Action.
  • downshift audit command that runs the auditor with any configured LLM.
  • Anthropic and Gemini runners, LiteLLM and LangChain detection, a JS/TS scanner.
  • Eval inputs sampled from real logs with PII redaction.