OrchestrAI: one helper, five features

What Bob found, for 0.46 Bobcoins

• Split the helper into 5 features, one per module that calls the LLM. code_planner has no function of its own and is dispatched through a generic fallback.
• Resolved every model: gpt-4-0613 from config.yml for four modules, and a per-module override to gpt-3.5-turbo-16k for debugger in the pipeline YAML.
• Rebuilt every prompt the way the app assembles it at runtime, including the two modules that build their own user message.
• Found the volume multipliers: debugger retries up to 3 times per run, and modify_codebase loops until the user stops.

The bill static analysis cannot see

downshift estimate docs/case-study/orchestrai/downshift.audit.json \
  --base docs/case-study/orchestrai/downshift.scan.json \
  -c docs/case-study/orchestrai/downshift.yaml --completion-tokens 1500

What this shows

1. 01Static analysis alone underestimates this bill about 4.5x. It sees one call where the app makes five, seven with debugger retries.
2. 0295% of projected spend is four features on gpt-4-0613, one of OpenAI's most expensive legacy models.
3. 03The author already downshifted once, by hand: debugger runs on gpt-3.5-turbo-16k. On gpt-4-0613 it would cost about $1,734 a month instead of $119. Downshift makes that call per feature, backed by evals.
4. 04A third-party pricing tracker lists gpt-4-0613 for deprecation on Oct 23, 2026. Downshift's eval step is how you would pick each replacement.

Assumptions, stated plainly

• Projection, not a bill: OpenAI list prices (checked Sep 26, 2026) x assumed volume of 200 pipeline runs a day.
• Output assumed at 1,500 tokens per call because no module sets max_tokens. Input counts only the static system prompts, so it is a floor.
• No evals were run on this repo (paid API calls, $0 budget), so there are no downgrade recommendations here. The SupportDesk demo shows the full loop.
• Only engineering_pipeline was audited. The four other pipelines use the same helper and pattern.

Full write-up, scan, audit and config on GitHub

mem0: 14 call sites, 4 real features

What Bob found, for 3.32 Bobcoins

• Split the OpenAI adapter into the 4 features that use it: fact extraction on add(), procedural memory, image description (vision only) and search reranking.
• Resolved every model and setting: gpt-5-mini by default, and a separate reranker config with its own temperature and token limit.
• Rebuilt the prompts, and noted that the async add and procedural paths send the same prompts as the sync ones.
• Two fix-ups by script, no Bob loop: the 13 untouched scan entries were merged back, and two long prompt constants Bob left as placeholders were copied from source.

The bill static analysis cannot see

downshift estimate docs/case-study/mem0/downshift.audit.json \
  --base docs/case-study/mem0/downshift.scan.json \
  -c docs/case-study/mem0/downshift.yaml

What this shows

1. 01Static analysis sees plumbing, not features: 14 call sites, 13 of them provider adapters or examples. The default setup runs 4 features, and cost belongs to features.
2. 02Static-only pricing underestimates this bill about 7.6x.
3. 03The reranker calls the LLM once per candidate document. At 10 candidates, 10,000 searches become 100,000 LLM calls: $675 a month, 29% of the bill, from one for loop.
4. 04Fact extraction re-sends a 33,653-character prompt on every add(). Input alone is about $382 of that feature's $1,582, a strong candidate for prompt caching or a cheaper model.

Assumptions, stated plainly

• Projection, not a bill: OpenAI list prices (checked Sep 26, 2026) x assumed volume of 10,000 adds and 10,000 reranked searches a day.
• Output is priced at max_tokens, a ceiling, so feature costs are an upper bound. Tokens are estimated from words; runtime text is not counted.
• No evals were run on this repo (paid API calls, $0 budget), so there are no downgrade recommendations here.
• Default setup only: the other provider adapters, the integration and the example were not audited or priced.

Full write-up, scan, audit and config on GitHub