Static analysis vs Bob
Config-driven code blinds static analysis
The ast scanner found 7 call sites and resolved 6 of their models. It cannot see through shared helpers or config lookups.
The Bob auditor started from the scan, read only the unresolved and shared call sites, and returned 8 logical call sites with every model resolved. It split the shared llm.py::ask helper into the two features that use it and found the hidden model in lang_of.
After Bob moved every model name into models.yaml, a fresh ast scan resolves 0 of 7 models. Config-driven code is normal in production, which is exactly why the auditor exists.
| Metric | ast | Bob | ast after refactor |
|---|---|---|---|
| Call sites | 7 | 8 | 7 |
| Models resolved | 6 | 8 | 0 |
| Prompts resolved | 5 | 8 | 5 |
| Enriched | 0 | 8 | 0 |
| Found by Bob | 0 | 3 | 0 |
| Split from helpers | 0 | 2 | 0 |
Changes
What the auditor changed
- enrichedsupportdesk/agent_assist.py::draft_replypurpose, output_contract, difficulty, grading
- enrichedsupportdesk/agent_assist.py::summarize_for_agentpurpose, output_contract, difficulty, grading
- enrichedsupportdesk/extract.py::extract_order_infopurpose, output_contract, difficulty, grading
- splitsupportdesk/llm.py::askinto supportdesk/triage.py::detect_sentiment, supportdesk/triage.py::tag_urgency
- resolvedsupportdesk/misc_utils.py::lang_ofmodel resolved: qwen2.5:7b; prompt resolved
- enrichedsupportdesk/policy.py::decide_refundpurpose, output_contract, difficulty, grading
- enrichedsupportdesk/triage.py::classify_categorypurpose, output_contract, difficulty, grading
- addedsupportdesk/triage.py::detect_sentimentvia supportdesk/llm.py::ask
- addedsupportdesk/triage.py::tag_urgencyvia supportdesk/llm.py::ask
Call sites
Side by side
ast scan · 7 sites
- supportdesk/agent_assist.py::summarize_for_agentqwen2.5:7b · prompt resolved
- supportdesk/agent_assist.py::draft_replyqwen2.5:7b · prompt resolved
- supportdesk/extract.py::extract_order_infoqwen2.5:7b · prompt resolved
- supportdesk/llm.py::askqwen2.5:7b · prompt not resolved
- supportdesk/misc_utils.py::lang_ofmodel unresolved · prompt not resolved
- supportdesk/policy.py::decide_refundqwen2.5:7b · prompt resolved
- supportdesk/triage.py::classify_categoryqwen2.5:7b · prompt resolved
Bob audit · 8 sites
- supportdesk/agent_assist.py::draft_replyqwen2.5:7b
- supportdesk/agent_assist.py::summarize_for_agentqwen2.5:7b
- supportdesk/extract.py::extract_order_infoqwen2.5:7b
- supportdesk/misc_utils.py::lang_offound by Bobqwen2.5:7b
- supportdesk/policy.py::decide_refundqwen2.5:7b
- supportdesk/triage.py::classify_categoryqwen2.5:7b
- supportdesk/triage.py::detect_sentimentfound by Bobvia supportdesk/llm.py::ask · qwen2.5:7b
- supportdesk/triage.py::tag_urgencyfound by Bobvia supportdesk/llm.py::ask · qwen2.5:7b