The problem
When is a small, locally-run AI good enough for a narrow, repeatable task, and when is the premium for paid cloud models worth paying?
Public companies hold quarterly earnings calls. The interesting content is the Q&A: what was asked, how management answered, where answers were partial, and how much of the call was scripted versus live. I built an end-to-end extractor that produces a structured research PDF, then used that product as the measurement harness.
How it works
Ingest, preprocess, extract, score, report.
Preprocessing handles FactSet and Motley Fool transcript formats, splits management prepared remarks from analyst Q&A, and tags who is speaking. Extraction returns a six-field structured output. Scoring adds a credibility rating, how much of the call was live Q&A (versus scripted), analyst attribution, and evasion flags (answers that seem to dodge the question). PDFs are watermarked by model.
Training the AI
Base model: Qwen3-8B, an 8-billion-parameter open-weight model, adapted via 4-bit QLoRA (a memory-efficient training method) on a GCP NVIDIA L4. Version 1 used 4,448 pattern-matched labels and was unusable (training loss 3.67). Version 2 used 499 structured labels from a GPT-4o review and became the production path (loss 0.016, ~9.4 hours). Local serving via Ollama (a tool for running AI models locally) using a compressed model file on a mini PC for measured CPU runs.
Cost vs. speed trade-off
Publication figure used in the research brief. Bubble size and labels include evasion-flag density, the percentage of questions where the model surfaced something worth reviewing.
Qwen3-8B v2
- Time per call (CPU / GPU est.)
- ~205s / ~34s
- Evasion flags surfaced
- ~11%
- First-pass
- 8/10 (10/10 after retry)
GPT-5.5
- Time per call
- ~67s
- Evasion flags surfaced
- ~39%
- First-pass
- 10/10
Claude Fable 5
- Time per call
- ~55s
- Evasion flags surfaced
- ~33%
- First-pass
- 10/10
Quarterly volume → annual cloud cost
Projection from measured per-call costs. Drag the slider to explore volume scenarios.
Calls screened per quarter
1,000 / quarter
| Model | Projected annual API cost |
|---|---|
| Qwen (local) | $0 |
| GPT-5.5 | $0 |
| Fable 5 | $0 |
Annual cost = quarterly calls × 4 × cost per call. Local hardware and power excluded.
Decision framework
How I read the results: solid reference tables, not interactive widgets.
When to lean which way
| Situation | Lean toward |
|---|---|
| Low volume, vendor data sharing acceptable | GPT-5.5 |
| Need max speed/flags and QA budget exists | Fable 5 |
| High volume, privacy, or unit-cost control | Local Qwen (GPU if speed matters) |
| Unclear volume / early pilot | Start with a paid cloud model; reassess at ~500–1,000 calls per quarter |
Limits of this note
| Constraint | Implication |
|---|---|
| 10 companies tested | Directional, not large-sample proof |
| Credibility / evasion scores | Rule-of-thumb scores for review, not independently verified |
| GPU speed estimate | Estimated from throughput ratios, not re-run |
| Scope | Research note. Not a product claim. |
Main lesson from training: label quality beat label quantity. The extraction and scoring workflow matters more than which model you use.