Research note · Applied GenAI · Markets
Behavioral Diagnostic Engine
A synthetic-to-production case study in AI-assisted risk coaching for portfolio managers.
RESEARCH NOTE
July 2026
Alex Behm · Research notes
Bias recovery100% · 0 FP
Fine-tune cost~$0.70
Base → adapter9.0 → 9.5
Inference$0 local
Stack Qwen3-8B QLoRA · GCP L4 train · Ollama GGUF serve · math-first eval

Thesis. Math can detect portfolio-manager behavioral bias from trade ledgers alone. An LLM should only narrate a structured scorecard — in language a risk committee will actually read — not hunt for patterns in raw blotter data.

1. Problem

Portfolio managers exhibit predictable behavioral biases: disposition effect (selling winners too early, holding losers too long), overconfidence (excess turnover), and cost-basis anchoring (refusing to sell below purchase price). These biases destroy risk-adjusted returns, but they are rarely measured systematically. Most risk reports flag P&L and VaR; they do not flag why the P&L looks the way it does.

2. Approach

Synthetic Market → Control & Biased Traders → Deterministic Evaluator → Scorecard JSON → Fine-Tuned Coach (Qwen3-8B QLoRA) → CRO Memo

Every stage is deterministic and auditable except final coach generation. The LLM is not asked to find bias — it receives a structured scorecard and narrates it.

Biases detected (v1)

BiasMetricDetection
Disposition effectPGR / PLRRealized gains ≫ realized losses; paper losses held
OverconfidenceAnnualized turnoverExcess trading vs baseline; trade count
Cost-basis anchoringUnderwater share + sell-below-costPositions held underwater; zero sells below cost

Phase 3 gate (non-negotiable)

Before any LLM step, the evaluator had to pass a hard gate:

  • 100% recovery of all three injected biases on the biased trader
  • Zero false positives on the control trader
  • Detection uses only prices, positions, and trades — never injection tags

Result: PASS. Control clean; biased flagged on all three dimensions.

3. Fine-tuning

Dataset

  • 700 training pairs (scorecard → CRO memo)
  • 100 validation pairs
  • 2 gold real pairs (control + biased from Phase 3)
  • Labels: deterministic CRO memo writer (citation-faithful, $0 API)

Training

ParameterValue
Base modelQwen3-8B
Method4-bit QLoRA (HF PEFT)
GPUNVIDIA L4 (GCP, ~$0.70/hr)
Epochs3
Train / eval loss0.275 / 0.144
Train time~55 min
Trainable params43.6M / 8.2B (0.53%)

Local deploy   GGUF LoRA adapter (84 MB) → ollama create bde-coach. No cloud GPU required for inference.

4. Results — base vs fine-tuned

Same scorecard prompts, same hardware (local Ollama, AMD Radeon 890M iGPU).

CaseBase Qwen3-8BFine-tuned BDE coach
Biased (3 flags)10/10 quality10/10 quality
Control (clean)8/10 quality9/10 quality
Avg wall time~48s~27s

Heuristic scores are close because both models write diagnosis-like text. Domain correctness is not close:

BaseFine-tuned
FormatGeneric markdown reportCRO-style memo
PGR / PLRMisreads as “growth/loss rate”Correct disposition interpretation
Turnover 6.27Calls it “~6.3%”High annualized turnover
Control caseVerbose metric dumpClean “no material flags”

Sample output (biased case, condensed)

SUBJECT: Cost-basis Anchoring, Overconfidence / excess turnover, No-disposition discipline

DIAGNOSIS: PGR = 0.0462 · PLR = 0.0000 · turnover ≈ 6.27 (High) · underwater = 63.42% · sell-below-cost = 0.00%.

INTERVENTION: Prohibit get-even holds below cost; cap gross turnover; require edge thesis for re-entry within five sessions of profit-taking.

5. Economics

StageCost
Synthetic data generation$0 (deterministic)
Evaluator$0 (deterministic)
Dataset labeling$0 (deterministic CRO writer)
Fine-tuning (L4, ~1 hr)~$0.70–0.90
Inference (local)$0 (Ollama on existing hardware)
Cloud GPU total
< $1
Single L4 training run
API spend
$0
No frontier labels at scale
Ongoing serve
$0
Local after train
Vendor lock-in
None
Open weights + local stack

6. What this proves

#ClaimEvidence
1Synthetic data works for bias detection100% injected-bias recovery, 0 false positives — no real PM ledgers required to gate the detector
2Deterministic eval before LLM narrationAuditable path: wrong memo → wrong scorecard → specific metric flag
3Fine-tuning changes behavior, not just scoresCRO form + correct PGR/PLR; ~1.7× faster than base
4Local inference is viableFull pipeline after training runs on one machine with no cloud dependency

7. Limitations & next steps

ConstraintImplication
v1 biases onlyHerding, recency, loss aversion remain for v2
Single-path market5 names × 252 days; multi-path MC would stress-test robustness
Synthetic traders onlyReal PM ledgers needed for generalizability claims
Single-turn coachNo multi-turn coaching dialogue yet

Artifacts

ArtifactLocation
Research note (source MD)01_Research_Note.md
Sample scorecards + memosSample_Outputs/ · site assets/bde/
Phase gates (0–6b)docs/PHASE*_GATE.md
Training datasetdata/datasets/ (802 pairs)
Fine-tuned adapterartifacts/model/bde-coach-v1/ (PEFT)
Local Ollama modelbde-coach:latest

Built July 2026. All code, data, and model weights are local. No API keys, no vendor lock-in, no ongoing cloud costs.