Adversarial Thesis Testing

Does forcing bull and bear cases change an AI judge more than one balanced research memo with the same length budget?
Alex Behm · August 2026 · draft v0.3 · Project Mantis

Read this first. This study does not say which setup is “right about the market.” The theses are synthetic demos. I only measure whether the judge’s probability moves when the research setup changes (verdict displacement).

1. The problem

Teams increasingly ask language models for a verdict on an investment claim. A popular fix for one-sided answers is to force a bull case and a bear case first. That sounds rigorous. It is not obviously better than one researcher who must cover both sides on the same token budget.

Two confounds matter:

Length. More text can look like more reasoning even when structure is irrelevant.
Tone authorship. If you cast the bear as a short seller, the number may move because you rewrote the prompt, not because “debate topology” works.

2. Setup

Every comparison uses the same thesis text, the same frozen document folder, and the same judge instructions and output form.

Setup (plain name)	What the judge receives	Why it exists
Naive lab label: A	Thesis only. No research memo.	“Just ask the model” baseline.
Both-sides memo lab label: B	One researcher must argue for and against (~5,000 tokens).	Controls for length / deliberation.
Split bull + bear lab label: C	Separate bull and bear writers (~2,500 each), combined for the judge.	The “adversarial” treatment, same total budget as the memo.

Writers use a local 8B model (qwen3:8b). The judge is a different model family so writers are not graded by their own style. Grid: 3 demo theses × 5 intensity settings × 3 setups × 10 repeats = 450 runs (444 clean, 1.3% error).

Architecture diagram of three research setups feeding one judge — Figure 1. Same thesis and documents feed three research setups, then the same judge form.

3. Results

Average probability by setup

Mean judge probability by research setup — Figure 2. Naive ~0.37 · both-sides memo ~0.46 · split bull/bear ~0.45.

Paired shifts (same thesis, setting, and repeat)

Comparison	n	Mean shift	Spread (σ)
Both-sides memo − Naive	149	+0.087	0.125
Split debate − Naive	145	+0.075	0.140
Split debate − Both-sides memo	145	−0.012	0.092

Paired probability shifts between setups — Figure 3. Research moves the judge. Split debate does not beat the equal-budget memo.

Intensity: the real dial inside split debate

Holding the bull at “advocate” and only making the bear harsher moves mean probability from about 0.51 (mild bear) to 0.37 (short-seller bear), roughly 14 points.

Judge probability falls as bear intensity rises — Figure 4. Prompt authorship moves the number more than debate topology did.

4. Kill-condition quality (50-run sample)

Each verdict must name a primary falsifier (a concrete condition that would kill the thesis). I scored a stratified sample of 50 outputs:

Specific: concrete threshold/event, not vibes
Grounded in the given docs: fair given the frozen folder
Publicly checkable in form: settleable later from prices / filing-style lines / announced events

After a first pass plus a conservative automated rule pass: 100% / 94% / 68% on those three; 64% pass all three. This audit is not dual-human inter-rater reliability; methods are in the coding audit JSON.

Falsifier quality pass rates — Figure 5. Naming a condition is easy. Making it settleable is harder.

5. Interpretation

Across three demo theses and 450 runs, a research packet moves the judge about 8–9 probability points versus asking cold. Splitting that packet into bull and bear does not beat one equal-budget both-sides memo. Making the bear meaner moves the split-debate setup by about 14 points. The required “kill condition” field almost always produces a specific, on-document sentence; only about two-thirds look publicly settleable.

6. Limits

No accuracy claim. Synthetic demos only.
Parallel/blind writers only. No live rebuttal round, tools, or open web.
Three theses. Consistent pattern, not a 40-name book.
Falsifier audit is a 50-run sample with a rule-based second pass.

7. Reproduce

cd ~/Projects/ai-portfolio-projects/adversarial-thesis-testing
env -u PYTHONPATH uv run pytest -q
env -u PYTHONPATH uv run python scripts/analyze_phase2.py
env -u PYTHONPATH uv run python scripts/coding_second_pass.py

Companion study page: projects/att.html · Machine-readable stats: phase3_summary.json (internal analysis snapshot of the full grid; not a separate experiment).