Fallacy-Bench

Which AI models make the most reasoning mistakes during an argument, and does each have a favourite?

We designed Fallacy-Bench to compare models through five-round debates. Each model takes opposing positions in separate conversations, answering the same questions under matched conditions. Topics combine controlled scenarios with material drawn from philosophy, news, Medium, X, and Reddit.

The evaluation catalogue draws on Wikipedia’s lists of fallacies and cognitive biases. Each entry includes a definition, explanation, and examples, covering errors such as strawman arguments, false dichotomies, sweeping generalisations, and “no true Scotsman.”

TypeSafe’s Jev assesses the transcripts through dedicated evaluation roles: Fallacy Jev, Heuristic Jev, Bias Jev, and Appeal Jev. These apply category-specific criteria to the complete exchange, distinguishing appropriate uses of heuristics and appeals from misuse.

Existing benchmarks helped clarify what we wanted to measure. CriticBench examines how models generate answers, critique reasoning, and make corrections across five reasoning domains. Our focus is the mistakes models introduce while defending a position, without being instructed to use faulty arguments. This makes the design particularly suited to examining argumentative habits that persist through disagreement.

We designed the analysis around a model-by-model breakdown of error types. Readers should be able to inspect which mistakes Jev flags, the passages behind those judgments, and how the pattern changes by topic, assigned position, and debate round. An overall ranking can show which model accumulates more errors; the breakdown can reveal whether those errors cluster around misrepresenting opponents, overgeneralising, or changing definitions when challenged.

Repeated debates and swapped positions help separate recurring tendencies from isolated responses. Reporting error rates alongside raw counts helps account for differences in answer length. Tracking what happens after a challenge adds another useful distinction: an initial mistake and a mistake repeatedly defended should remain visible as different behaviours.

The intended advantage is a more specific basis for comparing models used in discussion, analysis, and advice. A reader could examine whether a model repeatedly forces complex decisions into two options, or whether it corrects that tendency when questioned.

Limitations

The evaluation itself may introduce bias. Jev models may be better at recognising familiar, frequently discussed examples of logical fallacies and cognitive biases than obscure or unusual instances. Even when the evaluation criteria explicitly name a less common error, a model may fail to identify it when it appears in a complex debate transcript, particularly if confidence is low or its training gives limited coverage of that context. Such false negatives could distort error counts, category breakdowns, and any model ranking.

This risk cannot be ruled out by prompting alone. Expert human reviewers trained in critical thinking could provide an independent check and may identify mistakes that automated judges miss, but the labour costs of comprehensive professional review put that approach outside the current experiment’s scope. Accordingly, Jev's judgments should be treated as model-generated assessments, and the resulting comparisons will require validation before they can support claims about relative accuracy.

What comes next

Results are pending. A follow-up article will present the model comparisons and supporting examples once the assessment is complete and the evaluation method has been validated.