How do paired test cases reveal bias?

Each set is identical except for one demographic attribute. If your AI produces systematically different outcomes — different scores, tone, or recommendations — that vary only with that attribute, it points to disparate treatment worth investigating.

Does passing these tests prove my AI is fair?

No. Passing means these specific cases did not surface a disparity. Real fairness evaluation also needs representative data, intersectional testing, statistical analysis across many cases, and domain expertise. Treat this as a fast first screen.

What is demographic parity?

Demographic parity is the idea that outcomes should not depend on protected attributes when those attributes are irrelevant to the task. These counterfactual prompts test for it by holding everything else constant.

Should I test intersections?

Yes, where it matters. Bias can appear at the intersection of attributes (for example age and gender together) even when each alone looks fine. Generate sets for combined attributes in addition to single ones.

Does my task description leave the browser?

No. The prompts are generated locally from templates and built-in attribute lists. Nothing you enter is uploaded or stored.

Bias Test Case Generator

Bias test case generator

If your AI screens applicants, drafts recommendations, or scores requests, you want to know whether it treats people differently based on attributes that should not matter. The bias test case generator produces paired prompts — identical except for one varied demographic attribute — so you can run them through your system and look for output that changes when only that attribute changes.

How it works

You describe the task and pick which protected attributes to vary: gender, race or ethnicity, age, or nationality. For each attribute the tool generates a matched set of prompts where the attribute cycles through representative values while everything else stays fixed. You run each set against your AI and compare results. A systematic difference that tracks only the varied attribute is a signal of disparate treatment to investigate. Generation is local; nothing is uploaded.

Tips and notes

Compare within a set. The signal is the difference across the matched prompts, not any single output.
Aggregate over many cases. One pair is anecdotal; run many and look for consistent patterns.
Test intersections. Bias can hide where two attributes combine even when each looks clean alone.
A clean screen is not proof. Pair this with statistical analysis, representative data, and domain expertise.