The cross-review experiment: 32 models judging each other, and the size of the bias

What happens when models score models

This 2026 China native multimodal LLM tier list scores with human judgement plus public evidence. But “model evaluation” is rapidly shifting to another form: models as judges — GPT scoring Claude, Qwen scoring Doubao, a blind set of model outputs deciding the ranking.

That form has a question nobody has publicly measured: the judges have biases. Do models rate their own family higher? Do same-vendor models inflate each other? Do strong and weak judges use the same scale?

This site ran a reproducible cross-review: eight models as judges, blind-scoring outputs from all 32, measuring three biases. The finding first: the bias is structural, and both its direction and magnitude are predictable — which determines how any “models scoring models” board should be read.

Experiment design

• Judge set (8): Qwen3.8-Max, Doubao Seed 2.0 Pro, GLM-5.3, Kimi K3, DeepSeek-V4.1-Flash, InternVL 4.0, Ant Ling 2, NetEase Fuxi 2. Five vendors, four licence types, overall scores from 77.5 to 95.4.
• Subject set (32): all listed models, outputs uniformly trimmed to 500 characters with model names and vendor markers removed.
• Rubric: absolute 0–100 with a one-line reason; each judge scores each subject three times, median taken.
• Blind conditions: judges see only outputs — no model name, vendor or price. Self-preference can only leak through “output style”, which is part of the bias being measured.
• Aggregation: each subject gets eight judge scores; highest and lowest dropped, then averaged and compared against our human scores.

Scripts, prompts and rubrics ship with the site data and rerun after every expansion.

Bias one: self-preference — universal, but bounded

Method: for each judge, compare its average score for “own family” (same vendor, same series) against its average for other models.

Judge	Own-family premium	Max within family	Direction
Qwen3.8-Max	+3.8	+5.1 (to Qwen3.8-Flash)	Consistently high
Doubao Seed 2.0 Pro	+3.4	+4.6 (to Doubao’s generation)	Consistently high
GLM-5.3	+2.9	+4.1 (to GLM-5)	Consistently high
Kimi K3	+3.6	+4.8	Consistently high
DeepSeek-V4.1-Flash	+2.2	+3.5 (to DeepSeek-OCR 2)	Consistently high
InternVL 4.0	+1.8	+2.6	Consistently high
Ant Ling 2	+4.2	+5.1	Consistently high; small sample
NetEase Fuxi 2	+3.1	+4.3	Consistently high

Eight judges, eight consistent directions — self-preference is 100% present, averaging +3.2, maximum +5.1.

That magnitude cannot overturn a ranking (our method treats anything inside 0.5 as one band; 5 points moves roughly 8–10 places), but it can do three dangerous things: turn a tie into a first place, open “gaps” inside one band, and manufacture false discrimination between close candidates. When you see a models-scoring-models board where #1 leads #2 by 2 points, first ask whether that gap exceeds the judges’ self-preference.

Bias two: family bias — more dangerous than self-preference

Self-preference affects only one’s own family. Family bias affects “style similars”: judges tend to rate models whose output style resembles their own higher, even across vendors.

Method: for each judge, compute the gap between its scores for the five most style-similar and five least style-similar models (similarity from clustering output length, sentence entropy, Markdown usage and other surface features).

Judge	Family bias (similar − dissimilar)	Most concentrated dimension
Qwen3.8-Max	+4.6	Agents, documents
Doubao Seed 2.0 Pro	+4.1	Chinese, video description
GLM-5.3	+5.2	Documents, charts
Kimi K3	+3.8	Long-text structure
DeepSeek-V4.1-Flash	+2.6	Chart extraction
InternVL 4.0	+3.3	Perception description
Ant Ling 2	+4.4	Chinese long-form
NetEase Fuxi 2	+4.9	Generation tasks

Family bias averages +4.1, maximum +5.2, concentrated in the agent and document dimensions. Why those two? Because their scoring depends most on “format regularity” — JSON structure of function calls, field completeness of documents — and format regularity is precisely the easiest style feature to imitate. By contrast, family bias on video and generation is clearly smaller (< 2.5), because outputs differ widely and styles are hard to copy.

That is why family bias is more dangerous than self-preference: it acts not on “one’s own” but on “everything like oneself” — one vendor’s format preference gets amplified by every judge that imitates it, and the whole board tilts toward one output format. And the relationship between format and capability was always weaker than people assume.

Bias three: score compression — strong judges use narrower scales

Method: each judge’s score spread across the 32 models (highest minus lowest).

Judge	Overall rank	Score spread	Behaviour
Qwen3.8-Max	#1	11 points	Extreme compression: all 32 fall between 82 and 93
GLM-5.3	#4	14 points	High compression
Doubao Seed 2.0 Pro	#2	16 points	Compression
Kimi K3	#5	19 points	Moderate
DeepSeek-V4.1-Flash	#12	22 points	Moderate
InternVL 4.0	#16	26 points	Dispersed
Ant Ling 2	#22	29 points	Dispersed
NetEase Fuxi 2	#29	34 points	Extreme dispersion: all 32 fall between 51 and 85

Compression correlates with judge strength: the strongest judge spans 11 points, the weakest 34 — a 3× difference.

The explanation is direct: to a strong model every output is “fine”, and distinguishing differences requires a very fine scale, so it compresses 32 models into one mass; to a weak model good-bad differences are stark, so it opens a 34-point spread. This produces a counter-intuitive consequence: using the strongest model as judge flattens the middle and lower field into an indistinguishable mass — exactly hiding the gaps that most need separating. Using weak judges produces wide spreads but systematic offsets, equally distorting the ordering.

Folding the three biases into a discount table

Now we can answer “how should a models-scoring-models board be read”. Our approach gives each judge a discount factor:

Bias type	Average magnitude	Discount advice
Self-preference	+3.2	Subtract 3 from family scores before comparing
Family bias	+4.1	Apply a 10% discount to format-heavy dimensions (agents, documents)
Score compression	Strong judge spread 11	Within the top 10, a strong judge’s board has almost no discrimination

Combining all three: on any “models scoring models” board, rank differences inside the top five are probably smaller than the total judge bias. That does not make model cross-review useless — it remains effective for wide-band screening (top 10 versus bottom 20), because gaps above 30 points exceed all biases combined. What fails is precise ordering.

Cross-review and this tier list

This site does not use model cross-review scores for ranking, for exactly the reasons in the table above. But we use the cross-review results in two places:

• Error estimation. The reasonable uncertainty band on human scores is set by reference to the score distribution of same-span judges — one empirical source of the “±1.5 points” band for grade-B evidence on the methodology page.
• Format audit. In the agent and document dimensions where family bias concentrates, we additionally hand-audit the ratio of “format score” to “content correctness score”, preventing the ranking from tilting toward output format.

The tier list measures capability; the cross-review measures judging reliability. The latter is one error source of the former — the value of this matrix is that next time you see an “AI judge” board, you hold a discount factor.

Reproduction

Judge set, prompts, rubric, blind protocol and aggregation scripts ship with the site data; rerun after every expansion, with the bias matrix updated per version in the changelog. Subject outputs are uniformly trimmed to 500 characters with identifying features removed; each judge scores three times and takes the median; judges are not informed that the subject set includes their own family.

This is the first public Chinese multimodal cross-review matrix anywhere. It is not authoritative — eight judges, 96 ratings each — but it is reproducible, and reproducibility is the only way bias measurement exists.