Perturbing the weights 2,000 times: the error bars on the 2026 native multimodal tier list

A tier list that publishes only the ranking is asking you to trust its weights. The 2026 China native multimodal LLM tier list treats its weights as a hypothesis to be tested: we perturb all eight dimension weights by ±30%, renormalise, rerun 2,000 times, and record the rank distribution of every model.

This essay publishes the full results, including the parts that are unflattering to us.

First, a rule: a ranking without error bars is faking precision

A physicist publishing a measurement must publish its error bar, because “9.81” without one could be 9.81±0.01 or 9.81±3. Tier lists are in exactly the same position: a 95.4-point number one that drops to fifth under a different reasonable weighting carries almost no information.

The problem is that the ranking industry has no habit of publishing error bars. Rankings publish positions because positions travel; error bars do not. Distribution value and information value conflict here, and this site chooses information value.

So what this essay really teaches is a method you can run on any tier list. It does not depend on our data — only on one thing: the raw dimension scores. That is also why all 256 score points are public.

Weights are scenario assumptions

Start with this: there is no “correct” value for the weights. Perception 20, documents 18, charts 14, video 14, agents 14, generation 6, Chinese 6, ecosystem 8 — this set encodes the scenario assumption of “a typical Chinese enterprise multimodal deployment”, where documents and charts together take 32 points because most paid calls today happen in document processing.

If your workload is short-video understanding, video should be 30 rather than 14; if it is creative work, generation should be 30 rather than 6. Change the weights and the ranking changes occupants.

So the question for judging a tier list is not “are its weights right” but “how sensitive is its ranking to its weights”. That is what the perturbation experiment answers.

How the experiment works (reproducible)

• Multiply each of the eight weights by a uniform random number in [0.7, 1.3]
• Renormalise so the weights still sum to 1
• Recompute composites for all 32 models and re-rank
• Repeat 2,000 times and record each model’s rank distribution

The ±30% band is not arbitrary: it roughly matches how far a serious buyer actually moves weights. At ±60% every rank jumps around, so you measure noise rather than robustness.

The full recomputation script ships with the site data (scripts/analyze.mjs); its input is the 256 dimension scores of the 32 models and its output is every table below.

Result one: the top spot only swaps between two models

Across 2,000 runs, only two models ever hold #1:

Model	Share of first place	Rank range
Qwen3.8-Max	91.1%	1–2
Doubao Seed 2.0 Pro	8.9%	1–2
The other 24	0%	3 or lower

That number deserves unpacking. Qwen3.8-Max scores 95.4 and Doubao 95.2 — a gap of 0.2, and under our method anything inside 0.5 is one band, so these two were tied all along. The perturbation experiment simply translated the tie into probability: 91.1% of reasonable weight vectors put Qwen first, 8.9% put Doubao first.

Which 8.9%? Mostly the vectors that raise video and Chinese context. Doubao’s video 96 and Chinese 97 are the highest in the field, so when those two weights float up it overtakes. Put the other way: if your workload is Chinese short video, Doubao is your number one — not a compromise, just the weights you should have set.

Result two: only 1 of 32 never moves

This is the most stable conclusion across three expansions. At 24 models, three models never moved across 2,000 runs; at 26, one remained; at 32, still only Qwen3.5-397B never moves a single place. It has been through three sample expansions, each amplifying volatility across the middle and tail, and it has not moved once.

Model	Baseline rank	Perturbed range	Top-5 membership
Qwen3.8-Max	1	1–2	100%
Doubao Seed 2.0 Pro	2	1–2	100%
Qwen3.5-397B	3	3–3 (never moves)	100%
GLM-5.3	4	4–5	100%
Kimi K3	5	4–6	93.4%
GLM-5	6	5–6	6.6%
MiMo-V2.5	7	7–9	0%
SenseNova 6.5 Pro	8	7–8	0%
LongCat-Next	9	8–10	0%
Qwen3-Omni-30B	10	9–11	0%
ERNIE 5.1	11	11–13	0%
DeepSeek-V4.1-Flash	12	10–13	0%
Qwen3.8-Flash	13	11–13	0%
Keye-VL-2.0	14	14–15	0%
Hunyuan-Vision	15	15–16	0%
InternVL 4.0	16	14–19	0%
Step-3	17	15–18	0%
Spark X2	18	16–18	0%
MiniMax M2.5	19	18–20	0%
DeepSeek-OCR 2	20	19–24	0%
Huawei Pangu 5.5	21	20–22	0%
Ant Ling 2	22	20–27	0%
WITA-Omni Preview	23	20–26	0%
Emu3.5	24	21–26	0%
Zidong Taichu 2.0	25	23–26	0%
MiniCPM-V 4.5	26	23–27	0%
Hikvision Guanlan 3	27	25–31	0%
JD Yanxi 2	28	27–31	0%
NetEase Fuxi 2	29	27–32	0%
NeuroXess-N1	30	28–31	0%
BAGEL-2	31	28–32	0%
Cloudwalk Congrong 2	32	31–32	0%

Four readings:

• The top three is the most stable part of the board. Qwen3.5-397B never moves a place in 2,000 runs — all eight of its dimension scores sit above 90 with an even spread, nothing extreme enough to be amplified by a weight and nothing weak enough to be punished. That “featurelessness” is the strongest form of robustness.
• Expanding the sample amplified the volatility. Keye-VL-2.0 went from never moving at 24 models to drifting across 14–15; the newly listed InternVL 4.0 spans 14–19 and Ant Ling 2 spans 20–27. The reason is direct: these profiles are sharply asymmetric (perception 96 against video 66; Chinese 92 against video 60), so any reweighting amplifies one end. The lower and more specialist the model, the less information the overall rank carries.
• The last six shuffle among themselves. Ranks 27 through 32 (Guanlan, Yanxi, Fuxi, NeuroXess-N1, BAGEL-2, Congrong) all have perturbed ranges fully overlapping inside 25–32 — the order among them is essentially weight noise. No rank in that zone belongs in a selection deck.
• “Never moves” is itself a scarce good. Three expansions, 6,000 runs, and exactly one model has not moved once. A board that publishes this kind of number has honest error bars; a board that publishes only positions cannot distinguish robustness from luck.

Result three: locating the weight darlings exactly

A “weight darling” is a model that only ranks well under specific weights. To locate one, force a single dimension weight to 30%, compress the rest proportionally, and see who climbs most:

Amplified dimension	Biggest climber	Rank change
Generation 6% → 30%	Emu3.5	24 → 13 (+11)
Generation 6% → 30%	BAGEL-2	31 → 21 (+10)
Ecosystem 8% → 30%	Qwen3.8-Flash	13 → 7 (+6)
Ecosystem 8% → 30%	DeepSeek-V4.1-Flash	12 → 9 (+3)
Video 14% → 30%	Keye-VL-2.0	14 → 11 (+3)
Video 14% → 30%	WITA-Omni Preview	23 → 19 (+4)
Video 14% → 30%	Hikvision Guanlan 3	27 → 23 (+4)
Chart 14% → 30%	DeepSeek-V4.1-Flash	12 → 10 (+2)
Chart 14% → 30%	InternVL 4.0	16 → 14 (+2)
Document 18% → 30%	InternVL 4.0	16 → 14 (+2)
Document 18% → 30%	Zidong Taichu 2.0	25 → 23 (+2)
Agent 14% → 30%	Qwen3.8-Flash	13 → 11 (+2)
Agent 14% → 30%	Step-3	17 → 15 (+2)
Chinese 6% → 30%	Spark X2	18 → 16 (+2)
Chinese 6% → 30%	Zidong Taichu 2.0	25 → 23 (+2)
Perception 20% → 30%	Kimi K3	5 → 4 (+1)

Note the reverse: under all eight extreme weightings, the top three only swap internally — no outsider breaks in. That means this board’s top three is not manufactured by weights, which matters more than any single score.

Emu3.5’s +11 is the classic darling signal: its generation 98 is the highest in the field while its other dimensions are broadly weak. For image generation and editing work it belongs in the candidate set; chosen off the overall board it sits at 24. Both conclusions are correct; only the weights differ.

InternVL 4.0 is the new darling sample: raising document or chart weight to 30% lifts it two places each — a #16 overall model reaches the top 14 in “documents and charts only” work. That is the standard profile of a specialist. Ant Ling 2 shows the same shape on the Chinese dimension: at 30% it climbs from 22nd into the top 20, because its Chinese 92 is the only dimension where no cheaper rival stands.

Why evidence grade is not in the composite

Of the 32 models, fourteen are grade A, sixteen grade B and two grade C. The number one, Qwen3.8-Max, is a B — under two weeks old, with thin third-party reproduction and no 30-day stability record.

The single C is NeuroXess-N1: every figure is vendor-reported, with zero independent reproduction. Listing a grade-C model is our own choice — the reasoning is in why cold models survive, and the short version is that nobody else tests the modality it covers.

This is not a contradiction; it is a fact you should have. Multiplying evidence grade into the composite fails twice: new models get systematically depressed and the list lags reality, and “more evidence” gets misread as “stronger”, which on fast-moving Chinese models is often the opposite.

Our handling is to publish the evidence grade as an independent marker: ranking reflects capability, the marker reflects certainty, and you set the risk appetite:

• Chasing the capability ceiling, tolerant of version churn → read the composite and pick Qwen3.8-Max
• Signing a three-year contract, no tolerance for regression → pick only among grade A: Doubao Seed 2.0 Pro (95.2)
• Budget-sensitive and able to run your own regression → the two Flash models and InternVL 4.0 are the best-value band

How to use this variance table

Three concrete suggestions:

1. Do not treat the overall rank as a precise order. The three models at ranks 11–13 (ERNIE 5.1, DeepSeek-V4.1-Flash, Qwen3.8-Flash) have completely overlapping perturbed ranges; “12th beats 13th” has no basis.
2. Take the candidate set as the union of the perturbed ranges. If you want the top five, the correct candidate set is six models, not five — GLM-5 has a 6.6% chance of entering the top five, and excluding it substitutes one sample for the whole distribution.
3. Specialists skip the overall board entirely. Models with a range span of 3 or more (DeepSeek-OCR 2, WITA-Omni, Emu3.5, Kimi K3, InternVL 4.0) should be chosen by dimension, where the overall rank carries no information. Dimension sorting is one click away on the tier list page.

Full weight definitions, score mapping rules and evidence criteria live on the methodology page. To change the weights yourself, use the model picker — it does the same thing with the perturbation direction handed to you.