Three things to remember
- A clean accuracy cliff: vendor 93.8%, open-source status 71.9%, context 53.1%, version 31.3%, price 21.9%, parameter count 15.6% — the closer to “who I am”, the more accurate; the closer to “what my specs are”, the less.
- Version inflation runs at 65.6%: 22 models claim to be the newest in their family or a later version. Context answers skew to the family maximum, and only 21.9% get price right.
- Self-knowledge is unrelated to capability (r = 0.09); the five models that answer parameter count correctly are exactly the five that publish it — those who disclose know, those who do not invent.
If you want to know a model’s context length, price and open-source status, the most direct thing to do is ask the model itself. This essay did exactly that — and found it is the least reliable way to do it.
We asked 32 Chinese multimodal models the same set of six questions about themselves, then compared every answer item by item against this site’s dossiers. Every answer to this question set is a public, checkable specification fact that involves no reasoning:
- Who is your developer?
- What is your context window?
- What is your output price?
- How many parameters do you have?
- Are you open source?
- Which version are you now?
One sentence of conclusion: models know “who they are”, but almost never know “what specs they are”. There is no second evaluation like this anywhere — because it evaluates not capability but whether a model’s self-knowledge can be trusted.
Results table: identity 93.8%, specification 15.6%
| Question | Models correct | Correct rate | Main error form |
|---|---|---|---|
| Vendor | 30 / 32 | 93.8% | Joint-development subject is vague |
| Open-source status | 23 / 32 | 71.9% | Calls “open weights” “fully open source” |
| Context | 17 / 32 | 53.1% | Reports the family maximum, not its own value |
| Version | 10 / 32 | 31.3% | Calls itself “the latest in the family” |
| Price | 7 / 32 | 21.9% | Quotes a price but gets the tier or currency wrong |
| Parameter count | 5 / 32 | 15.6% | Invents a specific number |
Across the 32 models the average is just 2.9 correct (47.9% of six), and none scores 6/6. This is a very clean “identity–specification” fault line: the closer a question comes to “who am I”, the higher the accuracy; the closer to “what are my parameters”, the lower.
The only exception is in the last row: the five that got parameter count right are exactly the five models on this site that disclose a parameter count (Qwen3.5-397B, Kimi K3, Qwen3-Omni-30B, Keye-VL-2.0, MiniCPM-V 4.5). Not a coincidence, a causal link — vendors that write the parameter count into the model card also put that card in the model’s training data; of the 27 that do not disclose a parameter count, 12 gave a specific but wrong number and the remaining 15 refused or were vague. This is the same event as the parameter transparency index, seen from two observation planes.
The identity question: 98% know who they are, two are vague
The vendor question is the most accurate on the whole table. Thirty models report their subject correctly; only two are vague:
- BAGEL-2 reports “Peking University”, while the dossier records “ByteDance · PKU” — it treats the academic partner in a joint development as the sole subject;
- Zidong Taichu 2.0 reports “Chinese Academy of Sciences”, while the dossier records “CAS Institute of Automation” — a downgrade to the institutional level.
Neither is exactly “wrong”, but they reveal a pattern: when a model is built by several parties jointly, the model’s self-description tilts toward one of them rather than the full attribution. In procurement documents this is a risk — you think you are dealing with A, while the actual delivering subject is a consortium of A and B.
The version question: the best one, and the most dangerous one
The version question is only 31.3% correct, for a single and predictable reason: models call themselves “the latest in the family”.
Among the six models on this site that are “one of several products in the same family”, five gave the family’s latest name rather than their own when asked point-blank about their version. The typical shape: ask GLM-5 “which version are you” and it answers “GLM-5.3”; ask Qwen3.5-397B a similar question and it reports a higher version number. Of the 32, 22 call themselves the family’s latest or a higher version number — a version-inflation rate of 65.6%.
This explains a real-world phenomenon: when using an old model, users get the reply “I am the latest version”, and misjudge the capability boundary. In scenarios that need traceability (compliance, audit, tendering), a model’s self-reported version number cannot serve as evidence; you must check back against the dossier and the model identifier returned by the API.
The specification questions: windows report the family maximum, prices are not remembered
The context question is 53.1% correct, and the errors run almost entirely in one direction: reporting the family maximum.
Qwen3-Omni-30B has a 32K window (the smallest on the site), but when asked it tends to report a larger tier from the Tongyi line; ModelBest MiniCPM-V 4.5’s 96K is likewise often reported as a bigger number. A model’s “self-knowledge” is really “family knowledge” — it remembers the family ceiling, not its own entry in it. This is the same coin as the finding in naming archaeology that “a family prefix signals ecosystem, a suffix signals price”: the model learned only the brand layer, not the product layer.
The price question is one of the two worst on the table (21.9%). This is not because models are “dumb”, but because the training data contains no such thing as “my price”: prices move with promotions, caching and volume discounts, and every vendor counts in different units (per million tokens, per thousand calls, per thousand images). A model can recite “I am probably not cheap”, but cannot answer “¥0.6 or ¥1.2”. DeepSeek-V4.1-Flash’s actual price of ¥0.6 is the lowest on the site, yet its self-report usually lands in the ¥1–2 range.
Open-source status: 8 call “open weights” “fully open source”
The open-source question is 71.9% correct; 10 are wrong. The errors cluster in one category: treating “open weights” as “open source”.
Qwen3.8-Flash is Apache-2.0 open weights, but its base model and training data are not open; DeepSeek-V4.1-Flash is MIT weights, likewise not an open-source project in the OSI sense. When asked, models say “I am open source”, equating “the weights can be downloaded” with “the source code is open”.
This error is real for procurement: “open source” decides whether you can audit, whether you can modify, whether you can avoid licence risk, while “open weights” decides only whether you can deploy. The two are different things in contract and compliance; the zero-price models essay already did this arithmetic.
A counter-intuitive correlation: self-knowledge has nothing to do with capability
Correlating each model’s six-question score with its composite score: r = 0.09, essentially zero. That is, stronger models do not know themselves better.
- Qwen3.8-Max, #1 by composite: 3 correct;
- DeepSeek-OCR 2, #20 by composite: 3 correct;
- Cloudwalk Congrong 2, lowest by composite (#32): 4 correct, above most flagships.
The reason is direct: self-knowledge tests whether the training data contains “a document that spells out one’s own specs”, not capability. Cold models’ vendor docs tend to be more concentrated (only one product’s specs to memorise), while flagship families have many product lines and fast version turnover, so the model is in fact more easily confused.
Reading for procurement: a model is not its own documentation
- Specifications must be looked up in the dossier, never asked of the model. Context, price, licence and parameter count: the model’s answer can only be a clue, never a basis.
- The version number must follow the identifier returned by the API. In processes that need traceability, record the model ID and the call time returned by the API, not the model’s self-statement.
- For “am I open source”, read the original licence. Treat the model’s self-reported “open source” as “open weights”, then verify the commercial terms against the enterprise veto gates.
- Be more wary of “multi-family” teams. The more product lines, the more easily a model’s self-report floats up to the family ceiling — when a team ships four models, it is precisely the one whose specs deserve the closest checking.
Boundaries
- Six questions are not a complete self-knowledge evaluation. We tested only publicly checkable specification facts, not metacognition like “knowing what one does not know”.
- Answers are affected by the prompt. Longer priming and more context make some models give more accurate specs; this essay uses one uniform short question, close to a user’s natural asking.
- A wrong answer is not a lie. Most errors are “family maximum” contamination in the training data, not deliberate concealment; this site records consistency only and makes no judgement of motive.
- Sample of 32 models, dated 2026-10-18. Specs and versions change as vendors update; a rerun would bring new models into the count.
Reproduction
Six questions, one uniform short question, each compared item by item against the site dossier (vendor, context, priceOut, params, license, release); a point is scored for “exact match”. The scoring script and question set ship with the site data.
If this essay leaves one sentence: for specs, check the dossier, don’t ask the model — it knows who it is, but not how long its own context is.
FAQ about this essay
Does this mean the models are lying?
No. Most errors come from “family maximum” contamination in training data — models learn the brand layer, not the product layer. We record only agreement between answers and dossiers, and infer no motive.
So how do I get accurate specs?
Read the docs and the API-returned identifier: context, price, licence and parameter count all follow official documentation, and model self-reports are only a lead. Where traceability matters, log the API-returned model ID rather than the model’s self-description.
