Moonshot AI has introduced PerceptionBench, a benchmark designed to test whether multimodal AI models can correctly interpret images before they are asked to reason about them. The evaluation reported by The Decoder found that none of the 16 frontier models tested reached 60% overall accuracy, with the leading system scoring 59.7%.

The result matters because many failures attributed to poor reasoning may begin earlier: the model may have misread the image, counted incorrectly, missed a visual relationship, or imagined an object that was not present. PerceptionBench attempts to isolate that first stage from general knowledge and multi-step reasoning, giving AI builders a more precise way to diagnose multimodal failures.

A benchmark built around basic visual skills

Moonshot AI’s research team, which also develops the Kimi assistant, said PerceptionBench evaluates visual perception independently of logical reasoning and outside knowledge. Each question is intended to be answerable from the image itself.

The benchmark divides perception into ten skill areas: visual relations, counting, attributes, depth and 3D understanding, localization, comparison, fine-grained recognition, context integration, optical character recognition, and hallucination. The categories were reportedly derived from observed model errors rather than imposed as a purely theoretical taxonomy.

The team analyzed 42 existing open-source benchmarks and found limited overlap in the errors they measured. Moonshot AI therefore created an internal pool of more than 17,000 verified questions and is releasing 3,000 tasks. According to The Decoder’s account, 60% of the released questions came from documented model errors, while the remainder were reformulated with augmented images.

The public dataset and evaluation code are available through the MoonshotAI/PerceptionBench project on GitHub. That release should allow researchers and model developers to test whether the reported weaknesses reproduce outside Moonshot AI’s own evaluation process.

Frontier models remain below the benchmark’s threshold

The Decoder reported that GPT-5.6 Sol achieved the highest overall result at 59.7%, followed by Kimi K3 at 58.5%, Claude Fable 5 at 57.2%, Gemini 3.1 Pro at 56.2%, and GPT-5.5 at 55.8%. Open-source models listed in the report included Qwen3.5-397B-A17B at 47.5% and GLM-4.6V at 32.5%.

These figures should be treated as benchmark results reported through The Decoder, not as a universal ranking of multimodal capability. Performance can vary with prompting, image composition, task selection, and evaluation design. The models and scores are also presented in the source material without an independent replication in the evidence available for this article.

The category results appear more revealing than the leaderboard. Models with similar aggregate scores reportedly diverged substantially by skill. Hallucination was the weakest area on average: GPT-5.6 Sol scored 26.9% on the sub-test, while Gemini 3.5 Flash reportedly reached 50.6% despite ranking lower overall. The task tests whether a model can correctly answer “zero” when an image contains none of the object being queried, rather than inventing one.

That pattern suggests that a single multimodal score can conceal serious weaknesses in specific workflows. A model may be dependable at reading text in an image but unreliable at counting, spatial comparison, or determining whether an object exists at all.

Why perception errors look like reasoning failures

Many production tasks combine visual interpretation with a chain of decisions. An assistant might inspect a dashboard, identify a warning, compare two product images, and then recommend an action. If the first visual observation is wrong, later reasoning can remain internally consistent while still producing the wrong answer.

PerceptionBench’s approach breaks complex questions into perception-only sub-questions. Moonshot AI’s researchers argue that this makes it possible to identify the earliest failed step rather than labeling the entire result as a reasoning error.

The finding is consistent with earlier work cited by The Decoder, although those studies used different tests. The WorldVQA benchmark reportedly found that its best-performing model scored 47.4% on tasks separating object recognition from reasoning. Another study using BabyVision reported that Gemini 3 Pro reached 49.7% on basic visual tasks, compared with 94.1% for humans. The source attributes that gap to a verbalization bottleneck in which visual information loses fidelity when converted into language.

Those comparisons do not establish that all multimodal systems fail in the same way, but they reinforce the narrower point behind PerceptionBench: strong language generation and broad reasoning performance do not guarantee reliable visual input processing.

What the results mean for AI product teams

For builders, the immediate implication is architectural rather than cosmetic. A multimodal model should not automatically be trusted to perform counting, inventory checks, document extraction, spatial navigation, or visual quality control simply because it can describe an image fluently.

Teams deploying visual systems may need task-specific evaluations for the exact failure modes that affect their workflow. A retail system should test object presence and fine-grained attributes. A document product should measure OCR and layout interpretation. A robotics or industrial application should separately assess localization, depth, and spatial relationships before adding planning or tool use.

The benchmark also supports using verification steps around high-impact visual outputs. A second model, a conventional computer-vision tool, structured extraction, or human review may be appropriate when an incorrect count or missed warning carries financial, safety, or compliance consequences. These safeguards add latency and cost, but PerceptionBench’s scores indicate that fluent answers are not sufficient evidence of accurate perception.

For model vendors, the release creates pressure to report category-level results rather than only aggregate multimodal benchmarks. Developers comparing GPT-5.6 Sol, Kimi K3, Claude Fable 5, or Gemini 3.1 Pro may reach different conclusions depending on whether their priority is OCR, hallucination resistance, counting, or spatial understanding.

What to watch next

The first signal to watch is independent reproduction of PerceptionBench results across model versions and inference settings. Because the benchmark and code are publicly available, outside evaluations can test whether the reported gaps persist beyond Moonshot AI’s setup.

Researchers should also examine contamination and task construction, particularly the share of questions derived from known model errors and the effect of augmented images. Those details will help determine whether the benchmark measures general visual ability or strongly targets failure patterns already observed by its authors.

For buyers, the useful follow-up is not simply which model tops the overall list. It is whether vendors publish reliable, category-specific results and whether improvements transfer to real images, changing layouts, noisy camera input, and the edge cases found in production.

Creati.ai perspective

PerceptionBench is most valuable as a diagnostic benchmark, not as a definitive verdict on multimodal AI. Its central contribution is to separate seeing from reasoning, a distinction that many product evaluations blur. A model that explains an image convincingly can still be wrong about what the image contains.

The practical lesson for AI teams is straightforward: test the visual primitive that a workflow depends on, measure failure costs, and place verification where perception errors can propagate. Until multimodal systems become more dependable at these supposedly simple tasks, model selection should be based on the specific visual job—not on a general capability score alone.