LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios

2609.09790 ↗ · 2026-09-09 · cs.CV, cs.AI, cs.CL

Hanjing Zhou, Mingze Yin, Ying Lian, Jun Ma, Chang-Yu Hsieh, Yanbing Zhou

The finding

Headline claim

State-of-the-art vision-language models, including proprietary frontier systems, fall well short of human logistics-expert performance on a new real-world warehouse hazard-identification benchmark, with fine-grained perception in cluttered surveillance imagery the main bottleneck and pervasive risk-averse response bias present across models.

The test

A benchmark of 2,476 images and 2,918 videos sourced from one year of Cainiao warehouse surveillance, distilled into 5,394 visual samples and 10,274 human-validated VQA pairs spanning 18 objects, 20 risk types, and 39 subtasks across perception, knowledge, and reasoning; 20 open and proprietary LMMs evaluated with three-repeat runs, LLM-as-judge for open-ended items, plus 10 novice humans and 1 logistics expert.

Effect size & scope

Best overall scores around 0.70 to 0.71 (GPT-5.5, Qwen3.5-Plus, Claude-Opus-4.7) versus 0.95 for one logistics expert and 0.73 for novice students; fine-grained perception is weakest, with open-source averages of 0.39 (FG-S) and 0.36 (FG-C) below even novice humans; thinking mode helps risk reasoning but not perception; only 25% of models pass the fairness audit. Scope: single industrial domain, mostly one company's footage, one expert as the human ceiling.

Limitations

Authors concede reliance on template-generated questions, partial synthetic rebalancing of rare risks via image editing (roughly 900 samples with Nano Banana 2), LLM-as-judge for open-ended scoring, and a single expert defining the human upper bound. Unstated: potential train-on-benchmark contamination risk for Qwen-family models given Cainiao affiliation questions, narrow geographic and operational source of footage, and that the expert's near-ceiling score on template questions may partly reflect familiarity with question format rather than vision alone.

Counterpoint

A vendor-authored benchmark built partly with automated question templates and AI-edited images could overstate deployment gaps or bake in annotation idiosyncrasies; LLM-as-judge scoring is itself model-sensitive; results could reflect prompt formatting and answer-matching artifacts rather than genuine perception deficits; n=1 expert makes 'human expert level' a thin reference point.

Economy implication

Little or nothing yet for AI economics at large. It signals that vision-based industrial safety automation is not deployable unattended today, which matters mainly for logistics-operations AI vendors pricing safety products and for warehouse operators deciding whether human monitors remain necessary over the next 12 to 24 months.

Who this matters for

Technical Capability: Shows frontier multimodal models still fail at fine-grained perception in dense, low-quality real-world imagery, a concrete ceiling on agentic vision deployment outside clean web-style images.

Labor & Knowledge Work: Warehouse safety monitoring remains a task where humans outperform models by a wide margin, implying no near-term AI substitution for safety-inspection roles.

Regulatory & Legal: The documented risk-averse bias and fairness-audit failures (only 25% pass) complicate certifying LMMs for safety-compliance monitoring.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs' practical deployment in real-world settings. The industrial dataset is publicly available under the CC BY-NC-SA 4.0 license.