Why are visual and physical AI systems still struggling to hit

A survey of 700+ industry professionals reveals that the real culprit isn't the model architecture—it's the data bottleneck. While we've seen LLMs scale effortlessly on web-scraped text, physical AI (robotics, autonomous systems, industrial vision) is hitting a wall because high-quality, diverse, and labeled spatial data is incredibly hard to source at scale.

The gap between simulation and reality

The biggest friction point right now is the "sim-to-real" gap. Most teams rely on synthetic data to train their agents, but these models often crumble when they hit a real-world environment with unpredictable lighting, occlusions, or physical textures. The data we generate in simulators is too "clean," leading to models that are overfitted to a perfect world.

To get around this, a lot of teams are trying a more hands-on guide approach to data collection, using "shadow mode" where the AI observes a human operator for thousands of hours before ever taking control. But the labeling cost for this kind of multimodal data is astronomical compared to simple text classification.

Where the workflow is actually breaking

If you look at the production pipeline, the majority of the time is wasted on data curation rather than actual training. The professionals surveyed pointed out a few specific failure points:

• Edge Case Discovery: Models work 95% of the time, but that last 5% of "long-tail" scenarios causes catastrophic failure in physical environments.
• Labeling Noise: Human annotators often disagree on spatial boundaries in 3D point clouds or video frames, introducing noise that degrades model precision.
• Data Silos: In large industrial setups, the data needed to fix a bug is often locked in a proprietary sensor log that the AI team can't easily access or parse.

Moving toward an LLM agent for data

The trend is shifting toward using a more sophisticated AI workflow where LLM agents are used to autonomously curate and filter datasets. Instead of humans manually scrubbing through thousands of hours of video, these agents can be prompted to find specific failure modes or "interesting" anomalies that need more training samples.

For anyone trying to build a real-world deployment from scratch, the lesson is clear: stop obsessing over the model parameters and start obsessing over the data pipeline. The winner in physical AI won't be the one with the biggest model, but the one with the cleanest, most representative dataset of the physical world.

If you want the full breakdown, there's a whitepaper available that goes into the specific percentages of where these teams are spending their budgets.