Evaluating Physical Reasoning in LLM Agents Requires Construction Benchmarks
1School of Engineering, Westlake University · 2School of Computing Science, University of Glasgow
Abstract
Physical reasoning is central to agents that step into the physical world. Models now excel at code and math, but their physical reasoning capabilities remain largely untested. Existing physical benchmarks evaluate goal-state matching rather than functional performance. This position paper argues that evaluating physical reasoning in LLM agents requires construction benchmarks, where physics verifies functional outcomes. To operationalize physical reasoning evaluation, we identify five capabilities spanning the full agentic loop from goal to verified artifact. Our audit of eleven physical benchmarks finds only two that combine realistic physics with functional verification, the minimum requirements for testing physical reasoning; both are construction benchmarks. Neither covers all five capabilities, yet frontier models already achieve near-zero success as task complexity increases. Progress depends on the breadth of benchmarks we build. We call on academia, game studios, and industry labs to expand construction benchmarks along both axes, widening capability coverage and deepening physical challenges. Their breadth determines how fully agents move from code and math to reshaping the physical world.
The standard
An evaluation of physical reasoning must meet two minimum requirements.
- Functional verification. Physics judges the outcome. The artifact works or it does not.
- Realistic physics. Gravity, friction, and collision constrain every action along the way.
Together they define one standard for evaluation, functional verification under irreversible intervention, which spans the full causal chain from intent to verifiable artifact. We decompose that chain into five capabilities: goal-driven design, physics-aware planning, embodied execution, functional evaluation, and closed-loop refinement.
The living audit
The paper audits eleven benchmarks against the five capabilities. The audit continues after the paper is frozen. New benchmarks are assessed against the same criteria and added to the awesome-agentic-physical-reasoning repository, which also explains how to submit one. The table below is rendered from that repository's data/benchmarks.json.
✓ tested · ✗ not tested · ◐ partial · ? unclear from the published text. Both = realistic physics and functional verification.