NVIDIA Analysis releases SpatialClaw, a no-training framework for spatial reasoning. It targets persistent weaknesses within the Imaginative and prescient Language Mannequin (VLM). These fashions nonetheless wrestle to find out the place objects are, how they’re associated to one another, and the way they transfer in 3D.
SpatialClaw doesn’t retrain the mannequin. as an alternative, motion interface Brokers are used to invoke recognition instruments. The analysis crew claims that the interface is the bottleneck. Their resolution is to deal with the code as an motion interface. Throughout 20 benchmarks, SpatialClaw reaches a median accuracy of 59.9%. That is 11.2 factors forward of current spatial agent SpaceTools.
What’s SpatialClaw?
SpatialClaw is an agent loop wrapped round a stateful Python kernel. The kernel is preloaded with an enter body and a set of primitives. The popularity instrument is a plain Python callable instrument. Outputs akin to masks, depth maps, digicam geometry, and trajectories are common Python variables.
The kernel exposes six public entry factors. InputImages Holds the sampled body. Metadata Preserves body price, period, and body index. instruments Expose perceptual and geometric primitives. present() Embed the picture within the following context of the agent: vlm Dispatch the question to a different VLM session. ReturnAnswer() Submit your closing reply.
Two recognition instruments are central. instruments.Reconstruct Wraps Depth Something 3 and returns per-frame depth, digicam built-in, exterior, and dense level maps. instruments.SAM3 Wrap SAM 3 to generate picture or video masks from textual content, level, or field prompts. This framework provides light-weight utilities. instruments.Geometry, instruments.Masks, instruments.Time, instruments.Graphand instruments.Draw.
No coaching required. The identical system prompts, instrument set, and hyperparameters are run for all benchmarks and backbones.
Why motion interfaces are vital
The analysis crew studied three motion interfaces for a similar query. Take into account measuring the closest distance between the heater and the door.
- single passcode Write one full program and run it as soon as. Decide to the total technique earlier than taking a look at intermediate masks or depth maps. False assumptions propagate on to the reply.
- Structured instrument calls Name named instruments by way of a hard and fast JSON schema. The output can’t be freely mixed with NumPy or SciPy to characterize test-time calculations. The result’s improper as a result of there aren’t any pre-registered instruments for the closest level calculation.
- area nails Configure the instrument in code, examine and repair the outcomes. First we calculate the centroid distance after which discover that the centroid makes use of the median worth. agent switches to
scipy.spatial.KDTreeTo seek out the true nearest neighbor. Submit 0.9439 m in opposition to a floor fact of 0.9 m.
benchmark
SpatialClaw was examined on 20 benchmarks throughout 5 classes. These vary from single picture, multi-view, basic, video and 4D, and basic video understanding. All six backbones examined improved over the no-tool baseline. The spine ranges in parameters from 26B to 397B throughout the Qwen3.5/3.6 and Gemma4 households.
Managed comparisons isolate interfaces. All three variants share the identical toolset and prompts. Solely the motion interface is completely different.
| motion interface | Common (bench 20) | Δ vs. no instrument |
|---|---|---|
| Device-free baseline | 53.4 | – |
| single passcode | 55.2 | +1.8 |
| Structured instrument calls | 56.7 | +3.3 |
| SpatialClaw (code as motion) | 59.9 | +6.5 |
Gemma4-31B spine, 20 benchmark common.
For earlier spatial brokers on the identical Gemma4-31B spine, the hole widens.
| methodology | interface | common | Δ vs Spatial Claw |
|---|---|---|---|
| Vadal | single go | 40.5* | −19.4 |
| py area | single go | 47.8 | −12.1 |
| SpaceTools – Device Shed | Structured instrument calls | 48.7 | −11.2 |
| area nails | code as motion | 59.9 | one of the best |
The most important impression is on dynamic duties. For Gemma4-31B, DSI-Bench elevated by +17.6 factors and MindCube elevated by +15.3 factors. These classes require cascading geometric calculations throughout frames and viewpoints.
The attribution of LLM as a choose explains the victory over structured instrument calls. Code composition accounts for 52.2% of that. Management move accounts for 19.5%, and the remaining 28.3% is interface impartial.
Contained in the 5 stage loop
Every pattern runs a 5 step loop: Plan, generate code, run code, assemble suggestions, and submit solutions. Planners create methods with out taking a look at movies. The primary agent then writes one Python cell per step. Static AST checkers reject unsafe code earlier than it’s executed. The loop repeats till: ReturnAnswer() known as or 30 steps elapse.
The official repository runs on LangGraph workflows and protracted Jupyter kernels. The spine works by vLLM. Notion runs behind the FastAPI GPU service. One quickstart runs one benchmark on one machine.
git clone --recursive https://github.com/NVlabs/SpatialClaw.git
cd SpatialClaw
bash spatial_agent/scripts/setup.sh
cp .env.instance .env # add API keys, or self-host vLLM
python -m spatial_agent.entrypoints.run
--dataset spatial_agent/config/dataset/erqa.json
--model spatial_agent/config/mannequin/gemini-3-pro.json
--concurrency 4A consultant agent cell organizes notion in geometry and modifies it as follows.
# Reconstruct the scene, then phase each objects in a single video go
recon = instruments.Reconstruct.Reconstruct(InputImages)
seg = instruments.SAM3.segment_video_by_text(["radiator heater", "door"])
present(seg.visualize(1)) # examine the masks first
# Closest-point distance by way of KD-tree, not centroids
pts_h = seg.get_masked_points(recon, body=1, object=0) # object 0 = heater
pts_d = seg.get_masked_points(recon, body=2, object=1) # object 1 = door
dists, _ = scipy.spatial.KDTree(pts_d).question(pts_h, ok=1)
ReturnAnswer(float(dists.min()))The agent selects primitives from the query itself. The gap query invokes KD-tree search and vector norms. The route query is determined by the dot product. No category-specific routing was utilized.
Utilization instance
This design is appropriate for issues that require step-by-step geometric reasoning. Particular examples embody:
- Robotics and embodied brokers Measure the space between objects earlier than performing.
- Multi-view inspectionthe dealing with route of the thing is recovered from a number of digicam angles.
- Video and 4D evaluation Monitor object or digicam motion throughout frames.
- Q&A on indoor scenes“The place is the door relative to the sink?”
No coaching is required, permitting groups to scale a deployed VLM with out new information or fine-tuning.
interactive explainer