Neural Voxel Dynamics: Learning Volumetric Feature
Advection for 3D Physics in V-JEPA Latent Space

Abstract

Contributions

Three claims, in the order the paper makes them:

Method

Pixel observations entangle distinct factors: camera perspective, scene geometry, and material-dependent dynamics. Against 2D generative models on one side and explicit 3D simulators on the other, Neural Voxel Dynamics is a latent middle ground. The design is materially implicit — one operator handles heterogeneous phenomena without any material label; geometrically grounded — the voxel structure makes occupancy and occlusion explicit rather than something to infer from entangled 2D latents; and perspective-independent — anchored in a world frame, so it accepts multiview input and a predicted trajectory can be rendered from new viewpoints without re-running it.

Sparse tokenization and the advector

To avoid the cubic cost of dense grids, f_θ operates on a sparse set of active voxels and their neighbours: the union of occupied and occluded voxels across the context window, dilated morphologically to leave room for predicted motion. Capacity concentrates on scene-relevant geometry, and cost scales with object complexity rather than with grid resolution.

The advector is a Diffusion Transformer variant whose layers alternate between two attention modes. Local spatial attention restricts each voxel to a k × k × k sparse neighbourhood (5 layers, k = 5), mimicking the step-wise force propagation of classical solvers such as MPM and giving cost linear rather than quadratic in the number of active tokens. Temporal attention regroups tokens by spatial location so each voxel attends causally along its own trajectory. Since one step propagates information only within a local neighbourhood, longer-range transport is composed over stacked layers and successive prediction steps. Adaptive layer normalization modulates the network by the flow-matching time τ, and the output feature velocities are scattered back into the lattice.

Force conditioning and viewpoint

The control is a time-varying action descriptor u_t ∈ ℝ¹²: contact point, applied direction, magnitude, temporal duration, an active-frame indicator, and local / global flags, expressed and normalized in world coordinates to reduce ambiguity. It is injected on two complementary paths — local integration, where the force is projected to the token dimension and added element-wise to each voxel token as a spatially uniform contextual field, and global modulation, where it is encoded into conditioning tokens concatenated to the sequence so self-attention can modulate the scene-level transition.

Crucially, u_t is not a privileged simulator state: it specifies no material parameters such as stiffness or viscosity, no engine state, and no per-particle variables. The advector never receives them, and material-dependent behaviour emerges purely from the latent state and the learned transition. Actions are drawn only for the CLEVRER/MuJoCo data; on datasets without such annotations the action reduces to the zero vector and the model predicts implicit dynamics.

Observations come from a single fixed viewpoint, but because the state is anchored in the world frame rather than the camera, the formulation accepts multiview input where it is available. Moving cameras remain out of scope: sparse tokenization scales poorly to them, as it does to unbounded environments and high-frequency deformation.

Experiments

Setup & evaluation protocol

We evaluate across three datasets spanning heterogeneous physics: a CLEVRER-style simulated set with explicit forces (rigid collisions), PhysInOne (fluids with implicit force), and PhysGaia (smoke and fluids). Ours-GT uses ground-truth depth (available for CLEVRER and PhysInOne); Ours-Estimate uses MoGe-estimated depth for all three. Kubric MOVI-C is held out entirely and used only for zero-shot testing.

All methods receive the first frames and generate the following ones. Because our model predicts V-JEPA latents directly, every baseline's generated RGB video is encoded through the same V-JEPA encoder before scoring, so all methods are measured in a common space over 16 frames. Force is provided in each baseline's native interface — 3D force for PhysCtrl/PhysGaussian, force projected to 2D camera space for PhysGen, and as text for CogVideoX.

To avoid favoring models optimized under the same latent objective, we report three non-V-JEPA views of physical correctness alongside latent L2: occupancy (3D geometry), 2D/3D latent flow (motion), and optical flow / PSNR on decoded RGB. A 2D-latent baseline (no 3D projection, same flow-matching objective) isolates the contribution of lifting to 3D.

1 · Latent prediction loss

We first measure prediction accuracy directly in V-JEPA latent space. Each entry reports 2D / 3D latent L2 loss (↓), under single/multi-camera and ground-truth/estimated-depth protocols, and broken down by dynamics category. Our method stays low and stable across material types and camera configurations.

2 · RGB reconstruction (decoder)

A small V-JEPA-to-video decoding head transfers the predicted latent dynamics back to RGB. We report frame-level reconstruction (PSNR / MSE) against reference videos and show decoded clips.

3 · Flow estimation

We compare predicted latents against ground-truth trajectories via latent flow consistency (lower is better). Kubric MOVI-C is zero-shot — never used for training. Ours-Estimate is trained with diffusion forcing and history-guided diffusion throughout.

4 · Flow → real videos

The estimated 2D flow drives a large video model (i2v + 2D trajectories, Wan-Move) on real footage, and we score the physical plausibility of the generated videos with Physics-IQ.

5 · Occupancy loss

A non-V-JEPA, geometry-level test: predicted 3D occupancy vs. ground-truth occupied voxels. Occupancy IoU is the primary measure (higher is better); the mass ratio is a diagnostic. Ours-Estimate is reported at occupancy thresholds 0.4 / 0.5 / 0.6.

Occupancy IoU is the primary geometric metric; the mass ratio is a diagnostic, not a standalone ranking. A value near one is meaningful mainly for approximately closed, mass-conserving objects — it need not stay constant for phenomena such as smoke, and a method can hit ∼1 while placing the occupied volume incorrectly, so it should be read jointly with IoU.

6 · Force as input

The action is a 12-D per-frame descriptor (contact point, direction, magnitude, duration, and active / local / global flags). Here we compare driving it explicitly, as a supplied vector, against leaving it at zero so the dynamics must be inferred from past frames alone.

(a) Different explicit forces

Same initial scene, driven by different explicit force vectors. Each row shows the input frames on the left and a prediction vs. GT comparison on the right.

(b) Implicit forces

Force inferred from past-frame motion without an explicit force input. As above, the input frames are on the left and a prediction vs. GT comparison on the right.

Ablations

We study the key design choices: voxel resolution, voxel channels, the depth estimator, the projection loss, input views, the occupancy/occlusion losses, and the training objective.

Against an equal-capacity deterministic regression head, generative flow matching is decisively better (Table 15). For longer rollouts we switch the binary occupancy output to a continuous value compatible with flow matching / diffusion forcing, so binary focal / BCE losses no longer apply; vanilla history-guided diffusion is the best configuration and is used for Ours-Estimate.

The method is not tied to any single depth predictor — VGGT works too (slightly worse). Because we predict occupancy rather than hallucinate unseen regions with a generative prior, and a JEPA encoder can be jointly trained to absorb depth cues, geometry can in principle be learned more directly from video; that extension is outside this work's scope.

Failure examples

Typical failure modes. The main one is noise accumulation over long autoregressive rollouts. Scenes far from the training distribution also strain both the decoder and the predictor together, and geometric fidelity is bounded by the V-JEPA encoder resolution and monocular-depth accuracy, so very high-frequency deformation is hard to capture at the current voxel resolution.

Conclusion