MotionMontage

Towards Controllable Cinematic
Human–Camera Motion Generation

Anonymous Authors

A continuous run, two cartwheels and a bow, filmed across three shots with tracking and a close-up orbit.
Continuous motion, precise framing, and camera cuts — in one generation process.

Abstract

Cinematic 3D performances require tightly coupling human motion with camera movement, part-level close-ups, and mid-action shot cuts. However, existing text-driven methods mostly generate human motion or camera trajectories separately and are restricted to single-shot, full-body framing. To bridge this gap, we construct MotionMontage, a large-scale dataset of 269K human–camera sequences spanning 540 hours, featuring continuous camera movements, part-level close-ups, and multi-shot transitions. Combining procedural synthesis with 3D movie reconstructions, it pairs each sequence with geometry-grounded descriptions of both the action and cinematographic intent. Building on this dataset, we propose MontageDiT, a flow-matching diffusion transformer that jointly denoises 3D human motion and multi-shot camera trajectories on a shared time grid. By combining subject-centric coordinates, shot-masked attention, and image-space framing supervision, MontageDiT handles within-shot dynamics, part-level close-ups, and hard cuts in a single generation process. Experiments on joint and motion-conditioned generation demonstrate marked gains in trajectory realism, framing accuracy, and temporal smoothness.

01 / Dataset

MotionMontage Dataset

MotionMontage data construction combines movie-based human reconstruction and synthetic camera planning, followed by automatic filtering and human checks, to produce captioned human–camera sequences.
Synthetic and reconstructed data are used to generate multi-shot human–camera sequences.

Dataset comparison

Comparison with existing datasets
DatasetCameraHumanShot attributesScale
TrajectoryCaptionMotionCaptionMulti-shotClose-upHoursFramesSamples
RealEstate10K✓✕✕✕✕✕12111.0M79.0K
DynPose100K✓✕✕✕✕✓1576.8M100.0K
CCD✓✓✕✕✕✓504.5M25.0K
DataDoP✓✓✕✕✕✓11311.0M29.0K
KIT-ML✕✕✓✓✕✕120.8M4.0K
HumanML3D✕✕✓✓✕✕292.0M14.0K
Motion-X++✓✕✓✓✕✓18119.5M120.0K
E.T.✓✓✓✕✕✓12011.0M115.0K
PulpMotion✓✓✓✓✕✓31422.0M193.0K
Auteur✓✓✓✓✕✓—2.7M34.0K
MovieShots✕✕✕✕✕✓51.4—46,857
ShotVerse-Bench✓✓✕—✓✓31.312.0M20.5K
HoloCine✕✓✕✓✓———400.0K
MotionMontage✓✓✓✓✓✓539.842.0M269.4K

✓ Documented presence   ✕ Not provided   — Unverified or unreported. Swipe to view all columns.

02 / Method

MontageDiT

MontageDiT architecture: a frozen Motion DiT conditions the Camera DiT at each shared denoising step, with shot-masked attention and a temporal low-pass head.
Joint human–camera generation on a shared time grid.

03 / Results

Main comparison

Quantitative results on the MotionMontage test set
MethodMotionCameraFraming
FID ↓R@1 ↑R@2 ↑R@3 ↑MM-Dist ↓Div. →FDCLaTr ↓CLaTr-S ↑Cov. ↑R@3 ↑FDFrm ↓Out ↓
Single-shot test set
GT0.00151.8671.8181.452.9619.607090.601.00095.4400.000
DIRECTOR——————14.8571.670.70893.600.080.587
MontageDiT*——————2.5589.590.93994.350.010.001
PulpMotion2.72428.2944.2655.044.5518.34253.0977.990.53094.1070.034.768
MontageDiT0.22850.6670.3980.273.0679.43216.0780.950.77596.262.320.045
Multi-shot test set
GT0.00150.3969.9679.993.0179.534081.941.00071.2600.022
MontageDiT*——————3.8281.330.90673.350.130.101
MontageDiT0.32149.4868.6878.423.1439.1677.0677.590.84962.622.930.145
Full test set
GT0.00151.5771.5381.312.9739.613087.101.00087.2900.006
MontageDiT*——————2.2886.260.92487.380.060.026
MontageDiT0.24350.5370.1480.003.0859.3738.9179.600.80082.922.330.069

* Camera generation from ground-truth motion, compared with DIRECTOR; joint generation is compared with PulpMotion. Bold: best per pair. Camera R@3: CLaTr retrieval; FDFrm: FDFraming (×102); Out: out-of-frame rate (%); → closer to GT is better. Swipe to view all columns.

Single-shot comparison

MontageDiT Ours
PulpMotion
Human Caption

Camera Caption

Multi-shot generation

Global viewCamera view
Human Caption

Camera Caption

Downstream video generation

3D motion & camera
Generated video