Abstract
Video generative models have become increasingly powerful, but long-range consistency remains challenging to achieve because even a few dozen frames require impractically long transformer sequence lengths. We show that this issue can be mitigated by generating video using coarse-to-fine rollout within a multi-scale token space. Our approach is simple: first, we pre-train an autoencoder that compresses each frame into a hierarchy of tokens, with levels ranging from the typical latent resolution to only a handful of tokens per frame. The coarsest levels capture the most consequential information—such as scene layout and semantics—while finer levels add high-frequency appearance and texture. Then, we train a video diffusion model to generate these tokens using coarse-to-fine rollout. By carefully controlling the level of detail at which frames are generated and used as context during each rollout step, we are able to preserve long-range consistency in geometry and object permanence while spending less compute on the long-range consistency of less perceptually relevant details. We validate this approach using a custom dataset of long Minecraft videos, where it produces substantially more consistent rollouts compared to existing baselines.
Results
Each method sees up to 256 frames of the same context, then begins autoregressive rollout conditioned on ground-truth actions (forward/left/right). Our method produces rollouts that are significantly more consistent with the ground truth compared to FramePack and standard autoregressive rollout given the same transformer sequence length budget.
Quality vs. Consistency
Our metrics are designed to independently measure two aspects of video generation: consistency and quality. Consistency is a model's ability to accurately recall previously seen content; quality is the ability to generate high-quality frames during rollout, regardless of whether those frames are consistent or not. We plot these metrics as a function of rollout length, averaged over 1,000 test set examples. LPIPS is measured between generated and corresponding ground-truth frames. Our model significantly outperforms the baselines on both consistency and quality. Note that both baselines eventually become worse than random frames on consistency due to exposure bias, while our method rolls out stably.
Side Note: Why Minecraft?
We use a custom-rendered Minecraft dataset that's similar to the ones used in TECO and Diffusion Forcing as a testbed for long-range consistency because unlike commonly used datasets, it fulfills all of the following requirements:
- Dataset size: large enough to train a generative model
- Video length: arbitrarily long videos (not just dozens or low hundreds of frames)
- Recall: content that frequently exits and later re-enters the frame, forcing models to remember
- Fine-grained conditioning: signals like poses or actions that can steer the model toward previously-seen content
Method
Our method has two key components: a hierarchical autoencoder and a latent diffusion model that uses coarse-to-fine rollout. The diffusion model operates in the autoencoder's multi-level latent space.
Hierarchical Autoencoder: Latent Space
The hierarchical autoencoder creates a multi-resolution latent space where level $\ell$ corresponds to a grid of $\frac{H}{2^\ell} \times \frac{W}{2^\ell}$ tokens. Visualize the hierarchical latent space by interacting with the figure below.
Side Note: How are multi-level hierarchical latents different from resolution cascades?
We compare our hierarchical autoencoder to a "downscaled" equivalent, where the coarser latents are downscaled (mean-pooled) versions of the finest latents. The "downscaled" equivalent simulates what a cascaded diffusion model based on resolution cascades sees; hierarchical latents provide much better reconstructions given small token budgets. Use the drop-down menu to switch the above figure to visualizing the "downscaled" equivalent.
Hierarchical Autoencoder: Training Procedure
To train the hierarchical autoencoder, we first convert the input image into a multi-resolution pyramid by repeatedly downscaling it by 2x along each dimension. Next, we patchify the entire pyramid's images using the same kernel size, which naturally yields fewer tokens for coarser pyramid levels. Then, we feed the resulting tokens through a transformer encoder. The encoder's output is our hierarchical latent space. During training, we randomly select a level to decode. To decode a particular level, we zero out the tokens corresponding to the other levels and feed all tokens through a transformer decoder. The decoder creates an image pyramid; we supervise on MSE at all pyramid levels and additionally supervise on LPIPS at the highest-resolution pyramid level.
Generation via Coarse-to-Fine Rollout
Here's what full-resolution autoregressive rollout looks like. Each frame is represented by one grid cell. Conditioned on two context frames, the model denoises two new frames at once. The previous step's denoised frames become the current step's context frames.
Here's what our method's rollout looks like. Each grid cell still corresponds to one frame, but the rows correspond to different hierarchy levels (i.e., levels of detail). The finest level (top row) requires 256 tokens per frame, while subsequent levels require 4x fewer tokens than the previous level per frame. Our model rolls out by first generating many coarse frames, then refining them level by level. The refinement process not only conditions on past frames, but also on coarse future frames.
A few important notes:
- The number of tokens the model sees (context or denoised) is constant across rollout steps! And in fact, it's the exact same number that the full-resolution autoregressive rollout model uses. In other words, with three hierarchy levels, you get roughly 10x the context compared to naive rollout.
- The animation above shows three hierarchy levels, with a context of up to 21 frames. Increasing the number of levels (e.g., to 4, as in our Minecraft models) yields an additional 64 frames of context. One can also multiply the number of tokens at each level by some factor $k$, denoising $k$ frames at the finest level instead of just one.
What about FramePack?
The animation below shows what FramePack does. For the same sequence length, FramePack has the same context length and per-context-frame token allocation as our model. However, it performs much worse. We hypothesize that this is because unlike our model, it does not predict many future frames at once, meaning that the model has no incentive to look into the distant past at training time (the recent past is generally enough to predict the close future). We tested FramePack with a hierarchical latent space (as shown below) and with its original approach to varying the number of per-frame tokens, which is to patchify context frames with differing patch sizes.
BibTeX
@inproceedings{chandratreya2026millivid,
title = {MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation},
author = {Chandratreya, Ishaan Preetam and Charatan, David and Van Hoorick, Basile and Zakharov, Sergey and Guizilini, Vitor and Isola, Phillip and Sitzmann, Vincent},
booktitle = {arXiv},
year = {2026},
url = {https://davidcharatan.com/millivid},
}Acknowledgements
We thank Andrew Song and Hannah Schlueter for their feedback during the process of writing and editing the paper. This work was supported by the Toyota Research Institute (TRI) University 3.0 (URP) program, the National Science Foundation under Grant No. 2211259, by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, by the MIT-Google Program for Computing Innovation, by AMD via the MIT AI Hardware Program, and by a 2025 MIT Office of Research Computing and Data Seed Grant. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of any other entity.