Speculative Sampling

1. Why small chunks?

Why pursue the smallest possible bidirectional chunks? Consider a simple interactive video generation system. The generator delivers video in chunks, either periodically or non-periodically. Each chunk contains T seconds of video. Assume that generation is always faster than playback and meets every delivery deadline. In this stable system, the player receives a new chunk at regular intervals of T. Let K denote the number of temporal latents in a chunk and δ the video duration represented by each latent:

Interactive control signals V keep arriving, and we assume that their arrival times are uniformly distributed within a period T. A control signal first waits for R seconds; the generator then takes C to generate a video chunk; after a further wait Q, the player starts playing the chunk at the boundary of the next playback period.

We encounter a frustrating fact: no matter how we shift the start of generation or choose the scheduling strategy, the expected total waiting time R + Q has a lower bound of half the chunk duration T:

This means that even if sampling were instantaneous and network delay negligible, TTFF could still be very large, depending on the chunk period.

The chunk period depends on both the number of frames in each bidirectional chunk and the system’s nominal frame rate. This corresponds to two routes for reducing TTFF: reducing the number of frames per chunk (chunk size), or increasing the temporal frame rate used for DiT sampling (sample_fps). The latter is disruptive to existing video data assets and is not common practice. This leaves reducing chunk size—the preference for small bidirectional chunks discussed above.

2. Why not the smallest?

But why stop short of using the smallest possible bidirectional chunk size? To investigate, we conduct a controlled ablation: for different chunk sizes, we run the full streaming post-training and evaluation pipeline starting from teacher forcing. See Section 4.2 of the paper for details.

We observe that as chunk size increases from its minimum of one latent, video quality first rises sharply, then quickly begins to plateau, while still improving as the bidirectional modeling chunk grows. This also explains why the more balanced choice of chunk size 3 has become a mainstream setting in the community.

CHUNK-SIZE ANALYSIS / WAN1.3B

Another interesting observation is that increasing chunk size does not significantly increase the time needed to denoise a video chunk. For example, with Wan1.3B using Ulysses SP on a single H200 node, increasing chunk size from 1 to 7 raises the time only from 77.5 ms to 78.6 ms. Taking this further, with LTX-2.3 under the same setup, increasing chunk size from 1 to 45 (about 15 seconds) raises the time only from 274 ms to 507 ms. This suggests that, at the token lengths used in streaming sampling, the compute kernels are far from fully utilized.

3. Speculative sampling

These two observations motivate three questions. For a frame-wise model with chunk size 1, we ask:

1. Could we append additional noise latents as padding after the current video chunk at each sampling call, obtaining a larger bidirectional modeling window and improving its modeling capability?
2. What padding window length hits the sweet spot between modeling difficulty and sampling time?
3. Do we really have to discard the predictions from these padding latents afterward?

Our answer is Speculative Sampling: it jointly generates a large chunk under the current condition, assuming it will remain unchanged, and commits the predictions incrementally. Cached predictions are reused while the condition holds; otherwise, the replaceable suffix is discarded and regenerated.

Our method preserves the bidirectional modeling window of large chunks while supporting the fine-grained interaction of small chunks.

4. Managing a revocable future

To manage prediction reuse, replacement, and refill, we develop a minimal implementation of Speculative Sampling that can be described by a two-state machine.

In the Consume state, the system reuses predictions from the cache and commits them gradually. In the Produce state, the model runs a generation job either to refill the cache or to replace an uncommitted suffix.

When a new condition arrives, playback continues from the committed prefix. The controller keeps the part that can no longer be changed and regenerates the remaining future from the earliest feasible point. If the condition does not change, it simply refills the cache before the available predictions run out.

Our method predicts ahead using the latest observed condition, assuming that it will remain unchanged. If the condition stays the same, the cached predictions are reused—a cache hit. If the condition changes, the affected predictions become a cache miss and must be generated again, introducing some redundant computation.

Our TTFF analysis shows that Speculative Sampling removes the T/2 expected waiting-time lower bound, replacing it with terms tied to computation. This shifts TTFF optimization from reducing chunk duration to accelerating per-chunk sampling. See the paper for the full analysis.