Why Is Speculative Decoding Fast?

Speculative decoding is fast not because it reduces computational work, but because it changes how GPUs allocate that work. By using a small draft model to propose multiple tokens at once and having a large model verify them in parallel, speculative decoding shifts token generation from being memory-bound (waiting for data to load) to compute-bound (fully utilizing GPU processing power), thereby improving throughput on individual sequences despite actually performing more total operations.