DiffusionGemma Technical Report

DiffusionGemma is an experimental language model that uses discrete diffusion to generate text significantly faster than traditional autoregressive models by refining blocks of 256 tokens in parallel rather than decoding one token at a time. Created by fine-tuning the Gemma 4 mixture-of-experts model, it achieves approximately 1,500 output tokens per second on a single H100 GPU while maintaining capabilities like thinking mode and multimodal input support. The model was trained efficiently using a two-stage pipeline consuming less than 10% of the original model's training tokens, combining supervised fine-tuning with reinforcement learning and sampler distillation.