CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

CODA is a GPU kernel abstraction that optimizes Transformer training by reformulating memory-bound operations (normalization, activations, residual updates) as GEMM-plus-epilogue programs, keeping intermediate tensor outputs on-chip rather than moving them through global memory. The approach uses composable epilogue primitives while preserving the performance structure of hand-optimized GEMMs, achieving high performance across representative Transformer workloads and covering nearly all non-attention computation in standard Transformer blocks. This method addresses a significant bottleneck in Transformer training systems by reducing data movement overhead in otherwise highly optimized stacks.