Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

Researchers found that training a single transformer layer can match or exceed the performance gains of full-parameter reinforcement learning training in large language models. Across multiple models and RL algorithms, they discovered that RL improvements are highly concentrated in middle-layer transformer blocks, while input and output layers contribute substantially less. This suggests that RL adaptation is not uniformly distributed across model layers, opening potential efficiency improvements for LLM post-training.