Knowledge Distillation of Black-Box Large Language Models

Researchers introduce Proxy-KD, a novel method that uses a proxy model to transfer knowledge from proprietary black-box large language models (like GPT-4) to smaller models, overcoming the limitation of inaccessible internal states. The approach outperforms traditional knowledge distillation techniques by enabling more effective knowledge transfer despite the teacher model's opacity. This method offers a practical solution for improving smaller models using powerful closed-source LLMs.