AirLLM 70B inference with single 4GB GPU

AirLLM is a technology that enables running large language models with dramatically reduced memory requirements—allowing 70B parameter models to run on a single 4GB GPU without quantization, distillation, or pruning. The tool supports increasingly larger models, with recent updates enabling 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and the 2.8T parameter Kimi K3 on under 4GB by streaming sparse mixture-of-experts models one expert at a time. The project provides a simple AutoModel interface supporting multiple model families including Llama, DeepSeek, Qwen, and others, with features like FP8 support and CPU inference capability.