Jamesob's guide to running SOTA LLMs locally

This guide provides practical recommendations for running state-of-the-art large language models locally, with hardware configurations ranging from $2,000 (using 2x RTX 3090s for models like Qwen) to $40,000 (using 4x RTX 6000 Pro GPUs with 384GB VRAM for near-Opus-level performance). The author shares his custom build using last-generation EPYC processors and DDR4 RAM paired with PCIe4 switches to enable direct GPU-to-GPU communication, optimizing costs while maintaining strong performance for local LLM deployment.