An Open Field Guide to Inference Engineering

This open field guide introduces inference engineering—the practice of running large language models in-house—as an emerging specialization for platform engineers. Organizations are increasingly moving away from third-party AI APIs due to cost savings (up to 80%), data privacy concerns, and improved open-source models that now compete with commercial offerings like OpenAI's. The guide explains that inference engineering builds on traditional SRE skills like scheduling, capacity planning, and load balancing, but applies them to GPU workloads and model serving rather than traditional infrastructure.