Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
|
TrnRT is an open-source inference engine for large language models optimized for AWS Trainium hardware, offering an OpenAI-compatible API and serving GLM-5.3-Flash from a single trn2.48xlarge instance. It features NKI megakernels for efficient decoding, support for 1M-token context with prefix caching, and offline compilation to reduce serving overhead. The project is now available on GitHub with documentation and benchmarks for developers interested in deploying LLMs on Trainium infrastructure.