A disappearing Service Processor (2025)

Oxide engineers debugged a critical issue where their Service Processor would disappear from the network when integrated into their next-generation Cosmo sled within racks, despite the host system remaining powered on. Through systematic investigation involving LED debugging modifications and physical SWD debug header connections, they identified that task starvation in their custom Hubris operating system was causing the networking task to be unable to respond, with stack overflows in the kernel being a primary suspect. The team improved their debugging capabilities by modifying task restart delays and switching to observable LED blinking patterns to diagnose issues even without network access.