HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

HackerRank open-sourced an ATS tool that scores resumes out of 100, but the author found it produces wildly inconsistent results—the same resume scored anywhere from 66 to 99 across multiple runs, meaning candidates would fail 65% of the time at an 85-point cutoff through pure luck. The inconsistency stems from a fundamental design flaw: LLMs cannot reliably make subjective judgments (like evaluating project complexity), and categories with detailed rubrics are still noisy while categories with minimal guidance are useless regardless of consistency. The tool demonstrates that using LLMs for subjective resume evaluation is inherently unreliable, even with low temperature settings designed to reduce randomness.