Benchmarking Pocket-Scale Inference

Artificial Analysis benchmarks small AI models (under 8GB) on mobile phones like the iPhone 17 Pro and Galaxy S26 Ultra, measuring both inference performance and model intelligence across real-world usage scenarios. The benchmark evaluates models on metrics including end-to-end generation time, output speed, peak memory usage, and performance across various intelligence tests at different context lengths. The testing aims to provide comprehensive data on how small language models perform on pocket-scale devices, partnering with Liquid AI to gather actual device measurements.