Performance Evaluation of Modern GPU Accelerator-Based Edge Systems: A Holistic Approach

Amid the growing demand for low-latency edge computing, this study presents a comprehensive performance evaluation of modern GPU (Graphics Processing Unit)-accelerated edge systems, using NVIDIA’s leading Jetson series as representative platforms. We assess these devices across a broad spectrum of workloads—including scientific computing, memory-bound tasks, conventional AI inference, and modern generative AI via a Large Language Model (LLM) case study. Results reveal significant performance scaling across the Jetson family. The high-end Jetson Orin NX delivers an average speedup of 42.7× higher frame rate in traditional AI and a 5.3× speedup in High-Performance Computing (HPC) workloads over the entry-level Nano. Our LLM case study, which establishes a clear hardware threshold for generative AI, shows the Orin NX provides approximately 15% faster token generation. Conversely, the Orin Nano emerges as the leading device for overall efficiency, demonstrating superior cost-performance and power-performance ratios, and proving more energy-frugal in the LLM task for the same high-quality output. Our findings provide practical, evidence-based guidelines for selecting optimal edge devices by balancing absolute performance against critical cost and energy constraints.