The Tracer Files: Cracking the Case of Performance Impact in Tracing Linux File I/O for I/O-Intensive Applications

Tracing file I/O operations in Linux is critical for understanding and optimizing application performance, particularly in I/O-intensive environments. However, the choice of tracing tool can dramatically impact the application throughput. This paper investigates the performance overhead associated with two widely used tracing tools - strace and bpftrace - when applied to file I/O operations. Through empirical evaluation using a controlled benchmark over 100 epochs, we compare the throughput distributions of training a machine learning model under both tools against a baseline without tracing. The results reveal that strace introduces significant performance degradation, reducing throughput by 17%. In contrast, bpftrace, leveraging the efficiency of BPF, reduces the throughput by only 1% compared to the baseline, while maintaining consistent performance across trials.