Comparing high resolution timestamps in computer clusters

Gaining a deep understanding of the detailed execution behavior and performance of complex software systems requires accurate tracing and analysis tools. Tracing tools such as the Linux trace toolkit (LTT) provide a detailed log of both system and user level events which may later be accessed and visualized. The information logged for each event includes the event type, a timestamp, and event type specific information. Most computer central processing units (CPU) now come with a cycle counter, thus enabling fast access, very high resolution (cycle level), timestamp values for events. Problems arise when tracing events from computer clusters containing numerous CPUs with independent clocks. Even CPUs in the same multiprocessor computer may not be fully synchronized, and in some cases run at different clock frequencies. Visualization tools, when presenting a timeline of events logged on different CPUs must therefore translate the individual timestamps to a common time reference, thus taking into account the clock delay for each CPU. Several experiments were conducted in order to characterize the CPU cycle counter accuracy and measure the synchronization accuracy achievable using conventional computer networks. The resulting data is of immediate use for all applications concerned with computer clocks or network synchronization accuracy. It is used in the forthcoming version of the Linux trace toolkit to select the most appropriate mechanisms to compute a common time reference for events coming from different CPUs.