Monitoring data-structure evolution in distributed message-passing programs

Monitoring the evolution of data structures in parallel and distributed programs, is critical for debugging its semantics and performance. However, the current state-of-art in tracking and presenting data-structure information on parallel and distributed environments is cumbersome and does not scale. We present a methodology and tool that automatically tracks memory bindings (not the actual contents) of dynamic data-structures of message-passing C programs, and inter-processor data-structure movement, using PVM on distributed environments. With the help of a number of examples we show that in addition to determining the impact of memory allocation overheads on program performance, graphical views can help in debugging many memory access errors. Traditional debuggers in distributed environments rely on existing sequential debuggers on each machine and simply provide an interface for querying and controlling each processor's debugging session. However, to quickly locate the processor and to explain reasons for the error, we resort to run-time checking and trace based visualizations of memory access behavior across all processors. In an effort to reduce trace file size, only updates of pointer values and memory management functions are captured.