Design, Implementation, and Evaluation of Trellis-SDP for File-Level Data Parallelism

Although data parallelism is a well-known computational model, there are few programming systems that are both easy to program (for simple applications) and able to work across administrative domains. For data sets (e.g., collections of image data) that are often inherently distributed, there is a need for a simple data-parallel programming system. We describe the design, implementation, and an evaluation of Trellis-SDP, a simple data-parallel programming system that facilitates the rapid development of data- intensive applications. Trellis-SDP is layered on top of the Trellis infrastructure, a software system for creating overlay metacomputers: user-level aggregations of computer systems. Trellis-SDP is based on file-level data parallelism and provides a Master-Worker programming framework in which the worker components can run self-contained, new or existing binary applications. We evaluate our programming system with a non-trivial seismic data processing application.