Prediction-capable data compression algorithms for improving transmission efficiency on distributed systems

Network bandwidth is a limited and precious resource in distributed computing environments. Insufficient bandwidth will severely degrade the performance of a distributed computing task in exchanging massive amounts of data among the networked hosts. A feasible solution to save bandwidth is to incorporate data compression during transmission. However blind, or unconditional, compression may only result in waste of CPU power and even slow down the overall network transfer rate, if the data to be transmitted are hard to compress. We present a prediction-capable lossless data compression algorithm to address this problem. By adapting to the compression speed of a host CPU, current system load, and network speed, our algorithm can accurately estimate the compression time of each data block given, and decide whether it should be compressed or not. Experimental results indicate that our prediction mechanism is both efficient and effective, achieving 93% of prediction accuracy at the cost of only 3.2% of the execution time of unconditional compression.