Implementation of multi-threaded erasure coding under multi-processing environments

Galois Field arithmetic forms the basis of Reed-Solomon erasure coding techniques to protect storage or communication systems from failures. Most recent implementations of Galois Field arithmetic rely on 128-bit vector instructions, such as Intel's Streaming SIMD Extensions, which allows us to perform fast Galois Field computations. However, these implementations are not optimized for multi-threaded and multi-processing environments. Most of the real servers address multiple requests concurrently and it is desirable to use all the parallelism that hardware provides to efficiently handle the coding workload. This short paper gives some of the details about how to leverage multi-threading capabilities of the modern hardware in multi-processing environments, and demonstrates the significant performance improvements over the single-threaded applications on commodity microprocessors using Jerasure 2.0 library as a case study.