StarData: Distributed Map-Reduce Framework Atop Serverless Computing Platforms

With the rapid development of information networks, intelligent terminals, the Internet of Things, and various sensing devices continue to expand. A large amount of isomorphic and heterogeneous data has been generated. How to process these data efficiently and in real time has become the leading research direction of extensive data analysis and processing. However, the traditional big data processing platform requires help to meet the basic requirements for efficiently processing large amounts of data quickly. The Map-Reduce extensive data processing framework proposed by Google can process various types of homogeneous and heterogeneous data. Users can analyze and process data in the computing cluster after configuring it as needed. We propose StarData. It is an extensive data processing framework based on a service-oriented computing platform. By building the Map-Reduce processing framework on AWS Lambda, a serverless computing platform, we can realize the requirement of parallel processing of large-scale data in a short time. StarData is a lightweight extensive data analysis and processing framework with good cost-effectiveness. The efficiency of comprehensive data analysis and processing is optimized by reasonably adjusting the number of functions, files, file size, and resource allocation.