Record Skipping in Parallel Data Processing Systems

Cloud computing with massive parallel executionengines is a common method to handle big data. But cloudcomputing and the characteristics of the data (semi structured, from different sources, probably cluttered) cause the developmentand execution of data processing jobs to be failure prone. Thisasks for processing engines to be fault tolerant. This paper introduces an approach to skip faulty or alreadyprocessed records after a failure. This enables the executionengine to recover from a failure if the data is flawed. Additionally skipping records allows a recovery with less restarts. Theintroduced record skipping solution relies on the possibility toreproduce a previous data distribution pattern of a job run. Thisis done by logging the order of consumed data. The evaluationshows that the overhead of the described technique is negligibly small.