Fault-Tolerant and Elastic Streaming MapReduce with Decentralized Coordination

Kumbhare, Alok and Frincu, Marc and Simmhan, Yogesh and Prasanna, Viktor K (2015) Fault-Tolerant and Elastic Streaming MapReduce with Decentralized Coordination. In: 2015 IEEE 35th International Conference on Distributed Computing Systems, JUN 29-JUL 02, 2015, 2015 IEEE 35th International Conference on Distributed Computing SystemsOhio State Univ, Columbus, OH, pp. 328-338.

PDF 2015_IEEE_Int_Con_Dis_Com_Sys_328_2015.pdf - Published Version Restricted to Registered users only Download (488kB) | Request a copy

Abstract

The MapReduce programming model, due to its simplicity and scalability, has become an essential tool for processing large data volumes in distributed environments. Recent Stream Processing Systems (SPS) extend this model to provide low-latency analysis of high-velocity continuous data streams. However, integrating MapReduce with streaming poses challenges: first, the runtime variations in data characteristics such as data-rates and key-distribution cause resource overload, that in-turn leads to fluctuations in the Quality of the Service (QoS); and second, the stateful reducers, whose state depends on the complete tuple history, necessitates efficient fault-recovery mechanisms to maintain the desired QoS in the presence of resource failures. We propose an integrated streaming MapReduce architecture leveraging the concept of consistent hashing to support runtime elasticity along with locality-aware data and state replication to provide efficient load-balancing with low-overhead fault-tolerance and parallel fault-recovery from multiple simultaneous failures. Our evaluation on a private cloud shows up to 2.8x improvement in peak throughput compared to Apache Storm SPS, and a low recovery latency of 700 - 1500 ms from multiple failures.

Item Type:	Conference Proceedings
Series.:	IEEE International Conference on Distributed Computing Systems
Additional Information:	Copy right of this article belongs to the IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA
Department/Centre:	Division of Interdisciplinary Sciences > Supercomputer Education & Research Centre
Date Deposited:	08 Oct 2016 06:33
Last Modified:	08 Oct 2016 06:33
URI:	http://eprints.iisc.ac.in/id/eprint/54756

Actions (login required)

View Item