Model-driven scheduling for distributed stream processing systems

Shukla, Anshu and Simmhan, Yogesh (2018) Model-driven scheduling for distributed stream processing systems. In: JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING, 117 . pp. 98-114.

PDF Jou_Par_Dis_Com_117_98_2018.pdf - Published Version Restricted to Registered users only Download (2MB) | Request a copy

Abstract

Distributed Stream Processing Systems (DSPS) are ``Fast Data'' platforms that allow streaming applications to be composed and executed with low latency on commodity clusters and Clouds. Such applications are composed as a Directed Acyclic Graph (DAG) of tasks, with data parallel execution using concurrent task threads on distributed resource slots. Scheduling such DAGs for DSPS has two parts-allocation of threads and resources for a DAG, and mapping threads to resources. Existing schedulers often address just one of these, make the assumption that performance linearly scales, or use ad hoc empirical tuning at runtime. Instead, we propose model-driven techniques for both mapping and allocation that rely on low-overhead a priori performance modeling of tasks. Our scheduling algorithms are able to offer predictable and low resource needs that is suitable for elastic pay-as-you-go Cloud resources, support a high input rate through high VM utilization, and can be combined with other mapping approaches as well. These are validated for micro and application benchmarks, and compared with contemporary schedulers, for the Apache Storm DSPS. (C) 2018 Elsevier Inc. All rights reserved.

Item Type:	Journal Article
Publication:	JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING
Publisher:	ACADEMIC PRESS INC ELSEVIER SCIENCE, 525 B ST, STE 1900, SAN DIEGO, CA 92101-4495 USA
Additional Information:	Copy right of this article belong toACADEMIC PRESS INC ELSEVIER SCIENCE, 525 B ST, STE 1900, SAN DIEGO, CA 92101-4495 USA
Department/Centre:	Division of Interdisciplinary Sciences > Supercomputer Education & Research Centre
Date Deposited:	12 Jun 2018 16:01
Last Modified:	12 Jun 2018 16:01
URI:	http://eprints.iisc.ac.in/id/eprint/59981

Actions (login required)

View Item