Large scale graph processing in a distributed environment

Upadhyay, N and Patel, P and Cheramangalath, U and Srikant, YN (2018) Large scale graph processing in a distributed environment. In: International Workshops on Parallel Processing, Euro-Par 2017, 28 -29 August 2017, Santiago de Compostela, pp. 465-477.

PDF Springer_EPPPW_465-477_2018.pdf - Published Version Restricted to Registered users only Download (821kB) | Request a copy

Abstract

Large graphs are widely used in real world graph analytics. Memory available in a single machine is usually inadequate to process these graphs. A good solution is to use a distributed environment. Typical programming styles used in existing distributed environment frameworks are different from imperative programming and difficult for programmers to adapt. Moreover, some graph algorithms having a high degree of parallelism ideally run on an accelerator cluster. Error prone and lower level programming methods (memory and thread management) available for such systems repel programmers from using such architectures. Existing frameworks do not deal with the accelerator clusters. We propose a framework which addresses the previously stated deficiencies. Our framework automatically generates implementations of graph algorithms for distributed environments from the intuitive shared memory based code written in a high-level Domain Specific Language (DSL), Falcon. The framework analyses the intermediate representation, applies a set of optimizations and then generates Giraph code for a CPU cluster and MPI+OpenCL code for a GPU cluster. Experimental evaluations show efficiency and scalability of our framework.

Item Type:	Conference Paper
Publication:	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publisher:	Springer Verlag
Additional Information:	The copyright for this article belongs to the Springer International Publishing AG, part of Springer Nature.
Keywords:	Codes (symbols); Computer programming languages; Digital subscriber lines; DSL; High level languages; Optimization; Particle accelerators; Problem oriented languages, Degree of parallelism; Distributed architecture; Distributed environments; Experimental evaluation; Falcon; Graph processing; Imperative programming; Intermediate representations, Memory architecture
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	22 Aug 2022 09:16
Last Modified:	22 Aug 2022 09:16
URI:	https://eprints.iisc.ac.in/id/eprint/76120

Actions (login required)

View Item