Dynamic Strategies for High Performance Training of Knowledge Graph Embeddings

Panda, A and Vadhiyar, S (2022) Dynamic Strategies for High Performance Training of Knowledge Graph Embeddings. In: 51st International Conference on Parallel Processing, ICPP 2022, 29 August - 1 September 2022, Virtual, Online.

PDF icpp_2022.pdf - Published Version Restricted to Registered users only Download (2MB) | Request a copy

Abstract

Knowledge graph embeddings (KGEs) are the low dimensional representations of entities and relations between the entities. They can be used for various downstream tasks such as triple classification, link prediction, knowledge base completion, etc. Training these embeddings for a large dataset takes a huge amount of time. This work proposes strategies to make the training of KGEs faster in a distributed memory parallel environment. The first strategy is to choose between either an all-gather or an all-reduce operation based on the sparsity of the gradient matrix. The second strategy focuses on selecting those gradient vectors which significantly contribute to the reduction in the loss. The third strategy employs gradient quantization to reduce the number of bits to be communicated. The fourth strategy proposes to split the knowledge graph triples based on relations so that inter-node communication for the gradient matrix corresponding to the relation embedding matrix is eliminated. The fifth and last strategy is to select the negative triple which the model finds difficult to classify. All the strategies are combined and this allows us to train the ComplEx Knowledge Graph Embedding (KGE) model on the FB250K dataset in 6 hours with 16 nodes when compared to 11.5 hours taken to train on the same number of nodes without applying any of the above optimizations. This reduction in training time is also accompanied by a significant improvement in Mean Reciprocal Rank (MRR) and Triple Classification Accuracy (TCA). © 2022 ACM.

Item Type:	Conference Paper
Publication:	ACM International Conference Proceeding Series
Publisher:	Association for Computing Machinery
Additional Information:	The copyright of this article belongs to Association for Computing Machinery.
Keywords:	Classification (of information); Graph theory; Knowledge graph; Large dataset; reductions; Communication minimization; Gradient quantization; Gradient vectors; Graph embeddings; Knowledge graph embedding; Knowledge graphs; matrix; Quantisation; Selection of gradient vector.; Graph embeddings
Department/Centre:	Division of Interdisciplinary Sciences > Computational and Data Sciences
Date Deposited:	10 Mar 2023 10:23
Last Modified:	10 Mar 2023 10:23
URI:	https://eprints.iisc.ac.in/id/eprint/80936

Actions (login required)

View Item