Strategies for Fast I/O Throughput in Large-Scale Climate Modeling Applications

Sen, K and Vadhiyar, S and Vinayachandran, PN (2023) Strategies for Fast I/O Throughput in Large-Scale Climate Modeling Applications. In: 30th Annual IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2023, Proceedings - 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics, HiPC 2023, Goa, pp. 203-212.

PDF pro_2023.pdf - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

Large-scale HPC applications are highly data-intensive with significant times spent in I/O operations. Many large-scale scientific applications do not adequately optimize the I/O operations, leading to overall poor performance. In this work, we have developed two main strategies for providing fast I/O throughput for an important climate modeling application, namely, Regional Ocean Modeling System (ROMS) that uses NetCDF for I/O operations. The strategies include load balancing the I/O operations and selective writing of data. We have also implemented file striping to improve I/O performance. Our experiments with up to 1440 processor cores and 5 days of simulations showed that our load balancing strategy resulted in about 27 decrease in execution times over the default executions, our selective writing strategy resulted in a further decrease of about 30 and the optimized file striping resulted in a further decrease of about 12 in execution times. All the strategies combined together improved the overall performance of the application by about 70. © 2023 IEEE.

Item Type:	Conference Paper
Publication:	Proceedings - 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics, HiPC 2023
Publisher:	Institute of Electrical and Electronics Engineers Inc.
Additional Information:	The copyright for this article belongs to Institute of Electrical and Electronics Engineers Inc.
Keywords:	Collective I/O; Data intensive; Large-scales; Luster striping; Model application; Netcdf; Parallel I/O; Performance; Regional ocean modeling system; Regional ocean modeling system climate model, Climate models
Department/Centre:	Division of Interdisciplinary Sciences > Computational and Data Sciences Division of Mechanical Sciences > Centre for Atmospheric & Oceanic Sciences
Date Deposited:	02 Sep 2024 10:11
Last Modified:	02 Sep 2024 10:11
URI:	http://eprints.iisc.ac.in/id/eprint/84953

Actions (login required)

View Item