Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models

Ranjit, M and Ganapathy, G and Manuel, R and Ganu, T (2023) Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models. In: UNSPECIFIED, pp. 650-666.

PDF Pro_mac_lea_res_219_2023.pdf - Published Version Restricted to Registered users only Download (568kB) | Request a copy

Abstract

We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing, using multimodally-aligned embeddings from a contrastively-pretrained vision language model to retrieve relevant radiology text for a given image, and then using a general domain generative model, such as OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4, to generate a report based on the retrieved text. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire, leveraging the instruction following capabilities of generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 (�+ 25.88 ) and Semb score of 0.4026 (�+ 6.31 ). Our approach can be useful for different clinical settings, as it can augment the automated radiology report generation process with relevant content. It also allows to inject the user intents and requirements in the prompts, which can modulate the content and format of the generated reports according to the clinical setting. © 2023 M. Ranjit, G. Ganapathy, R. Manuel & T. Ganu.

Item Type:	Conference Paper
Publication:	Proceedings of Machine Learning Research
Publisher:	ML Research Press
Additional Information:	The copyright for this article belongs to ML Research Press.
Keywords:	Clinical settings; Davinci; Embeddings; Generation process; Generative model; Language model; Radiology reports; Report generation; Report writing, Radiology
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	04 Mar 2024 06:27
Last Modified:	04 Mar 2024 06:27
URI:	https://eprints.iisc.ac.in/id/eprint/84219

Actions (login required)

View Item