HEVAL: A New Hybrid Evaluation Metric for Automatic Speech Recognition Tasks

Sasindran, Z and Yelchuri, H and Prabhakar, TV and Rao, S (2023) HEVAL: A New Hybrid Evaluation Metric for Automatic Speech Recognition Tasks. In: UNSPECIFIED.

PDF Aut_spe_rec_und_wor_asr_2023.pdf - Published Version Restricted to Registered users only Download (267kB) | Request a copy

Abstract

Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems. Since WER considers only literal word-level correctness, new evaluation metrics based on semantic similarity such as semantic distance (SD) and BERTScore have been developed. However, we found that these metrics have their own limitations, such as a tendency to overly prioritise keywords. We propose HEVAL, a new hybrid evaluation metric for ASR systems that considers both semantic correctness and error rate and performs significantly well in scenarios where WER and SD perform poorly. Due to lighter computation compared to BERTScore, it offers 49 times reduction in metric computation time. Furthermore, we show that HEVAL correlates strongly with downstream NLP tasks. Also, to reduce the metric calculation time, we built multiple fast and lightweight models using distillation techniques without significant reduction in performance. © 2023 IEEE.

Item Type:	Conference Paper
Publication:	2023 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023
Publisher:	Institute of Electrical and Electronics Engineers Inc.
Additional Information:	The copyright for this article belongs to Institute of Electrical and Electronics Engineers Inc.
Keywords:	Distillation; Petroleum reservoir evaluation; Speech recognition, Automatic speech recognition; Automatic speech recognition system; Correctness rates; Error rate; Evaluation metrics; Literals; Semantic distance; Semantic similarity; Word error rate; Word level, Semantics
Department/Centre:	Division of Electrical Sciences > Electronic Systems Engineering (Formerly Centre for Electronic Design & Technology)
Date Deposited:	16 May 2024 11:05
Last Modified:	16 May 2024 11:05
URI:	https://eprints.iisc.ac.in/id/eprint/84556

Actions (login required)

View Item