iSecureHome: A deep fusion framework for surveillance of smart homes using real-time emotion recognition

Kaushik, H and Kumar, T and Bhalla, K (2022) iSecureHome: A deep fusion framework for surveillance of smart homes using real-time emotion recognition. In: Applied Soft Computing, 122 .

PDF App_Sof_Com_122_2022.pdf - Published Version Restricted to Registered users only Download (7MB) | Request a copy

Abstract

With the advent of AI, the internet of things (IoT) and human-centric computing (HCC), the world has witnessed a rapid proliferation of smart homes (SH). However, implementing a robust security system for residents of SH remains a daunting task. The existing smart homes incorporate security provisions such as biometric verification, activity tracking, and facial recognition. Integrating multi-sensor devices, networking systems and data storage facilities escalate the lifecycle costs of these systems. Facial emotions convey important cues on behaviour and intent that can be used as non-invasive feedback for contextual threat analysis. The early mitigation of a hostile situation, such as a fight or an attempted intrusion, is vital for the SH residents’ safety. This research proposes a real-time facial emotion-based security framework called iSecureHome for smart homes using a CMOS camera, which is triggered by a passive infrared (PIR) motion sensor. The impact of chromatic and achromatic features on facial Emotion Recognition (ER), as well as skin colour-based biases in current ER algorithms, are also investigated. A time-bound facial emotion decoding strategy is presented in iSecureHome that is based on EmoFusioNet—a deep fusion-based model—to predict the security concerns in the vicinity of a given residence. EmoFusioNet utilises stacked and late fusion methodologies to ensure a colour-neutral and equitable ER system. Initially, the stacked model synchronously extracts the chromatic and achromatic facial features using deep CNNs, and their predictions are then fed into the late fusion component. After that, a regularised multi-layer perceptron (R-MLP) is trained to fuse the results of stacked CNNs and generate final predictions. Experimental results suggest that the proposed fusion methodology augments the ER model and achieves the final train and test accuracy of 98.48% and 98.43%, respectively. iSecureHome also comprises a multi-threaded decision-making framework for threat analysis with efficient performance and minimal latency. © 2022 Elsevier B.V.

Item Type:	Journal Article
Publication:	Applied Soft Computing
Publisher:	Elsevier Ltd
Additional Information:	The copyright for this article belongs to Elsevier Ltd
Keywords:	Automation; Convolutional neural networks; Decision making; Digital storage; Face recognition; Forecasting; Intelligent buildings; Internet of things; Life cycle; Network security; Speech recognition, Algorithmic bias; Algorithmics; Emotion recognition; Facial emotions; Fusion methodology; Late fusion; Real-time emotion recognition; Security and surveillances; Smart homes; Threats analysis, Deep neural networks
Department/Centre:	Division of Mechanical Sciences > Centre for Product Design & Manufacturing
Date Deposited:	19 May 2022 06:18
Last Modified:	19 May 2022 06:19
URI:	https://eprints.iisc.ac.in/id/eprint/71985

Actions (login required)

View Item