Mishra, S and Sundaram, S (2021) Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation. In: Proceedings of the International Joint Conference on Neural Networks, 18 - 22 July 2021, Shenzhen.
|
PDF
IJCNN_2021.pdf - Published Version Download (1MB) | Preview |
Abstract
Existing methods for distillation do not efficiently utilize the training data. This work presents a novel approach to perform distillation using only a subset of the training data, making it more data-efficient. For this purpose, the training of the teacher model is modified to include self-regulation wherein a sample in the training set is used for updating model parameters in the backward pass either if it is misclassified or the model is not confident enough in its prediction. This modification restricts the participation of samples, unlike the conventional training method. The number of times a sample participates in the self-regulated training process is a measure of its significance towards the model's knowledge. The significance values are used to weigh the losses incurred on the corresponding samples in the distillation process. This method is named significance-based distillation. Two other methods are proposed for comparison where the student model learns by distillation and incorporating self-regulation as the teacher model, either utilizing the significance information computed during the teacher's training or not. These methods are named hybrid and regulated distillations, respectively. Experiments on benchmark datasets show that the proposed methods achieve similar performance as other state-of-the-art methods for knowledge distillation while utilizing a significantly less number of samples. © 2021 IEEE.
| Item Type: | Conference Paper |
|---|---|
| Publication: | Proceedings of the International Joint Conference on Neural Networks |
| Publisher: | Institute of Electrical and Electronics Engineers Inc. |
| Additional Information: | The copyright for this article belongs to the Authors. |
| Keywords: | Benchmarking; Distillation; Personnel training, Distillation process; Model knowledge; Modeling parameters; Self regulation; Self-regulated learning mechanisms; Teacher models; Training data; Training methods; Training process; Training sets, Deregulation |
| Department/Centre: | Division of Mechanical Sciences > Aerospace Engineering(Formerly Aeronautical Engineering) |
| Date Deposited: | 15 May 2023 10:11 |
| Last Modified: | 15 May 2023 10:11 |
| URI: | https://eprints.iisc.ac.in/id/eprint/81657 |
Actions (login required)
 |
View Item |
