Semi-supervised Verified Feedback Generation

Kaleeswaran, Shalini and Santhiar, Anirudh and Kanade, Aditya and Gulwani, Sumit (2016) Semi-supervised Verified Feedback Generation. In: 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), NOV 13-18, 2016, Seattle, WA, pp. 739-750.

PDF Pro_2016_24th_ACM_Sig_Int_Sym_Fou_Sof_Eng_739_2016.pdf - Published Version Restricted to Registered users only Download (416kB) | Request a copy

Abstract

Students have enthusiastically taken to online programming lessons and contests. Unfortunately, they tend to struggle due to lack of personalized feedback. There is an urgent need of program analysis and repair techniques capable of handling both the scale and variations in student submissions, while ensuring quality of feedback. Towards this goal, we present a novel methodology called semi-supervised verified feedback generation. We cluster submissions by solution strategy and ask the instructor to identify or add a correct submission in each cluster. We then verify every submission in a cluster against the instructor-validated submission in the same cluster. If faults are detected in the submission then feedback suggesting fixes to them is generated. Clustering reduces the burden on the instructor and also the variations that have to be handled during feedback generation. The verified feedback generation ensures that only correct feedback is generated. We implemented a tool, named CoderAssist, based on this approach and evaluated it on dynamic programming assignments. We have designed a novel counter-example guided feedback generation algorithm capable of suggesting fixes to all faults in a submission. In an evaluation on 2226 submissions to 4 problems, CoderAssist could generate verified feedback for 1911 (85%) submissions in 1.6s each on an average. It does a good job of reducing the burden on the instructor. Only one submission had to be manually validated or added for every 16 submissions.

Item Type:	Conference Proceedings
Additional Information:	Copy right for this article belongs to the ASSOC COMPUTING MACHINERY, 1515 BROADWAY, NEW YORK, NY 10036-9998 USA
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	10 Feb 2017 04:16
Last Modified:	10 Feb 2017 04:16
URI:	http://eprints.iisc.ac.in/id/eprint/56201

Actions (login required)

View Item