Denny, P and Kumar, V and Giacaman, N (2023) Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In: 54th ACM Technical Symposium on Computer Science Education, SIGCSE 2023, 15 - 18 March 2023, Toronto, pp. 1136-1142.
|
PDF
SIGCSE_2023.pdf - Published Version Download (1MB) | Preview |
Abstract
GitHub Copilot is an artificial intelligence tool for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about its potential impact on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60 of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development. © 2023 ACM.
| Item Type: | Conference Paper |
|---|---|
| Publication: | SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education |
| Publisher: | Association for Computing Machinery, Inc |
| Additional Information: | The copyright for this article belongs to the Authors. |
| Keywords: | Artificial intelligence; Education computing; Engineering education, Artificial intelligence tools; Cs1; Foundation models; Github copilot; Introductory programming; Language model; Large language model; Natural languages; Openai; Problem description, Students |
| Department/Centre: | Division of Physical & Mathematical Sciences > Mathematics |
| Date Deposited: | 05 Apr 2023 06:44 |
| Last Modified: | 05 Apr 2023 06:44 |
| URI: | https://eprints.iisc.ac.in/id/eprint/81157 |
Actions (login required)
 |
View Item |
