analysis As the legal and ethical implications of supporting AI models like Github’s Copilot continue to be clarified, computer scientists continue to find uses for large language models and urge educators to adapt.
Brett A. Becker, Assistant Professor at University College Dublin in Ireland provided the information The registry featuring pre-publication copies of two research papers examining the pedagogical risks and opportunities of AI programming code generation tools.
Papers were accepted at the SIGCSE Technical Symposium on Computer Science Education 2023, taking place March 15-18 in Toronto, Canada.
In June, GitHub Copilot, a machine learning tool that automatically suggests programming code in response to contextual prompts, emerged from a year-long technical preview amid concerns about the way its OpenAI Codex model was trained and the Effects of AI models on society merged into concentrated opposition.
Aside from unresolved copyright and software licensing issues, other computer scientists, such as University of Massachusetts Amherst computer science professor Emery Berger, have sounded the alarm that computer science pedagogy needs to be reevaluated given the expected proliferation and improvement of automated support tools.
In “Coding is hard – or at least it used to be: Educational opportunities and challenges of AI code generation” [PDF]Becker and co-authors Paul Denny (University of Auckland, Australia), James Finnie-Ansley (University of Auckland), Andrew Luxton-Reilly (University of Auckland), James Prather (Abilene Christian University, USA) and Eddie Antonio Santos (University College Dublin) argue that the education community needs to address the immediate opportunities and challenges presented by AI-driven code generation tools.
They say it’s reasonable to assume that computer science students are already using these tools to complete programming tasks. As such, policies and practices that reflect the new reality need to be devised sooner rather than later.
“We believe these tools will change the way programming is taught and learned in the near future – potentially significantly – and that they present numerous opportunities and challenges that warrant immediate discussion as we move towards the growing use of them.” Adjust tools.” the researchers state in their work.
These tools will change—potentially significantly—the way programming is taught and learned in the near future
The paper looks at several of the currently available assistive programming models, including GitHub Copilot, DeepMind AlphaCode, and Amazon CodeWhisperer, as well as less publicized tools such as Kite, Tabnine, Code4Me, and FauxPilot.
Given that these tools are moderately competitive with human programmers — for example, AlphaCode ranked in the top 54 percent of the 5,000 developers who entered Codeforces coding competitions — experts say AI tools can help students in a number of ways. This includes generating example solutions to help students validate their work, generating solution variations to increase students’ understanding of problems, and improving the quality and style of student code.
The authors also see benefits for educators, who could use auxiliary tools to create better student exercises, generate code explanations, and provide students with more descriptive examples of programming constructs.
Along with potential opportunities, there are challenges that educators must face. These code-emitting problem-solving tools could help students cheat on assignments more easily; The private nature of using AI tools reduces the risk of hiring a third party to do your homework.
The researchers also note that our way of thinking about attribution — which is central to defining plagiarism — may need to be revised, as supportive options can offer varying amounts of help, making it difficult to separate allowable from excessive support.
“In other contexts, we use spell checkers, grammar checking tools that suggest rephrasing, text prediction, and email auto-reply suggestions — all machine-generated,” the paper reminds us. “In a programming context, most development environments support code completion, which suggests machine-generated code.
We use spelling and grammar checking tools that suggest rephrasing…
“Distinguishing between different forms of machine suggestions can be challenging for academics, and it is unclear whether we can reasonably expect students unfamiliar with tool support to distinguish between different forms of machine-generated code suggestions. “
The authors say this raises a key philosophical question: “How much content can be machine-generated while the intellectual property is still attributed to a human?”
They also highlight how AI models fail to meet the attribution requirements set out in software licenses and fail to address ethical and environmental concerns about the energy used to create them.
The pros and cons of AI tools in education must be addressed, the researchers conclude, or educators will lose the opportunity to influence the development of this technology.
And they have little doubt it’s here to stay. The second paper, “Using Large Language Models to Enhance Programming Error Messages”, [PDF] provides an example of the potential value of large language models like Codex from Open AI, the foundation of Copilot.
The authors who applied were Juho Leinonen (Aalto University), Arto Hellas (Aalto University), Sami Sarsa (Aalto University), Brent Reeves (Abilene Christian University), Paul Denny (University of Auckland), James Prather (Abilene Christian University) and Becker Codex looked for typically cryptic computer error messages and found that the AI model can make errors more understandable by providing a simple English description – benefiting both teachers and students.
“Large language models can be used to create useful and beginner-friendly enhancements to programming error messages that sometimes outperform the original programming error messages in terms of interpretability and actionability,” the experts state in their article.
For example, Python might throw the following error message: “SyntaxError: unknown EOF while parsing.” Given the context of the affected code and the error, Codex would suggest this description to help the developer: “The error is caused because the code block follows the colon expected another line of code. To fix the problem, I would add another line of code after the colon.”
However, the results of this study say more about promise than actual benefit. Researchers fed the Codex model broken Python code and associated error messages to generate explanations for the problems, and rated these descriptions for: understandability; unnecessary content; have an explanation; have a correct explanation; have a solution; the correctness of the fix; and added value from the original code.
Results varied significantly in these categories. Most were understandable and included an explanation, but the model was much more successful in providing correct explanations for certain errors than others. For example, the error “cannot allocate function call” was correctly explained 83 percent of the time, while “unexpected EOF]while parsing” was correctly explained only 11 percent of the time. And the average overall fix of the error message was correct only 33 percent of the time.
“Overall, the reviewers felt that the content produced by Codex, ie the explanation of the error message and the proposed fix, represented an improvement over the original error message in just over half the cases (54 percent),” the report reads Paper.
The researchers conclude that while explanations of programming error messages and suggested fixes generated by large language models are not yet ready for production use and may mislead students, they believe that AI models can help in fixing code errors with more work could be sent.
Expect this work to keep the technology industry, academia, government and other interested parties engaged for years to come. ®
https://www.theregister.com/2022/10/20/ai_programming_tools_mean_rethinking/ AI Programming Tools May Rethink Compsci Education • The Register