Process overviewAbstract
The emergence of ChatGPT in late 2022 was a milestone in the field of Generative Artificial Intelligences, but it also caused a major stir in the academic world. For the first time, a simple interface allowed anyone to access a large language model and use it to generate text. These capabilities can undoubtedly have a relevant impact on teaching-learning methodologies and also on evaluation methods. With the aim of obtaining a real measure of ChatGPT’s possible performance in solving exams, it has been tested with exams from 15 Software Engineering courses in a Computer Engineering degree. In light of the results, ChatGPT is capable of achieving relevant performance in these exams; as it is able to pass a significant number of questions and problems of different nature in multiple courses. As a fundamental contribution, a detailed study of the results by question and problem typology is provided, which allows establishing recommendations to be taken into account in the design of evaluation methods. Additionally, an analysis of the impact of ChatGPT’s non-deterministic aspect on multiple-choice question answers is presented.
Roberto Rodriguez-Echeverria
INTIA Director and Associate Professor
Associate Professor at the University of Extremadura. Software passionate, Deep learner, MTB rider and father of 2.
José M. Conejero
Associate Professor
Associate Professor at the University of Extremadura. Research interests include model-driven development, data science, and machine learning.
Alvaro E. Prieto
Associate Professor
Associate Professor and Director of Electronic Administration at the University of Extremadura.