ChatGPT vs. the State University of New York Purchase College Mathematics/Computer Science Majors: Can These Students Defeat the AI Chatbot?

Abstract

This study aims to determine if the Artificial Intelligence chatbot, ChatGPT, can perform better on assignments, tests, and labs than Mathematics/Computer Science students at the State University of New York, Purchase College. Student grades and these materials are from Calculus II, Computer Science I, and Number Theory classes. The different versions of ChatGPT that were asked questions from the materials students completed are: GPT-3.5, GPT-4, and Wolfram GPT. The answers and grades that these chatbots generated were compared to each other and the students' answers and average grades. For each of the assignments, labs, and exams, the comparisons are based on which grade is the highest, out of the three different versions of ChatGPT and the students, as well as whether or not their answers utilize the most efficient strategy and is concise. GPT-4 outdid GPT-3.5 on Computer Science I problems and Wolfram GPT on Calculus II materials. Wolfram GPT eclipsed all versions of ChatGPT on the Number Theory exam. Yet, overall, students did better than all versions of ChatGPT. However, there are caveats to this finding because of the limitations this study encountered. On average one student from each class participated, thus the results are not totally conclusive. However, they do indicate that ChatGPT cannot be trusted to do math and computer science related tasks without human oversight. Therefore, corporations and students cannot wholly rely on ChatGPT. Based on ChatGPT's performance so far, it is likely in the future that it will learn enough to become as formidable as some fear.