Study Suggests ChatGPT Solves SAT Problems Better Than Undergraduates

(CTN News) – According to a new study, ChatGPT can solve problems at a level equal to or better than that of an undergraduate student.

When asked to solve reasoning problems that appear on intelligence tests or exams, such as the American college admission test, the SAT, researchers found that the GPT-3 large language model underpinning the chatbot performed about as well as US college undergraduates.

ChatGPT -3 was tested by psychologists at the University of California, Los Angeles by converting images into a text format that the model could process and by ensuring the model had never encountered such questions before, after converting the images to a text format that the model could process.

In a study of 40 UCLA undergraduates, the researchers found that ChatGPT -3 solved 80% of the problems correctly, well above the average of just below 60% for the human participants.

Furthermore, the researchers tasked the model with solving SAT “analogy” questions that they believe had not been published on the internet and, therefore, could not have been included in the large amount of data the model was trained on.

According to the UCLA team, the AI outperformed the average SAT score for college applicants.

The model performed poorly in another test. A passage of prose was matched with a short story conveying the same meaning.

This study found that GPT-3 performed worse than students, while GPT-4 – its improved successor – performed better.

Accordin to Taylor Webb, the lead author of the study, ChatGPT does not meet the standards of artificial general intelligence, or human intelligence.

There is no doubt that it isn’t a fully general intelligence at the level of humans. In a particular area, it has certainly made progress” said Webb, a UCLA postdoctoral researcher in the field of psychology.

The UCLA researchers added that without access to the inner workings of ChatGPT -3, which is developed by the San Francisco-based company OpenAI, they were unable to determine how the model’s reasoning abilities work and whether it has a human-like reasoning ability or is exhibiting a completely new kind of intelligence.

The ChatGPT -3 might be thinking a little like a human after all, said Keith Holyoak, a professor of psychology at UCLA. “However, people do not learn by ingesting the entire internet in a single sitting, so the training method is completely different,” he added.

In the end, it would be nice if we could find out if it’s really doing it like people do, or if it’s doing something brand new – a true artificial intelligence – which would just be amazing in and of itself.

SEE ALSO:

Twitter’s X Logo Sparks Musk-San Francisco Controversy