idw – Informationsdienst Wissenschaft

Nachrichten, Termine, Experten

Grafik: idw-Logo
Grafik: idw-Logo

idw - Informationsdienst
Wissenschaft

Science Video Project
idw-Abo

idw-News App:

AppStore

Google Play Store



Instanz:
Teilen: 
27.11.2023 13:12

Passau study shows: ChatGPT writes the better school essays

Kathrin Haimerl Abteilung Kommunikation
Universität Passau

    In a study published in the Nature journal "Scientific Reports", a research team from the University of Passau compared the quality of machine-generated content with essays written by secondary school students. The upshot: The AI-based chatbot performed better across all criteria, especially when it came to language mastery.

    The language model ChatGPT is making enormous progress. After version 3.5 had failed the Bavarian Abitur in early 2023, its successor version 4 earned a solid 2 nearly six months later.

    A study by the University of Passau has now been able to demonstrate to what extent AI-generated content could revolutionise the school system. The researchers also experimented with the two language model versions. In a study entitled "A large-scale comparison of human-written versus ChatGPT-generated essays" and published in the prestigious nature journal "Scientific Report", they concluded that the machine writes the better English essays. They had evaluated machine-generated texts and essays written by secondary school students according to guidelines established by the Ministry of Education of Lower Saxony.

    "I was surprised by how clear the outcome was," says Professor Steffen Herbold, who holds the Chair of AI Engineering at the University of Passau and initiated the study. Both Open AI chatbot versions scored higher than the students, with GPT-3 ranking in the middle and GPT-4 achieving the best score. "This shows that schools shouldn't turn a blind eye on these new tools."

    Reflecting on AI models

    The interdisciplinary study was carried out by the computer scientists in collaboration with computer linguist Professor Annette Hautli-Janisz and computer science didactician Ute Heuer. "I find it important to prepare teachers for the challenges and opportunities coming their way as artificial intelligence models become increasingly available," says computer science didactician Heuer.

    She initiated a training course on "ChatGPT – Opportunity and Challenge" that the research team conducted. This event, which had taken place in March 2023, was attended by 139 teachers, most of whom teach at German gymnasiums. The teachers were first briefed on selected technological ideas behind general text generators and ChatGPT. The practical stage then specifically involved English-language texts where the training course participants were left unaware of the origin of these texts.

    Using questionnaires, the teachers were asked to evaluate the essays presented to them based on grading scales established by the Ministry of Education of Lower Saxony. Content was evaluated based on the criteria topic, completeness, and logic as well as linguistic aspects like vocabulary, complexity, and language mastery. The research team from Passau defined a scale from 0 to 6 for each criterion, with 0 being the worst score and 6 the best.

    The machine scores above average in language mastery

    One hundred eleven teachers completed the entire questionnaire and evaluated a total of two hundred seventy English language essays. The research team found the biggest difference in language mastery where the machine scored 5.25 (GPT-4) and 5.03 points (GPT-3) respectively, whereas the students scored an average of 3.9 points. "This does not mean that students have poor English language skills. Rather, the scores achieved by the machine are exceptionally high," underscores Annette Hautli-Janisz, Junior Professor of Computational Rhetoric and Natural Language Processing at the University of Passau.

    For Hautli-Janisz, who analysed the texts from a linguistic perspective together with doctoral student Zlata Kikteva, the study provides further exciting insights into the machine's language development. "We have seen how the models change over time and are able to demonstrate with our studies that they have improved in performing the task we give them." The researchers have also been able to identify differences between human and machine-generated language: "When we read more AI-generated texts going forward, we'll have to ask ourselves whether and how that affects our human language," says Hautli-Janisz.

    About the research team

    Professor Steffen Herbold holds the Chair of AI Engineering at the University of Passau. In his research, he focuses on the quality of AI models. For the present study, he collaborated with Dr Alexander Trautsch to run the statistical analysis and set up a data collection platform. He had developed the study design together with Professor Hautli-Janisz.

    Annette Hautli-Janisz is Junior Professor of Computational Rhetoric and Natural Language Processing. Her research interest is in finding out how the argumentative skills of AI-powered language models develop. Besides carrying out the computer linguistic analysis in the study, she also came up with the idea of using a dataset of English language essays written by students available at the Technical University of Darmstadt. The essays were from an online homework forum where students had asked native speakers for feedback to improve their texts. This dataset has been used repeatedly in research.

    Ute Heuer is Akademische Direktorin and leads the team in charge of Computer Science Education at the University of Passau. She is responsible for computer science teacher education at the University of Passau, and her research include the didactics of programming education. As part of her work, she initiates training courses for teachers in order to give them an awareness of the opportunities and challenges resulting from the availability of artificial intelligence models.


    Wissenschaftliche Ansprechpartner:

    Professor Steffen Herbold
    Chair of AI Engineering
    Hans-Kapfinger-Straße 30
    94032 Passau
    E-mail: Steffen.Herbold@uni-passau.de

    Professor Annette Hautli-Janisz
    Junior Professor of Computational Rhetoric und Natural Language Processing
    Hans-Kapfinger-Straße 30
    94032 Passau
    E-mail: Annette.Hautli-Janisz@uni-passau.de

    Ute Heuer
    Computer Science Education
    Innstraße 33
    94032 Passau
    E-mail: Ute.Heuer@uni-passau.de


    Originalpublikation:

    https://www.nature.com/articles/s41598-023-45644-9


    Bilder

    Professor Steffen Herbold holds the Chair of AI Engineering at the University of Passau.
    Professor Steffen Herbold holds the Chair of AI Engineering at the University of Passau.
    University of Passau
    University of Passau

    Annette Hautli-Janisz is Junior Professor of Computational Rhetoric and Natural Language Processing at the University of Passau.
    Annette Hautli-Janisz is Junior Professor of Computational Rhetoric and Natural Language Processing ...
    University of Passau
    University of Passau


    Merkmale dieser Pressemitteilung:
    Journalisten, Lehrer/Schüler, Studierende, Wissenschaftler, jedermann
    Gesellschaft, Informationstechnik, Pädagogik / Bildung
    überregional
    Forschungsergebnisse, Schule und Wissenschaft
    Englisch


     

    Hilfe

    Die Suche / Erweiterte Suche im idw-Archiv
    Verknüpfungen

    Sie können Suchbegriffe mit und, oder und / oder nicht verknüpfen, z. B. Philo nicht logie.

    Klammern

    Verknüpfungen können Sie mit Klammern voneinander trennen, z. B. (Philo nicht logie) oder (Psycho und logie).

    Wortgruppen

    Zusammenhängende Worte werden als Wortgruppe gesucht, wenn Sie sie in Anführungsstriche setzen, z. B. „Bundesrepublik Deutschland“.

    Auswahlkriterien

    Die Erweiterte Suche können Sie auch nutzen, ohne Suchbegriffe einzugeben. Sie orientiert sich dann an den Kriterien, die Sie ausgewählt haben (z. B. nach dem Land oder dem Sachgebiet).

    Haben Sie in einer Kategorie kein Kriterium ausgewählt, wird die gesamte Kategorie durchsucht (z.B. alle Sachgebiete oder alle Länder).