idw – Informationsdienst Wissenschaft
Nachrichten, Termine, Experten
Nachrichten, Termine, Experten
Large-scale IU study demonstrates the potential of AI in enhancing grading fairness
• A team of researchers at IU has trained a new type of AI model using a large number of examination data sets from a variety of subjects at IU.
• The multi-level analysis of the data shows that AI-supported grading is able to reduce human subjectivity and inaccuracy.
• For legal and academic reasons, the research team recommends that AI should initially only be used to support human judgement.
• The AI model is applicable and adaptable across many disciplines.
Automating the grading of exams with open questions makes work considerably easier for teachers and can lead to better results for students by avoiding human inaccuracies and errors through an automated grading mechanism.
With their research paper 'Beyond human subjectivity and error: a novel AI grading system (2024)', a team from IU International University of Applied Sciences (IU) is demonstrating for the first time on a large scale how artificial intelligence can support teachers in automated grading. The IU research team has developed a novel grading system, called Automatic Short Answer Grading (ASAG), which can automatically grade answers to open-ended questions consisting of just a few sentences.
The ASAG system developed by the IU research team is based on a large language model, which was additionally trained with a very large dataset of exam data. The data comes from a variety of degree programmes at IU International University of Applied Sciences, covering a wide range of disciplines - from humanities to STEM subjects. According to the study, this broad spectrum should ensure that IU's ASAG model is applicable and adaptable across different disciplines.
AI model on average closer to benchmark score
In a multi-stage data analysis, the research team first used their ASAG model to show that AI can also analyse answers from previously unknown subject areas. A subsequent comparison with subject experts showed that, on average, ASAG's grading of student responses was closer to the official reference grade than the grading by subject experts. The greater agreement in grading suggests that AI-assisted grading can reduce human subjectivity in grading and potentially improve fairness.
"Our research provides further evidence of how AI will change the education system in the future. With this model, we aim to advance the idea of using AI in addition to human grading to increase consistency and fairness for students, while minimising negative impacts. This completely new approach not only promises more accurate grading results, but also opens unprecedented possibilities for the evolution of grading systems", says Dr Sven Schütt, CEO of IU International University of Applied Sciences.
"The integration of AI-supported models into the grading processes offers considerable advantages: Firstly, it provides examiners with an external benchmark for comparing their own assessment. This reduces variance and increases fairness, which benefits the students. Secondly, it relieves professors and tutors of tedious and repetitive tasks, allowing them to focus more on teaching and mentoring students”, says Prof Dr Thomas Zöller, Professor of Data Science and Artificial Intelligence at IU International University of Applied Sciences.
IU experts give an outlook for the future
According to the IU research team, the ASAG model could already be used as a rating system. However, according to IU experts, the legal and scientific prerequisites are not yet in place, as AI technology is still new and there is a lack of framework conditions, and new regulations are constantly emerging due to the rapid technological development in the field of AI. IU experts and auditors therefore recommend focusing initially on a model in which human auditors are supported by AI-based automation, for example for reconciliation and error prevention.
https://iu-international-university-of-applied-sciences-research-papers.s3.eu-ce...
https://www.iu.de/news/en/revolutionising-grading-iu-study-reveals-ai-potential-...
Grading errors for human and an AI generated grades
IU International University of Applied Sciences
Criteria of this press release:
Journalists, Scientists and scholars, Students, Teachers and pupils, all interested persons
Information technology
transregional, national
Research results, Transfer of Science or Research
English
Grading errors for human and an AI generated grades
IU International University of Applied Sciences
You can combine search terms with and, or and/or not, e.g. Philo not logy.
You can use brackets to separate combinations from each other, e.g. (Philo not logy) or (Psycho and logy).
Coherent groups of words will be located as complete phrases if you put them into quotation marks, e.g. “Federal Republic of Germany”.
You can also use the advanced search without entering search terms. It will then follow the criteria you have selected (e.g. country or subject area).
If you have not selected any criteria in a given category, the entire category will be searched (e.g. all subject areas or all countries).
