Revolutionising grading: IU study reveals AI potential for fairer assessment
Large-scale IU study demonstrates the potential of AI in enhancing grading fairness
With their research paper 'Beyond human subjectivity and error: a novel AI grading system (2024)', a team from IU is demonstrating for the first time on a large scale how artificial intelligence can support teachers in automated grading.
- A team of researchers at IU has trained a new type of AI model using a large number of examination data sets from a variety of subjects at IU.
- The multi-level analysis of the data shows that AI-supported grading is able to reduce human subjectivity and inaccuracy.
- For legal and academic reasons, the research team recommends that AI should initially only be used to support human judgement.
- The AI model is applicable and adaptable across many disciplines.
Erfurt, Germany, 12 April 2024. Automating the grading of exams with open questions makes work considerably easier for teachers and can lead to better results for students by avoiding human inaccuracies and errors through an automated grading mechanism.
With their research paper 'Beyond human subjectivity and error: a novel AI grading system (2024)', a team from IU International University of Applied Sciences (IU) is demonstrating for the first time on a large scale how artificial intelligence can support teachers in automated grading. The IU research team has developed a novel grading system, called Automatic Short Answer Grading (ASAG), which can automatically grade answers to open-ended questions consisting of just a few sentences.
The ASAG system developed by the IU research team is based on a large language model, which was additionally trained with a very large dataset of exam data. The data comes from a variety of degree programmes at IU International University of Applied Sciences, covering a wide range of disciplines - from humanities to STEM subjects. According to the study, this broad spectrum should ensure that IU's ASAG model is applicable and adaptable across different disciplines.
AI model on average closer to benchmark score
In a multi-stage data analysis, the research team first used their ASAG model to show that AI can also analyse answers from previously unknown subject areas. A subsequent comparison with subject experts showed that, on average, ASAG's grading of student responses was closer to the official reference grade than the grading by subject experts. The greater agreement in grading suggests that AI-assisted grading can reduce human subjectivity in grading and potentially improve fairness.
Figure: Showing the distribution of absolute grading errors for both the human regraders (left, blue) and the model (right, orange). The median absolute errors are 20.0 and 11.1 respectively. The model’s median absolute error is hence 44% smaller than the humans’ median absolute error.
Source: ‘Beyond human subjectivity and error: a novel AI grading system’ (2024). IU International University of Applied Sciences.
"Our research provides further evidence of how AI will change the education system in the future. With this model, we aim to advance the idea of using AI in addition to human grading to increase consistency and fairness for students, while minimising negative impacts. This completely new approach not only promises more accurate grading results, but also opens unprecedented possibilities for the evolution of grading systems", says Sven Schütt, CEO of IU International University of Applied Sciences.
"The integration of AI-supported models into the grading processes offers considerable advantages: Firstly, it provides examiners with an external benchmark for comparing their own assessment. This reduces variance and increases fairness, which benefits the students. Secondly, it relieves professors and tutors of tedious and repetitive tasks, allowing them to focus more on teaching and mentoring students”, says Prof. Dr Thomas Zöller, Professor of Data Science and Artificial Intelligence at IU International University of Applied Sciences.
IU experts give an outlook for the future
According to the IU research team, the ASAG model could already be used as a rating system. However, according to IU experts, the legal and scientific prerequisites are not yet in place, as AI technology is still new and there is a lack of framework conditions, and new regulations are constantly emerging due to the rapid technological development in the field of AI. IU experts and auditors therefore recommend focusing initially on a model in which human auditors are supported by AI-based automation, for example for reconciliation and error prevention.
With over 130,000 students, IU International University of Applied Sciences (IU) is the largest university in Germany. The private, state-recognised educational institution with its headquarters in Erfurt began operations in 2000 and is now represented in more than 35 German cities. Students from over 190 nations design their studies according to their needs: whether practice-integrated dual studies, flexible distance learning or individual “myStudium”, which combines online self-study and campus life. IU wants to give people worldwide access to personalised education for a fulfilled and self-determined life. In more than 250 bachelor's, master's and MBA programmes, including over 50 in English, IU teaches students key skills relevant to the future. A digitally supported learning environment and the use of AI solutions help students to achieve optimal learning results and experiences. IU is one of the first universities in the world to have developed and deployed its own AI-supported learning buddy. IU cooperates with over 15,000 companies and supports them in the academic training of professionals. Partners include Motel One, Vodafone, AWO and Deutsche Bahn. Further information at: iu.org
The AI-supported learning buddy "Syntea" is now available as a public demo version in the ChatGPT store at the following link: https://chat.openai.com/g/g-P5olqoCPc-syntea
Contact
|
IU International University of Applied Sciences Press Department |