Incorporating AI generated feedback in business simulations
It has now been two months since we implemented AI generated feedback into two of our most popular simulations, the Sustainability Management Simulation: Net Zero and the Circular Economy Simulation: Fashion Forward. This article explains what we have done and reflects on the experiences to date.
AI in business simulations
There are different ways that AI can be used to enhance the experience of students or instructors. These include chatbots, AI powered simulation content and AI generated feedback for students (with and without grading). At Sim Institute we decided to start by integrating AI generated feedback to students based on the rationale that they provide for their decisions in the simulation. In this way, AI is used to support, rather than replace, critical thinking.
AI generated feedback
Simulations are a great way for students to experience how their decisions lead to business results. Through the Class Results section, instructors have the opportunity to monitor student decisions and overall performance, and support them in the learning process. However, it is not always clear to outside observers why a student has made certain decisions. In some cases, students may get a high score by chance, after having made decisions without adequate reflection. Until the advent of AI, it was very time consuming for instructors to demand students to explain their reasoning and then give individualized feedback to them. Now, with Sim Institute’s AI generated feedback feature, instructors can ask students to articulate their thought process and give them AI generated constructive and individualized feedback on their reasoning. The AI feedback feature is informed by a set of parameters that were defined for each decision. These parameters serve as a framework for the AI to formulate its feedback.
Currently, the new feature is incorporated into our Sustainability Management Simulation: Net Zero and the Circular Economy Simulation: Fashion Forward. In both simulations, students need to select projects to implement in order to either reduce their company’s greenhouse gas emissions or to move towards a circular business model, while at the same time optimizing financial performance.
If the instructor has enabled the AI generated feedback, then for each selected initiative students are invited to submit the rationale for their choice.
After submission, the dashboard includes all the usual feedback statements, charts and tables, and also AI generated feedback on the reasoning provided by students. Below is an example.
The AI feedback contains personalized and constructive comments on the reasoning that the student provided. If there are grammatical errors then these are pointed out as well.
Both students and instructors can access this information for all years in the simulation at any time, including when their run is finished. Instructors can share and discuss the reasoning provided by students during the debrief or give further personalized feedback to individual students.
If the AI generated feature is enabled, it is not compulsory for students to enter the rationale to progress in the simulation. Instructors may tell students to enter reasoning for every decision in each year of the simulation or perhaps just for a subset of decisions (e.g. five decisions in total or all decisions in certain years). In this way, instructors can balance the value of the feature with the additional seat time that will result from its use.
AI grading?
We believe that in this way, students are optimally supported in their learning. Students also practice the formulation of coherent reasoning for decisions, a skill that risks being eroded by the use of AI in writing.
We have decided against giving AI generated grades for student work, leaving grading up to the instructor based on rubrics and criteria that are appropriate for the specific group of students in the course. In practice, the same work will get different grades in different circumstances, depending on factors such as the type of course (undergraduate or postgraduate), the ability of students in the class, their native language, the course subject matter, and more.
Challenges
AI generated feedback has been well received by users of our simulations. Students and instructors have shared with us that it helps the learning process and even leads students to make better decisions.
One benefit of the AI generated feedback is that it is original and does not sound formulaic. On the flipside, it still happens occasionally that a feedback sentence does not make sense to the student (hallucinations).
Another consideration is that students sometimes just want to move on with their decisions and don’t want to spend too much time writing their justifications. If students are under time pressure, it is best to minimize the use of the feature, for example by asking them to submit their reasoning only for a small number of decisions.
The way forward
So far, AI feedback is only given for yes/no decisions, where it is relatively straightforward to express a rationale. For decisions that require an amount to be filled in (e.g. prices, salary levels, expenditure on training), it is not easy to define if a decision represents a ‘low’ or a ‘high’ amount and it is challenging to define on what basis feedback should be provided. Going forward, we will work to make AI generated feedback available across a wider range of decisions and simulations. These efforts will be informed by continuing feedback from students and instructors, which we encourage you to share with us!
Find out more on www.siminstitute.com
