Accuracy and Effectiveness of an Orchestration Tool on Instructors' Interventions and Groups' Collaboration

Authors

LuEttaMae Lawrence, Utah State UniversityFollow
Emma Mercier, University of IllinoisFollow
Taylor Tucker Parks, University of IllinoisFollow
Nigel Bosch, University of IllinoisFollow
Luc Paquette, University of IllinoisFollow

Document Type

Article

Journal/Book Title/Conference

Computers and Education Open

Author ORCID Identifier

LuEttaMae Lawrence https://orcid.org/0000-0001-6066-5096

Taylor Tucker Parks https://orcid.org/0009-0005-3311-7795

Nigel Bosch https://orcid.org/0000-0003-2736-2899

Luc Paquette https://orcid.org/0000-0002-2738-3190

Volume

7

Publisher

Elsevier Ltd

Publication Date

7-21-2024

Journal Article Version

Version of Record

First Page

1

Last Page

14

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 License.

Abstract

This paper presents the development of a novel orchestration tool that predicts collaborative problem-solving (CPS) behaviors of undergraduate engineering groups and investigates the use of that tool by instructors. We explore the impact of receiving real-time, machine-learning, model-based prompts on 1) instructors' orchestration strategies, which are strategies instructors use to manage and facilitate collaborative activities, and 2) groups' participation, including how groups are engaged in CPS activities. The orchestration tool is a dashboard that notifies instructors of–and advises them on–monitoring and intervening with groups who may need collaborative support and guidance. We describe the accuracy of the models in predicting CPS behaviors and of instructors in identifying these behaviors in the classroom. We then describe how real-time prompts from models can affect instructors' orchestration strategies and students' participation. Our findings show that there is variability in the accuracy of our machine learning models and that instructors are better at identifying predictive behaviors as compared to the models. Instructors in this context engaged in orchestration strategies, like monitoring and probing when using the orchestration tool, and groups of students were largely talking while on-task across classes. We triangulate across data sources to examine the effectiveness of the orchestration tool in the classroom and share pedagogical and technical implications for the field.

Recommended Citation

LuEttaMae Lawrence, Emma Mercier, Taylor Tucker Parks, Nigel Bosch, Luc Paquette, Accuracy and effectiveness of an orchestration tool on instructors' interventions and groups' collaboration, Computers and Education Open, Volume 7, 2024, 100203, ISSN 2666-5573, https://doi.org/10.1016/j.caeo.2024.100203.