Although several algorithms from XAI (explainable artificial intelligence) ( Adadi and Berrada, 2018 Murdoch et al., 2019) have recently been published (e.g., LIME, SHAP) ( Ribeiro et al., 2016 Lundberg and Lee, 2017), no research has yet investigated the role that these explanation models (trained on top of predictive models) can play in: (a) discovering the decision-making process that drives AES, (b) fine-tuning predictive models to improve generalizability and interpretability, and (c) providing teachers and students with personalized, formative, and fine-grained feedback during the writing process. None has attempted to explain the whole decision process of AES, from holistic scores to rubric scores and from rubric scores to writing feature modeling. However, a vast swath of research tackles AES only holistically only a few have even developed AES models at the rubric level, the very first layer of explanation underlying the prediction of holistic scores ( Kumar et al., 2017 Taghipour, 2017 Kumar and Boulanger, 2020). In addition to the list of most globally important features, this study reports (a) a list of features that are important for a specific essay (locally), (b) a range of values for each feature that contribute to higher or lower rubric scores, and (c) a model that allows to quantify the impact of the implementation of formative feedback.Īutomated essay scoring (AES) is a compelling topic in Learning Analytics (LA) for the primary reason that recent advances in AI find it as a good testbed to explore artificial supplementation of human creativity. It leverages the state-of-the-art in natural language processing, applying feature selection on a pool of 1592 linguistic indices that measure aspects of text cohesion, lexical diversity, lexical sophistication, and syntactic sophistication and complexity. This study shows that faster (up to three orders of magnitude) SHAP implementations are as accurate as the slower model-agnostic one. As more hidden layers were added to the neural network, the descriptive accuracy increased by about 10%.
It has been found that the effect of deep learning can be best viewed when assessing the trustworthiness of explanation models.
In doing so, it evaluates the impact of deep learning (multi-layer perceptron neural networks) on the performance of AES. Building on previous studies where models were trained to predict both the holistic and rubric scores of essays, using the Automated Student Assessment Prize’s essay datasets, this study focuses on predicting the quality of the writing style of Grade-7 essays and exposes the decision processes that lead to these predictions. Although several algorithms from Explainable Artificial Intelligence have recently been published, no research has yet investigated the role that these explanation models can play in: (a) discovering the decision-making process that drives AES, (b) fine-tuning predictive models to improve generalizability and interpretability, and (c) providing personalized, formative, and fine-grained feedback to students during the writing process. Consequently, the AES black box has remained impenetrable. However, a vast swath of research tackles AES only holistically few have even developed AES models at the rubric level, the very first layer of explanation underlying the prediction of holistic scores. School of Computing and Information Systems, Faculty of Science and Technology, Athabasca University, Edmonton, AB, CanadaĪutomated essay scoring (AES) is a compelling topic in Learning Analytics for the primary reason that recent advances in AI find it as a good testbed to explore artificial supplementation of human creativity.
0 kommentar(er)
