Title: EFL placement testing for engineering students with automated essay scoring : new applications for natural language processing
Authors : Gautschi, Curtis
Conference details: 14. Ingenieurpädagogische Regionaltagung, Universität Bremen. 23-25 May, 2019
Issue Date: 2019
License (according to publishing contract) : Not specified
Type of review: Peer review (Abstract)
Language : English
Subjects : Automated essay scoring; Placement testing for writing; computerized assessment; EFL writing assessment
Subject (DDC) : 808: Rhetoric and writing
Abstract: The implementation of internationalisation strategies by higher education policymakers has led to a steady increase in English-taught programmes (Wächter & Maiworm 2014). The recognition of English as an economic (Ehrenreich 2010) and academic (Ljosland 2011) lingua franca also applies to Engineering, where the proficient use of English is highly valued (Minsch et al. 2017), and practical skills, such as the writing of technical reports, are expected outcomes from engineering students' university level education (Karras et al., 2015: 8-13). In spite of its importance, the placement testing of writing skills, regrettably, tends to be avoided on the grounds that it is too time-consuming. This is especially true when large numbers of new students are involved. However, advancements in computerized text analysis in the form of Automated Essay Scoring (AES) show great promise in responding to this need. This paper describes the design, results and further development of an AES tool and CEFR-level prediction algorithm that was created and experimentally implemented at a major University of Applied Sciences in Switzerland as part of an online English placement test for first-year engineering students. In line with current research in AES, the algorithm was developed employing a prediction-accuracy pseudo-black box approach (see Vanhove et al. 2019, Yannakoudakis 2013) using a small training corpus of texts with known CEFR levels (N=50). The tool's advantages will also be discussed. Written and run entirely in an R environment (R Core Team 2017, version 3.4.3) using the koRpus package (Michalke 2017) as the tool's workhorse, the user has complete control over the tool's implementation, and can integrate it with other advanced text analyses possible in R (e.g., text mining, word embedding). The algorithm requires a minimum of resources, is simple to use and is cost-effective. As the tool can handle bulk grading of large numbers of texts, it is ideal for placement testing. Further, the AES algorithm is efficient. In its experimental run, the AES tool calculated the CEFR levels of 400 student essays in 10 minutes of run time on an ordinary laptop computer. While gold-standard (human) validation evidence is still required, external validation tests demonstrate the accuracy of the tool. CEFR level prediction patterns were found to be closer to official CEFR levels of a selection of texts than grading by other online AES systems. Finally, further research perspectives and dissemination in the form of a shiny web app and an R package will be presented.
Departement: Applied Linguistics
Organisational Unit: Institute of Language Competence (ILC)
Publication type: Conference Other
URI: https://digitalcollection.zhaw.ch/handle/11475/17213
Appears in Collections:Publikationen Angewandte Linguistik

Files in This Item:
There are no files associated with this item.

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.