| Publikationstyp: | Konferenz: Sonstiges |
| Art der Begutachtung: | Keine Angabe |
| Titel: | Automated assessment of writing and pragmatic competence : a critical review |
| Autor/-in: | Gautschi, Curtis |
| et. al: | No |
| Tagungsband: | 17th International Pragmatics Conference : abstracts |
| Seite(n): | 261 |
| Angaben zur Konferenz: | 17th International Pragmatics Conference (IPrA), Winterthur (online), 27 June - 2 July 2021 |
| Erscheinungsdatum: | 2021 |
| Verlag / Hrsg. Institution: | International Pragmatics Association (IPrA) |
| Sprache: | Englisch |
| Schlagwörter: | Automated assessment; Pragmatics; Writing assessment |
| Fachgebiet (DDC): | 410.285: Computerlinguistik 808: Rhetorik und Schreiben |
| Zusammenfassung: | The automated scoring of essays and other written texts (AES), by means of natural language processing tools, is playing an ever-increasing role in the evaluation of writing ability. While able to provide quick, cost-effective and consistent feedback to learners of foreign languages (Shermis & Burstein, 2013), the use of computers to assess writing skills has been met with considerable scepticism (Condon, 2013). Of particular concern is the issue of construct relevance and representation in the automated assessment of writing (Weigle, 2013), which has relied heavily on a narrow range of linguistic features (e.g., McNamara, Crossley & McCarthy, 2010), cohesion markers or complexity indices (e.g., Coh-index; see Allen, Jacovina, & McNamara, 2016). Thus, while aspects of organizational competence (e.g., grammatical, lexical, syntax and text-level organization) are to some degree measurable, the valid measurement of pragmatic competence remains a challenge for AES (e.g., communicative facets including message conveyance, communicative intent, sensitivity to genre, and the socially constructed transfer of meaning - see, Allen et al., 2016; Bridgeman, Powers, Stone & Mollaun, 2011; Deane, 2013). This has been acknowledged by leaders in the high-stakes AES industry in their efforts to improve the measurement of pragmatic aspects in the communicative paradigm for both automated assessment of writing and speaking (e.g., Chen et al., 2018; Evanini, Hauck & Hakuta, 2017). This presentation seeks to problematize the issues raised above. First, an AES tool and CEFR-level prediction algorithm created and experimentally implemented at a major University of Applied Sciences in Switzerland as part of an online English placement test for first-year engineering students (Gautschi, 2020) is critically examined with respect to its design, results and coverage of pragmatic competence aspects. Second the AES tool is placed within the context of current AES research and approaches to modelling pragmatic competence, with a view to critically reviewing: • current scoring model approaches • claims of communicative and pragmatic competence assessment • advancements in scoring accuracy through deep learning / neural network techniques • the reliance on human-machine agreement for validation support. Finally, it is argued that the value of future AES in testing crucially requires the active involvement of linguists and applied linguists to safeguard construct coverage of pragmatic competence in high-stakes assessment of communicative language ability by means of automated writing assessment. |
| URI: | https://digitalcollection.zhaw.ch/handle/11475/23455 |
| Volltext Version: | Publizierte Version |
| Lizenz (gemäss Verlagsvertrag): | Lizenz gemäss Verlagsvertrag |
| Departement: | Angewandte Linguistik |
| Organisationseinheit: | Institute of Language Competence (ILC) |
| Enthalten in den Sammlungen: | Publikationen Angewandte Linguistik |
Dateien zu dieser Ressource:
Es gibt keine Dateien zu dieser Ressource.
Zur Langanzeige
Gautschi, C. (2021). Automated assessment of writing and pragmatic competence : a critical review [Conference presentation]. 17th International Pragmatics Conference : Abstracts, 261.
Gautschi, C. (2021) ‘Automated assessment of writing and pragmatic competence : a critical review’, in 17th International Pragmatics Conference : abstracts. International Pragmatics Association (IPrA), p. 261.
C. Gautschi, “Automated assessment of writing and pragmatic competence : a critical review,” in 17th International Pragmatics Conference : abstracts, 2021, p. 261.
GAUTSCHI, Curtis, 2021. Automated assessment of writing and pragmatic competence : a critical review. In: 17th International Pragmatics Conference : abstracts. Conference presentation. International Pragmatics Association (IPrA). 2021. S. 261
Gautschi, Curtis. 2021. “Automated Assessment of Writing and Pragmatic Competence : A Critical Review.” Conference presentation. In 17th International Pragmatics Conference : Abstracts, 261. International Pragmatics Association (IPrA).
Gautschi, Curtis. “Automated Assessment of Writing and Pragmatic Competence : A Critical Review.” 17th International Pragmatics Conference : Abstracts, International Pragmatics Association (IPrA), 2021, p. 261.
Alle Ressourcen in diesem Repository sind urheberrechtlich geschützt, soweit nicht anderweitig angezeigt.