Project proposal for the 2017/2018 MPhil course in Advanced Computer Science.
Note: Please discuss with me before applying for this project. I am available for meetings either before Nov 13th or after November 17th.
Language Modelling for Neural Error Detection
Proposer: Marek ReiSupervisor: Marek Rei
Description
Automated error detection requires systems that are able to identify various types of errors in texts written by language learners.
This includes basic errors, such as spelling or punctuation mistakes, but also more complex errors such as word order, verb agreement, incorrect idioms, etc.
For example:
_ _ X X _ _ _ _ _ _
We can invite also people who are not members .
_ _ X _ _ _ _ _
My friend eats two ice creams yesterday .
Recent work has shown that neural bidirectional LSTM models perform well at this task (Rei and Yannakoudakis (2016), Rei et al. (2016)). However, these models are trained using hand-annotated datasets, which are much smaller than available unlabeled corpora.
Rei (2017) showed that using a language modelling objective on the same labelled data improved performance for error detection, among other sequence labelling tasks. Peters et al. (2017) proposed a method for explicitly including unlabelled data, by training a separate language model and integrating features from its hidden state into the sequence labeller. They showed performance improvements on the CoNLL 2000 chunking datatset and CoNLL 2003 named entity recognition dataset.
This project will extend the latter approach to the task of error detection in learner texts. Since language models are designed to detect unusual word sequences and require only unlabelled data for training, their features can potentially be quite useful for the error detection model.
- The impact of language models trained on different corpora can be investigated. For example BNC, Wikipedia, UKWaC, news/social media, original or corrected texts from language learners. Several of these can also be combined, either at the corpus level or feature level.
- The effect on specific error types can be investigated using the toolkit by Bryant et al, 2017.
- Alternative types of language models can also be experimented with. For example, using output probabilities from a language model would allow for n-gram models to also be included. Yannakoudakis et al. (2011) used a method of checking n-grams for presence/absence in a large corpora, as an approximation of error detection, which can be included as an external feature.
- External spell checkers can also be considered very basic language models, while potentially providing the error detection systems with complementary information.
- While the new features will be targeting error detection, their impact on other sequence labeling tasks (NER, chunking, POS tagging) can also be investigated.
The existing error detection system, written in Theano, is available online:
https://github.com/marekrei/sequence-labeler
A Tensorflow version of this code will also be available soon. The project can start from this codebase and modify it to include the necessary extensions.
For evaluating performance on error detection, the manually labelled FCE (Yannakoudakis et al., 2011) and CoNLL14 (Ng et al., 2014) datasets are available.
References:
Compositional Sequence Labeling Models for Error Detection in Learner Writing
Marek Rei and Helen Yannakoudakis. 2016.
Attending to characters in neural sequence labeling models
Marek Rei, Sampo Pyysalo and Gamal K.O. Crichton. 2016.
Semi-supervised Multitask Learning for Sequence Labeling
Marek Rei. 2017.
Semi-supervised sequence tagging with bidirectional language models
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power. 2017
Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction
Christopher Bryant Mariano Felice Ted Briscoe. 2017
A New Dataset and Method for Automatically Grading ESOL Texts
Helen Yannakoudakis, Ted Briscoe, Ben Medlock. 2011
The CoNLL-2014 Shared Task on Grammatical Error Correction
Ng et al. 2014