In this post\u00a0I will look at different ways of extending word embeddings with character-level information, in the context of neural sequence labeling models. \u00a0You can find more information in the Coling 2016 paper\u00a0“Attending to characters in neural sequence labeling models<\/a>“.<\/p>\n We’ll investigate word representations in order to improve on the task of sequence labeling. In a sequence labeling setting, a system gets a series of tokens as input and it needs to assign a label to every token. The correct\u00a0label typically depends on both the context and the token itself. Quite a large number of NLP tasks can be formulated as sequence labeling, for example:<\/p>\n POS-tagging<\/strong><\/em> Error detection<\/strong><\/em> Named entity recognition<\/strong><\/em> Chunking<\/strong><\/em> In each of these cases, the model needs to understand how a word is being used in a specific context, and could also take advantage of character-level patterns and morphology.<\/p>\n <\/p>\n Our baseline model for sequence labeling is as follows. Each word is represented as a 300-dimensional word embedding. This is passed through a bidirectional LSTM with hidden layers of size 200. The representations from both directions are concatenated, in order to get a word representation that is conditioned on the whole sentence. Next, we pass it through a 50-dimensional hidden layer and then an output layer, which can be a softmax or a CRF.<\/p>\n This configuration is based on a combination of my previous work on\u00a0error detection (Rei and Yannakoudakis, 2016<\/a>), and the models from Irsoy and Cardie (2014)<\/a> and\u00a0Lample et al. (2016)<\/a>.<\/p>\n Now let’s look at an architecture that builds word representations from individual characters. We process each word separately and map characters to character embeddings. Next, these are passed through a bidirectional LSTM and the last states from either direction are concatenated. The resulting vector is passed through another feedforward layer,\u00a0in order to map it to a suitable space and change the vector size as needed.\u00a0We then have a word representation m<\/strong>, built from individual characters.<\/p>\n We still have a normal word embedding x<\/strong> for each word, and\u00a0in order to get the best of both worlds, we can combine these two representations. Following Lample et al. (2016)<\/a>, one method is simply concatenating the\u00a0character-based representation with the word embedding.<\/p>\n \\( The resulting vector can then be used in the word-level sequence labeling model, instead of the regular word embedding. The whole network is connected together, so that the character-based component is also optimised during training.<\/p>\n Concatenating the two representations works, but we can do even better. We start off\u00a0the same – character embeddings are passed through a bidirectional LSTM to build a word representation m<\/strong>. However, instead of\u00a0concatenating this vector with the word embedding, we now combine them using dynamically predicted weights.<\/p>\n A vector of weights z<\/strong> is predicted by the model, as a function of x<\/strong> and m<\/strong>. In this case, we use a two-layer feedforward component, with tanh<\/em> activation in the first layer and sigmoid<\/em> on the second layer.<\/p>\n \\( Then, we combine\u00a0x<\/strong> and m<\/strong> as a weighted sum, using z<\/strong> as the weights:<\/p>\n \\( This operation essentially looks at both word representations and decides, for each feature, whether it wants to take the value from the word embedding or from the character-based representation. Values close to 1\u00a0in z<\/strong> indicate higher weight for the word embedding, and values close to 0 assign more importance to the character-based vector.<\/p>\n This combination requires that the two vectors are aligned – each feature position\u00a0in the character-based representation needs to capture the same properties as that position in the word embedding. In order to encourage this property, we actively optimise for these vectors to be similar, by maximising their cosine similarity:<\/p>\n \\( E is the main sequence labeling loss function that we minimise during training, T is the length of the sequence or sentence. Many OOV words share the same representation, and we do not want to optimise for this, therefore we use a variable g that limits\u00a0optimisation only\u00a0to non-OOV words.<\/p>\n In this setting, the model essentially learns two alternative representations for each word – a regular word embedding and a character-based representation. The word embedding itself is kind of a universal memory – we assign 300 elements to a word, and the model is free to save any information into it, including approximations of character-level information. For frequent words, there is really no reason to think that the character-based representation can offer much additional benefit. However, using word embeddings to save information is very inefficient – each feature needs to be learned and saved for every word separately. Therefore, we hope to get two benefits from including characters into the model:<\/p>\n While we optimise for the cosine similarity of m<\/strong> and x<\/strong> to be high, we are essentially teaching the model to predict distributional properties based only on character-level patterns and morphology. However, while\u00a0m<\/strong> is optimised to be similar to x<\/strong>, we implement it in such a way that x<\/strong> is not optimised to be similar to m<\/strong> (using\u00a0disconnected_grad<\/a> in Theano). Because word embeddings are more flexible,\u00a0we want them to store exceptions as opposed to learning more general patterns.<\/p>\n The resulting combined word representation is again plugged directly into the sequence labeling model. All the components, including the attention component for dynamically calculating z<\/strong>, are\u00a0optimised at the same time.<\/p>\n We evaluated the alternative architectures on 8 different datasets, covering 4 different tasks: NER, POS-tagging, error detection and chunking. See the paper for more detailed results, but here is a summary:<\/p>\n [table] As can be seen, including a character-based component into the sequence labeling model helps on every task. In addition, using the attention-based architecture outperforms character concatenation on all benchmarks.<\/p>\n We also compared\u00a0the number of parameters in each model. While both character-based models require more parameters compared to a basic architecture using only word embeddings, the attention-based architecture is actually more efficient compared to concatenation. If the vectors are simply concatenated, this increases the size of all the weight matrices in the proceeding LSTMs, whereas the attention framework\u00a0combines them without increasing length.<\/p>\n Miyamoto and Cho (2016)<\/a> have independently also proposed a similar architecture, with some\u00a0differences: 1) They focus on\u00a0the task of language modeling, 2) they predict a scalar weight for combining the representations as opposed to making the decision separately for each element, 3) they do not condition the weights on the character-based representation, and 4) they do not have the component that optimises character-based representations to be similar to the word embeddings.<\/p>\n Since this model is aimed at learning morphological patterns, you might ask why are we not giving\u00a0actual morphemes as input to the model, instead of starting from individual characters. The reason is that the definition of an informative morpheme is likely to change between tasks and datasets, and this allows the model to learn exactly what it finds most useful. The model for POS tagging can learn to detect specific suffixes, and the model for NER can focus more on capitalisation patterns.<\/p>\n Combining word embeddings with character-based representations makes neural models more powerful and allows us to have better representations for infrequent or unseen words. One option is to concatenate the two representations, treating them as separate sets of useful features. Alternatively, we can optimise them to be similar and combine them using a gating\u00a0mechanism, essentially allowing the model to choose whether it wants to take each feature from the word embedding of from the character-based representation. We evaluated on 8 different sequence labeling datasets and found that the latter option performed consistently better, even with a fewer number of parameters.<\/p>\n See the paper for more details: I\u00a0have made the code for running these experiments publicly available on github:Sequence labeling<\/h2>\n
\nDT<\/span> \u00a0NN<\/span> \u00a0 \u00a0VBD<\/span> \u00a0 \u00a0 \u00a0NNS<\/span> \u00a0 \u00a0IN \u00a0 \u00a0 \u00a0DT<\/span> \u00a0 DT<\/span> \u00a0NN<\/span> \u00a0 \u00a0 CC \u00a0DT<\/span> \u00a0NN<\/span> \u00a0 .
\nThe pound extended losses against both the dollar and the euro .<\/code><\/p>\n
\n+ + \u00a0 \u00a0+ \u00a0x<\/span>\u00a0 \u00a0 \u00a0 \u00a0+ \u00a0 + \u00a0 \u00a0 \u00a0+ \u00a0 + \u00a0 \u00a0+ \u00a0 \u00a0x<\/span>\u00a0 \u00a0 \u00a0 +
\nI like to playing the guitar and sing very louder .<\/code><\/p>\n
\nPER<\/span> _ \u00a0 \u00a0 \u00a0_ \u00a0 _ \u00a0 \u00a0 \u00a0_ \u00a0ORG<\/span> \u00a0ORG<\/span> \u00a0 _ \u00a0TIME<\/span> _
\nJim bought 300 shares of Acme Corp. in 2006 .<\/code><\/p>\n
\nB-NP<\/span> \u00a0 \u00a0B-PP B-NP<\/span> I-NP<\/span> B-VP<\/span> I-VP<\/span> \u00a0 \u00a0 I-VP<\/span> I-VP<\/span> \u00a0 B-PP B-NP<\/span> B-NP<\/span> \u00a0O
\nService on \u00a0 the \u00a0line is \u00a0 expected to \u00a0 resume by \u00a0 noon today .<\/code><\/p>\nBasic neural sequence labeling<\/h2>\n
![\"\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2016\/12\/baseline_graph-300x153.png\")
<\/a><\/p>\nConcatenating character-based word representations<\/h2>\n
![\"concat_graph\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2016\/12\/concat_graph-300x245.png\")
<\/a><\/p>\n
\n\\widetilde{x} = [x; m]
\n\\)<\/p>\nAttending to\u00a0character-based representations<\/h2>\n
![\"attention_graph\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2016\/12\/attention_graph-300x238.png\")
<\/a><\/p>\n
\nz = \\sigma(W^{(3)}_z tanh(W^{(1)}_{z} x + W^{(2)}_{z} m))
\n\\)<\/p>\n
\n\\widetilde{x} = z\\cdot x + (1-z) \\cdot m
\n\\)<\/p>\n
\n\\widetilde{E} = E + \\sum_{t=1}^{T} g_t (1 – cos(m^{(t)}, x_t)) \\hspace{3em}
\ng_t =
\n\\begin{cases}
\n0, & \\text{if}\\ w_t = OOV \\\\
\n1, & \\text{otherwise}
\n\\end{cases}
\n\\)<\/p>\n\n
Evaluation<\/h2>\n
\nDataset,Task,#labels,Measure,Word-based,Char concat,Char attn
\nCoNLL00,chunking,22,F1,91.23,92.35,92.67<\/strong>
\nCoNLL03,NER,8,F1,79.86,83.37,84.09<\/strong>
\nPTB-POS,POS-tagging,48,accuracy, 96.42 ,97.22,97.27<\/strong>
\nFCEPUBLIC,error detection,2,F0.5,41.24,41.27,41.88<\/strong>
\nBC2GM,NER,3,F1*,84.21,87.75,87.99<\/strong>
\nCHEMDNER,NER,3,F1,79.74,83.56,84.53<\/strong>
\nJNLPBA,NER,11,F1,70.75,72.24,72.70<\/strong>
\nGENIA-POS,POS-tagging,42,accuracy,97.39,98.49,98.60<\/strong>
\n[\/table]<\/p>\nConclusion<\/h2>\n
\nhttps:\/\/aclweb.org\/anthology\/C\/C16\/C16-1030.pdf<\/a><\/p>\n
\nhttps:\/\/github.com\/marekrei\/sequence-labeler<\/a><\/p>\n