{"id":214,"date":"2014-10-18T12:13:28","date_gmt":"2014-10-18T12:13:28","guid":{"rendered":"http:\/\/www.marekrei.com\/blog\/?p=214"},"modified":"2019-09-27T23:37:15","modified_gmt":"2019-09-27T23:37:15","slug":"multilingual-semantic-models","status":"publish","type":"post","link":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/","title":{"rendered":"Multilingual Semantic Models"},"content":{"rendered":"

In this post I’ll discuss a model for learning word embeddings, such\u00a0that they end up\u00a0in the same space in different languages. This means we can find the similarity between some\u00a0English and German words, or even compare the meaning of two sentences in different languages. It is a summary and analysis of the paper by Karl Moritz Hermann and Phil Blunsom, titled\u00a0“Multilingual Models for Compositional Distributional Semantics<\/a>“, published at ACL 2014.<\/p>\n

The Task<\/h2>\n

The goal of this work is to extend the distributional hypothesis to multilingual data and joint-space embeddings. This would give\u00a0us the ability\u00a0to compare words\u00a0and sentences in different languages, and also make use of labelled training data from languages other than the target language. For example, below is an illustration of English words and their Estonian translations in the same semantic space.<\/p>\n

\"vector_space_model_multilingual\"<\/a><\/p>\n

 <\/p>\n

This actually turns out to be a very difficult task, because the distributional hypothesis stops working across different languages. While “fish” is an important feature of “cat”, because they occur together often, “kass” never occurs with “fish”, because they are in different languages and therefore used in separate sets of documents.<\/p>\n

In order to learn these representations in the same space, the authors construct a neural network that learns from parallel sentences (pairs of the same sentence in different languages). The model is then evaluated\u00a0on the task of topic classification, training on one language and testing on the other.
\n
\nA bit of a spoiler, but here is a visualisation of some words from the final model, mapped into 2 dimensions.<\/p>\n

\"multilingual_space1\"<\/a><\/p>\n

 <\/p>\n

The words from English, German and French are successfully mapped into clusters based on meaning. The colours indicate gender (blue=male, red=female, green=neutral).<\/p>\n

The Multilingual Model<\/h2>\n

The main idea is as follows: We have sentence \\(a\\) in one language, and we have a function \\(f(a)\\) which\u00a0maps that sentence into a vector representation (we’ll come back to that function). We then have\u00a0sentence \\(b\\), which is the same sentence just in a different language, and function \\(g(b)\\) for mapping it into a vector representation. Our goal is to have\u00a0\\(f(a)\\) and\u00a0\\(g(b)\\) be identical, because both of these sentences have the same meaning. So during training, we show the model a series of parallel sentences \\(a\\) and\u00a0\\(b\\), and each time we adjust the functions\u00a0\\(f(a)\\) and\u00a0\\(g(b)\\) so that they would produce more similar vectors.<\/p>\n

Here is a graphical representation of the model:<\/p>\n

\"\"<\/a><\/p>\n

 <\/p>\n

\\(b1\\),\u00a0\\(b2\\) and\u00a0\\(b3\\) are words in sentence\u00a0\\(b\\);\u00a0\\(a1\\),\u00a0\\(a2\\),\\(a3\\)\u00a0and\u00a0\\(a4\\) are words in sentence\u00a0\\(a\\). The red vectors in the middle are\u00a0the sentence representations that we want to be similar.<\/p>\n

Next, let’s talk about the functions \\(f(a)\\) and\u00a0\\(g(b)\\) that map a sentence to a vector. As you can see from the image above, each word is represented as a vector as well. The simplest option of going from words to sentences is to just add the individual word vectors together (the ADD model):<\/p>\n

\\(f_{ADD}(a) = \\sum_{i=1}^{n} a_i\\)<\/p>\n

Here, \\(a_i\\) is the vector for word number \\(i\\) in sentence \\(a\\). This addition is similar to a basic bag-of-words model, because it doesn’t preserve any information about the order of the words. Therefore, the authors have also proposed a bigram version of this function (the BI model):<\/p>\n

\\(f_{BI}(a) = \\sum_{i=1}^{n} tanh(a_{i-1} + a_i)\\)<\/p>\n

In this function, we step though the sentence, add together vectors for two consecutive words, and pass them through a nonlinearity (tanh). The result is then summed together into a sentence vector. This is essentially a multi-layer compositional network, where word vectors are first combined to bigram vectors, and then bigram vectors are combined to sentence vectors.<\/p>\n

One more component to make this model work – the optimization function. The authors define an energy function given\u00a0two sentences:<\/p>\n

\\(E(a,b) = || f(a) – g(b)\u00a0||^ 2\\)<\/p>\n

This means we find the Euclidean distance<\/a> between the two vector representations and take the square of it. This value will be big when the vectors are different, and small when they are similar.<\/p>\n

But we can’t directly\u00a0use this for optimization, because functions \\(f(a)\\) and\u00a0\\(g(b)\\) that always returned zero vectors would be the most optimal solution. We want the model to give similar vectors for similar sentences, but different vectors for semantically different sentences. Here’s a function for that:<\/p>\n

\\(E_{nc}(a,b,c) = [m + E(a,b) – E(a,c)]_{+}\\)<\/p>\n

We’ve introduced\u00a0a randomly selected sentence \\(c\\) that probably has nothing to do with\u00a0\\(a\\) or\u00a0\\(b\\). Our objective is to minimze the\u00a0\\(E_{nc}(a,b,c)\\) function, which means we want\u00a0\\(E(a,b)\\) (for related sentences) to be small, and\u00a0\\(E(a,c)\\) (for unrelated sentences) to be large. This form of training – teaching the model to distinguish between correct and incorrect samples – is called noise contrastive estimation. The formula also includes \\(m\\), which is the margin we want to have between the values of \\(E(a,b)\\) and\u00a0\\(E(a,c)\\).\u00a0The whole thing is passed through the function \\([x]_{+} = max(x,0)\\), which means that if\u00a0\\(E(a,c)\\) is greater than\u00a0\\(E(a,b)\\) by margin \\(m\\), then we’re already optimal and don’t need to adjust the model further.<\/p>\n

The authors also experiment with\u00a0a document-level variation of the model (the DOC model), where individual sentence vectors are combined into document vectors\u00a0and these are also optimized to be similar, in addition to the sentence vectors.<\/p>\n

Evaluation<\/h2>\n

The authors evaluate the system on the task of topic classification. The classifier is trained on one language (eg English) and the test results are reported on another language (eg German) for which no labelled training data was used. They run\u00a0two main experiments:<\/p>\n

    \n
  1. The cross-lingual document classification (CLDC<\/strong>) task, described by Klementiev et al. (2012). The system is trained on the parallel Europarl<\/a> corpus, and tested on Reuters RCV1\/RCV2. The language pairs used were English-German and English-French.<\/li>\n
  2. The authors built a new corpus from parallel subtitles of TED<\/strong> talks<\/a>\u00a0(not yet online at the time of writing this), based on a previous TED corpus for IWSLT<\/a>.\u00a0Each talk also has topic tags assigned to them, and the task is to assign a correct tag to every talk, using the document-level vector.<\/li>\n<\/ol>\n

     <\/p>\n

    Results<\/h2>\n

    First, results on the CLDC task:<\/p>\n

    \"\"<\/a><\/p>\n

    I-Matrix is the previous state-of-the-art system, and all the models described here manage to outperform it. The +-variations of the model (ADD+ and BI+) use the French data as an extra training resource,\u00a0thereby improving performance. This is an interesting result, as the classifier is trained on English and tested on German, and French seems completely unrelated to the task. But by adding it into the model, the English word representations are improved by having more information available, which in turn propagates on to having better German representations.<\/p>\n

    Next, experiments on the TED corpus:<\/p>\n

    \"multilingual_results2\"<\/a><\/p>\n

    The authors have performed a much larger number of experiments, and\u00a0I’ve only chosen\u00a0a few examples to show here.<\/p>\n

    The MT System is a machine translation baseline, where the test data is translated into the source language using a machine translation system. The most interesting scenarios for application are where the source language is English, and this is where the MT baseline often still\u00a0wins. So if we want to topic classification in Dutch, but we only have English labelled data, it’s best to just automatically translate the Dutch text into English before classification.<\/p>\n

    Experiments in the other direction (where the target language is English) show different results, and the multilingual neural models manage to win\u00a0on most languages. I’m curious about this difference – perhaps the MT systems are better tuned\u00a0to translate into English, but not as good when translating from English into other languages? In any case, I think with some additional developments the neural network model will be able to beat the baseline in both directions.<\/p>\n

    In most cases, adding the document-level training signal (ADD\/DOC) helped accuracy quite a bit. The bigram models (BI) however were outperformed by the basic ADD models on this task, and the authors suspect this is due to sparsity issues caused by less training data.<\/p>\n

    Finally, the ADD\/DOC\/joint model was trained on all languages simultaneously, taking advantage of parallel data in all the languages, and mapping all vectors into the same space. The results of this experiment seem to be mixed, leading to an improvement on some languages and decrease on others.<\/p>\n

    In conclusion, this is definitely a very interesting model, and it bridges the gap between vector representations of different languages, using only sentence-aligned plain text. Combining sentence-level and document-level training signals seems to give a fairly consistent improvement in classification accuracy. Unfortunately, in the most interesting scenario, mapping from English to other resource-poor languages, this system does not yet beat the MT baseline. But hopefully this is only the first step, and future research will further improve the results.<\/p>\n

    References<\/h2>\n

    Hermann, K. M., & Blunsom, P. (2014). Multilingual Models for Compositional Distributed Semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics<\/i> (pp. 58\u201368).<\/p>\n

    Klementiev, A., Titov, I., & Bhattarai, B. (2012). Inducing crosslingual distributed representations of words. In COLING 2012<\/i>.<\/p>\n","protected":false},"excerpt":{"rendered":"

    In this post I’ll discuss a model for learning word embeddings, such\u00a0that they end up\u00a0in the same space in different languages. This means we can…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-214","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nMultilingual Semantic Models - Marek Rei<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Multilingual Semantic Models - Marek Rei\" \/>\n<meta property=\"og:description\" content=\"In this post I’ll discuss a model for learning word embeddings, such\u00a0that they end up\u00a0in the same space in different languages. This means we can…\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/\" \/>\n<meta property=\"og:site_name\" content=\"Marek Rei\" \/>\n<meta property=\"article:published_time\" content=\"2014-10-18T12:13:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-09-27T23:37:15+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png\" \/>\n<meta name=\"author\" content=\"Marek\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marek\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/\",\"url\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/\",\"name\":\"Multilingual Semantic Models - Marek Rei\",\"isPartOf\":{\"@id\":\"https:\/\/www.marekrei.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png\",\"datePublished\":\"2014-10-18T12:13:28+00:00\",\"dateModified\":\"2019-09-27T23:37:15+00:00\",\"author\":{\"@id\":\"https:\/\/www.marekrei.com\/blog\/#\/schema\/person\/a145eb0a06ed4acf5b0f84a24b7a1191\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#primaryimage\",\"url\":\"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png\",\"contentUrl\":\"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.marekrei.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Multilingual Semantic Models\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.marekrei.com\/blog\/#website\",\"url\":\"https:\/\/www.marekrei.com\/blog\/\",\"name\":\"Marek Rei\",\"description\":\"Thoughts on Machine Learning and Natural Language Processing\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.marekrei.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.marekrei.com\/blog\/#\/schema\/person\/a145eb0a06ed4acf5b0f84a24b7a1191\",\"name\":\"Marek\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.marekrei.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/48a65414bfda6485aaa0703e548de0ed25292b5fe0d979ed8c28ad83cf5a82c0?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/48a65414bfda6485aaa0703e548de0ed25292b5fe0d979ed8c28ad83cf5a82c0?s=96&d=mm&r=g\",\"caption\":\"Marek\"},\"url\":\"https:\/\/www.marekrei.com\/blog\/author\/marek\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Multilingual Semantic Models - Marek Rei","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/","og_locale":"en_US","og_type":"article","og_title":"Multilingual Semantic Models - Marek Rei","og_description":"In this post I’ll discuss a model for learning word embeddings, such\u00a0that they end up\u00a0in the same space in different languages. This means we can…","og_url":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/","og_site_name":"Marek Rei","article_published_time":"2014-10-18T12:13:28+00:00","article_modified_time":"2019-09-27T23:37:15+00:00","og_image":[{"url":"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png"}],"author":"Marek","twitter_misc":{"Written by":"Marek","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/","url":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/","name":"Multilingual Semantic Models - Marek Rei","isPartOf":{"@id":"https:\/\/www.marekrei.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#primaryimage"},"image":{"@id":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#primaryimage"},"thumbnailUrl":"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png","datePublished":"2014-10-18T12:13:28+00:00","dateModified":"2019-09-27T23:37:15+00:00","author":{"@id":"https:\/\/www.marekrei.com\/blog\/#\/schema\/person\/a145eb0a06ed4acf5b0f84a24b7a1191"},"breadcrumb":{"@id":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#primaryimage","url":"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png","contentUrl":"http:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2014\/10\/vector_space_model_multilingual-300x300.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.marekrei.com\/blog\/multilingual-semantic-models\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.marekrei.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Multilingual Semantic Models"}]},{"@type":"WebSite","@id":"https:\/\/www.marekrei.com\/blog\/#website","url":"https:\/\/www.marekrei.com\/blog\/","name":"Marek Rei","description":"Thoughts on Machine Learning and Natural Language Processing","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.marekrei.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.marekrei.com\/blog\/#\/schema\/person\/a145eb0a06ed4acf5b0f84a24b7a1191","name":"Marek","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.marekrei.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/48a65414bfda6485aaa0703e548de0ed25292b5fe0d979ed8c28ad83cf5a82c0?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/48a65414bfda6485aaa0703e548de0ed25292b5fe0d979ed8c28ad83cf5a82c0?s=96&d=mm&r=g","caption":"Marek"},"url":"https:\/\/www.marekrei.com\/blog\/author\/marek\/"}]}},"_links":{"self":[{"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/posts\/214","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/comments?post=214"}],"version-history":[{"count":35,"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/posts\/214\/revisions"}],"predecessor-version":[{"id":1306,"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/posts\/214\/revisions\/1306"}],"wp:attachment":[{"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/media?parent=214"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/categories?post=214"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.marekrei.com\/blog\/wp-json\/wp\/v2\/tags?post=214"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}