{"id":180,"date":"2014-10-05T21:13:52","date_gmt":"2014-10-05T21:13:52","guid":{"rendered":"http:\/\/www.marekrei.com\/blog\/?p=180"},"modified":"2019-09-27T23:37:58","modified_gmt":"2019-09-27T23:37:58","slug":"political-ideology-detection","status":"publish","type":"post","link":"https:\/\/www.marekrei.com\/blog\/political-ideology-detection\/","title":{"rendered":"Political ideology detection"},"content":{"rendered":"

Neural networks have a range of interesting applications, and here I will discuss\u00a0on one them: recursive neural networks and the detection of political ideology. This post is a summary and analysis of a recent publication by Mohit Iyyer, Peter Anns, Jordan Boyd-Graber and Philip Resnik: “Political Ideology Detection Using Recursive Neural Networks<\/a>“.<\/p>\n

The Task<\/h2>\n

Given a sentence, we want the model\u00a0to detect the political ideology expressed in that sentence. In this research, the authors deal with US politics, so the possible options are liberal (democrats) or conservative (republicans). As a practical application we might consider a system that processes a large amount of news articles or public speeches to detect and measure\u00a0explicit\u00a0or hidden political bias of the authors.<\/p>\n

\"democrat-republican\"<\/a><\/p>\n

A traditional approach to this problem is a simple bag-of-words\u00a0model, where each word is treated as a separate feature, but this ignores any syntactic structure and even word order. As shown below, political ideology can be\u00a0compositionally complicated – while certain sections of the sentence are locally conservative, the way they are used in context makes the overall sentence liberal.\"political_sample_1\"<\/a><\/p>\n

Figure 1: Sample sentence from Iyyer et al. (2014). Blue nodes are liberal, red nodes are conservative, grey nodes are neutral.
\n<\/p>\n

Recursive Neural Network<\/h2>\n

The recursive neural network in this work is based on Socher et al. (2011), where it was used\u00a0for sentiment detection. The idea of semantic composition is that the meaning of a sentence or text is composed of its smaller subparts, and recursive neural networks aim\u00a0to model this by recursively combining vector representations of words and phrases.<\/p>\n

We represent each word as a vector of length d<\/em>. We can then construct a network that takes two\u00a0word vectors, concatenated into a vector of length 2d<\/em>, as input and outputs a new vector of length d<\/em>. We can then stack these networks together and use the output of one network as one of the inputs to another network. This way we can combine multiple words to phrases and phrases to sentences. We don’t need to have a separate network for each of these levels, and can just reuse the same network each time\u00a0–\u00a0which is why these are called recursive neural networks<\/em>.<\/p>\n

Mathematically, the composition happens as follows:<\/p>\n

\\(
\nx_c = f(W_L\\times x_a + W_R\\times x_b + b)
\n\\)<\/p>\n

\\(x_a\\) and\u00a0\\(x_b\\) are the vectors for the two input words\/phrases;\u00a0\\(x_c\\) is the output vector;\u00a0\\(W_L\\) and\u00a0\\(W_R\\) are the weight matrices for the left and right input;\u00a0\\(b\\) is the bias vector. If we think of the input as a concatenation of the vectors, this formula actually becomes the traditional single-layer neural network:<\/p>\n

\\(
\nx_c = f(W\\times x_{ab} + b)
\n\\)<\/p>\n

An example of this recurrent neural network in action is shown below:<\/p>\n

\"political_tree\"<\/a><\/p>\n

During training, an additional\u00a0softmax layer is added on top, which calculates the probability for both classes (liberal or conservative). The model is then trained to minimize the negative log-probability of the correct class, with L2-regularization.\u00a0\u00a0The error is backpropagated through the whole tree, so the information on the sentence level will be used to modify the individual word vectors, along with all the weights used in the composition.<\/p>\n

Also, if I understand the paper correctly, the final prediction from the model at test time is actually generated by averaging vectors for all the nodes in the sentence tree, concatenating this\u00a0with the root vector, and training a separate logistic regression model over that. Which seems a bit odd, because the recurrent network is quite capable of predicting the class label on the root node, and has been optimised to do so. I wonder what the performance difference would be, if the root-level predictions from the RNN were used directly for evaluation.<\/p>\n

Evaluation<\/h2>\n

The authors use two datasets for evaluation:<\/p>\n

    \n
  1. The Convote<\/strong> dataset (Thomas et al., 2006) contains transcripts of spoken text from the US Congressional floor debate. The experiments here use 7,816 sentences from the corpus.<\/li>\n
  2. The Ideological Books Corpus<\/strong> (IBC) (Gross et al., 2013) is a collection of books and articles by authors whose political bias\u00a0is well-known.\u00a0Iyyer et al. (2014)\u00a0extend the dataset by providing sentence-level and partial phrase-level annotation for 4,062\u00a0sentences<\/a>, crowd-sourced through Crowdflower<\/a>.<\/li>\n<\/ol>\n

    Both of these datasets are originally annotated on the level of the author and their political views, but this information can be used to also label each sentence with its most likely political class.<\/p>\n

    On both datasets, the authors apply heuristic filters to only keep sentences containing explicit bias in either direction,\u00a0and remove neutral sentences which take up majority of the corpora. However, as this filtering is based on\u00a0surface forms (essentially bag-of-words or bag-of phrases), it is unknown how this affects the final dataset. It would be interesting to also see results where the test set contains all of the original sentences.<\/p>\n

    Results<\/h2>\n

    The authors compare a number of different models.<\/p>\n