Update<\/strong>: the Deep Learning Summer School videos are now online<\/a>.<\/p>\n Alright, let’s get started.<\/p>\n During his first talk, Yoshua Bengio said “This is my most important slide”. You can see that slide below: Let’s say you have a classifier that needs to detect people that are male\/female, have glasses or\u00a0don’t have glasses, and are tall\/short. With non-distributed representations, you are dealing with 2*2*2=8 different classes of people. In order to train an accurate classifier, you need to have enough training data for each of these 8 classes. However, with distributed representations, each of these properties could be captured by a different dimension. This means that even if your classifier has never encountered tall men with glasses, it would be able to detect them, because it has learned to detect gender, glasses and height independently from all the other examples. The team of Yoshua Bengio have experimentally found that when optimising the parameters of high-dimensional neural nets, there effectively are no local minima. Instead, there are saddle points which are local minima in some dimensions but not all. This means that training can slow down quite a lot in these points, until the network figures out how to escape, but as long as we’re willing to wait long enough then it will find a way.<\/p>\n Below is a graph demonstrating a network during training, oscillating between two states: approaching a saddle point and then escaping it.<\/p>\n Given one specific dimension, there is some small probability \\(p\\) with which a point is a local minimum, but not a global minimum, in that dimension. Now, the probability of a point in a 1000-dimensional space being an incorrect local minimum in all<\/strong> of these would be \\(p^{1000}\\), which is just astronomically small. However, the probability of it being a local minimum in some<\/strong> of these dimensions is actually quite high. And when we get these minima in many dimensions at once, then training can appear to be stuck until it finds the right direction.<\/p>\n In addition, this probability \\(p\\) will increase as the loss function\u00a0gets closer to the global minimum. \u00a0This means that if we do ever end up at a genuine local minimum, then for all intents and purposes it will be close enough to the global minimum that it will not matter.<\/p>\n Leon Bottou had some useful tables with activation functions, loss functions, and their corresponding derivatives. I’ll keep these here for later.<\/p>\n <\/p>\n Update<\/strong>: As pointed out by commenters, the min and max functions in the ramp formula should be switched.<\/p>\n The current recommended strategy for initialising weights in a neural network is to sample values \\(W_{i,j}^{(k)}\\)\u00a0uniformly from \\([-b,b]\\), where<\/p>\n \\(b = \\sqrt{\\frac{6}{H_k + H_{k+1}}}\\)<\/p>\n \\(H_k\\) and \\(H_{k+1}\\) are the sizes of hidden layers before and after the weight matrix.<\/p>\n Recommended by\u00a0Hugo Larochelle, published by\u00a0Glorot & Bengio\u00a0(2010).<\/p>\n A few practical suggestions from\u00a0Hugo Larochelle:<\/p>\n If you implemented your backprop by hand and it’s not working, then there’s roughly 99% chance\u00a0that the gradient\u00a0calculation has\u00a0a bug. Use gradient checking to identify the issue. The idea is to use the definition of a gradient: how much will the model error change, if we increase a specific weight by a small amount.<\/p>\n \\(\\frac{\\partial f(x)}{\\partial x} \\approx\u00a0\\frac{f(x+\\epsilon) – f(x-\\epsilon)}{2\\epsilon}\\)<\/p>\n A more in-depth explanation is available here: Gradient checking and advanced optimization<\/a><\/p>\n Human motion tracking can be done with impressive accuracy. Below are examples from the paper\u00a0Dynamical Binary Latent Variable Models for 3D Human Pose Tracking<\/a> by Graham Taylor et al. (2010). The method uses\u00a0conditional restricted\u00a0Boltzmann machines.<\/p>\n Chris Manning and\u00a0Richard Socher have put a lot of effort into developing compositional models that combine neural embeddings with more traditional parsing\u00a0approaches. This culminated with a\u00a0Recursive Neural Tensor Network<\/a>\u00a0(Socher et al., 2013), which uses both additive and multiplicative interactions to combine word meanings along a parse tree.<\/p>\n And then, the model was beaten (by quite a margin) by the\u00a0Paragraph Vector<\/a> (Le & Mikolov, 2014), which knows absolutely nothing about the sentence structure or syntax. Chris Manning referred to this result as “a defeat for creating ‘good’ compositional vectors”.<\/p>\n However, more recent work using parse trees has again surpassed this result.\u00a0Irsoy & Cardie (NIPS, 2014)<\/a> managed to beat paragraph vectors by going “deep” with their networks in multiple dimensions. Finally, Tai et al. (ACL, 2015)<\/a> have improved the results again by combining LSTMs with parse trees.<\/p>\n The accuracies\u00a0of these models on the Stanford 5-class sentiment dataset are as follows:<\/p>\n [table width=”500″ colwidth=”100|50″ colalign=”left|center”] So it seems that, at the moment, models using the parse tree are beating simpler approaches. I’m curious to see if and when the next syntax-free approach\u00a0will emerge that will advance this race. After all, the goal of many neural models is not to discard the underlying grammar, but to implicitly capture it in the same network.<\/p>\n Chris Manning himself cleared up the confusion between the two words.<\/p>\n Distributed<\/strong>: A concept is represented as continuous activation levels in a number of elements. Like a dense word embedding, as opposed to 1-hot vectors.<\/p>\n Distributional<\/strong>:\u00a0Meaning is represented by contexts of use. Word2vec is distributional, but so are count-based word vectors, as we use the contexts of the word to model the meaning.<\/p>\n Comparison of dependency parsers on the Penn Treebank:<\/p>\n [table colalign=”left|center|center”] The last result is from Google “pulling out all the stops”, by\u00a0putting massive amounts of resources into training the Stanford neural parser.<\/p>\n Well, I knew a bit about Theano<\/a> before, but I learned a whole lot more during the summer school. And it is pretty awesome.<\/p>\n Since\u00a0Theano\u00a0originates from\u00a0Montreal, it was especially helpful to be able to ask questions directly from the people who are developing it.<\/p>\n Most of the information that was presented is available online, in the form of interactive python tutorials<\/a>.<\/p>\n Nvidia has a toolkit called Digits<\/a> that trains and visualises complex neural network models without needing to write any code.\u00a0And they’re selling DevBox<\/a> – a machine customised for running Digits and other deep learning software (Theano, Caffe, etc). It comes with 4 Titan X GPUs and currently costs $15,000.<\/p>\n Fuel<\/a> is a toolkit that manages iteration over your datasets – it can split them into minibatches, manage shuffling, apply various preprocessing steps, etc. There are prebuilt functions for some established datasets, such as MNIST, CIFAR-10, and Google’s 1B Word corpus.\u00a0It is mainly designed for use with Blocks<\/a>, a toolkit that simplifies network construction with Theano.<\/p>\n Remember “king – man + woman = queen”? Turns out that works with images as well (Kiros et al., 2015).<\/p>\n When we are at point \\(x_0\\) and take a step to \\(x\\), then we can estimate the function value in the new location by knowing the derivatives, using the Taylor series approximation.<\/p>\n \\( Similarly, we can estimate\u00a0the loss of a function, when we update parameters \\(\\theta_0\\) to \\(\\theta\\).<\/p>\n \\( where \\(g\\) contains the derivatives with respect to \\(\\theta\\), and \\(H\\) is the Hessian with second order derivatives\u00a0with respect to \\(\\theta\\).<\/p>\n This is the second-order Taylor approximation, but we could increase the accuracy by adding even higher-order derivatives.<\/p>\n Adam Coates presented a strategy for analysing the speed of\u00a0matrix operations on a GPU. It’s a simplified model that says your time is spent on either reading\/writing to memory or doing calculations. It assumes you can do both in parallel so\u00a0we are interested in which one of them takes more time.<\/p>\n Let’s say we are multiplying a matrix with a vector:<\/p>\n If \\(M=1024\\) and \\(N=512\\), then the number of bytes we need to read and store is:<\/p>\n \\( 4\\text{ bytes }\\times (1024\u00a0\\times\u00a0512 + 512 + 1024) = 2.1e6\\text{ bytes} \\)<\/p>\n And the number of calculations\u00a0we need to do is:<\/p>\n \\(2\\times 1024\\times 512 = 1e6\\text{ FLOPs}\\)<\/p>\n If we have a GPU that can do\u00a06 TFLOP\/s and has memory bandwidth of\u00a0 300GB\/s, then the\u00a0total running time will be:<\/p>\n \\(\\text{max}\\{2.1e6\\text{ bytes }\/ (300e9\\text{ bytes}\/s), 1e6\\text{ FLOPs} \/ (6e12\\text{ FLOP}\/s) \\} \\\\ This means the process is bounded by the \\(7\\mu s\\) spent on copying to\/from the\u00a0memory, and\u00a0getting a faster GPU would not make any difference. As you can probably guess, this situation gets better with bigger matrices\/vectors, and when doing matrix-matrix operations.<\/p>\n Adam also described the idea of calculating the intensity of an operation:<\/p>\n Intensity = (# arithmetic ops) \/ (# bytes to load or store)<\/p>\n In the previous scenario, this would be<\/p>\n Intensity = (1E6 FLOPs) \/ (2.1E6 bytes) = 0.5 FLOPs\/bytes<\/p>\n Low intensity means the system is bottlenecked on memory, and high intensity means it’s bottlenecked by the GPU speed. This can be visualised, in order to find which of the two needs to improve in order to speed up the whole system, and where the sweet spot lies.<\/p>\n Continuing from the intensity calculations, one way of increasing the intensity of your network\u00a0(in order to be limited by computation instead of memory), is to process data in minibatches. This avoids some memory operations, and GPUs are great at processing large matrices in parallel.<\/p>\n However, increasing the batch size too much will probably start to hurting the training algorithm and converging can take longer. It’s important to find a good balance in order to get the best results in the least amount of time.<\/p>\n It was recently revealed that\u00a0neural networks are easily tricked by adversarial examples. In the example below, the image on the left is correctly classified as a goldfish. However, if we apply the noise pattern shown in the middle, resulting in the image on the right, the classifier becomes convinced this is a picture of a daisy. The image is from\u00a0Andrej Karpathy’s blog post\u00a0“Breaking Linear Classifiers on ImageNet”<\/a>, and you can read more about it there.\u00a0 The noise pattern isn’t random though – the noise is carefully calculated, in order to trick the network. But the point remains: the image on the right is clearly still a goldfish and not a daisy.<\/p>\n Apparently strategies like ensemble models, voting after multiple saccades, and unsupervised pretraining have all failed against this vulnerability. Applying heavy regularisation helps, but\u00a0not before ruining the accuracy on the clean data.<\/p>\n Ian Goodfellow presented the idea of training on these adversarial examples.\u00a0They can be automatically generated and added to the training set. The results below show that in addition to helping with the adversarial cases, this also improves accuracy on the clean examples. Phil Blunsom presented the idea that almost all NLP can be structured as a language model. We can do this by concatenating the output to the input and trying to predict the probability of the whole sequence.<\/p>\n Translation:<\/p>\n \\(P(\\text{Les chiens aiment les os || Dogs love bones})\\)<\/p>\n Question answering:<\/p>\n \\(P(\\text{What do dogs love? || bones .})\\)<\/p>\n Dialogue:<\/p>\n \\(P(\\text{How are you? || Fine thanks. And you?})\\)<\/p>\n The latter two need to be additionally conditioned on some world knowledge. The second part doesn’t even need to be words, but could be labels or some structured output like dependency relations.<\/p>\n When Frederick Jelinek and his team at IBM submitted one of the first papers on statistical machine translation to COLING in 1988, they got the following anonymous review:<\/p>\n The validity of a statistical (information theoretic) approach to MT has indeed been\u00a0<\/em>recognized, as the authors mention, by Weaver as early as 1949. And was universally\u00a0<\/em>recognized as mistaken by 1950 (cf. Hutchins, MT \u2013 Past, Present, Future, Ellis\u00a0<\/em>Horwood, 1986, p. 30ff and references therein). The crude force of computers is not\u00a0<\/em>science. The paper is simply beyond the scope of COLING.<\/em><\/p>\n Apparently a very simple neural model can produce surprisingly good results. An example of translating from Chinese to English, from Phil Blunsom’s slides:<\/p>\n In this model, the vectors for the Chinese words are simply added together to form a sentence vector. The decoder\u00a0consists of a conditional language model which takes the sentence vector, together with vectors from the two recently generated English words, and generates the next word in the translation.<\/p>\n However, neural models are still not outperforming the very best traditional MT systems. They do come very close though. Results from “Sequence to Sequence Learning<\/a>\u00a0with Neural Networks”<\/a>\u00a0by Sutskever et al. (2014):<\/p>\n [table] Update:<\/strong> @stanfordnlp<\/a> pointed out that there are some recent\u00a0results where the neural model does indeed outperform the state-of-the-art traditional MT system. Check out “Effective Approaches to Attention-based Neural Machine Translation<\/a>” (Luong et. al., 2015).<\/p>\n Richard Socher demonstrated the MetaMind\u00a0image classification demo<\/a>, which you can train yourself by uploading images. I trained a classifier to detect Edison and Einstein (couldn’t find enough unique images of Tesla). 5 example images for both classes, testing on one held out image each. Seemed to work pretty well.<\/p>\n Mark Schmidt gave two presentations about numerical optimisation in different scenarios.<\/p>\n In a deterministic<\/strong> gradient method we calculate the gradient over the whole dataset and then apply the update. The iteration cost is linear with the dataset size.<\/p>\n In\u00a0stochastic<\/strong> gradient methods we calculate the gradient on one datapoint and then apply the update. The iteration cost is independent of the dataset size.<\/p>\n Each iteration of the stochastic gradient descent is much faster, but it usually takes many more iterations to train the network, as this graph illustrates:<\/p>\n In order to get the best of both worlds, we can use batching. More specifically, we could do one pass of the dataset with stochastic gradient descent, in order to quickly get on the right track, and then start increasing the batch size. The gradient error decreases as the batch size increases, although eventually the iteration cost will become dependent on the dataset size again.<\/p>\n Stochastic Average Gradient (SAG) is a method that gets around this, providing a linear convergence rate with only 1 gradient per iteration. Unfortunately, it is not feasible for large neural networks, as it needs to remember the gradient updates\u00a0for every datapoint, leading to large memory requirements.\u00a0Stochastic Variance-Reduced Gradient (SVRG) is another method that reduces this memory cost, and\u00a0only needs 2 gradient calculations per iteration (plus occasional full passes).<\/p>\n Mark said a student of his implemented a variety of optimisation methods (AdaGrad, momentum, SAG, etc). When asked, what he would use in a black box neural network system, the student said two methods: Streaming SVRG<\/a> (Frostig et al., 2015), and a method they haven’t published yet.<\/p>\n If you put “profile=True” into THEANO_FLAGS, it will analyse your program, showing a breakdown of how much is spent on each operation. Very handy for finding bottlenecks.<\/p>\n Following on from Ian Goodfellow’s talk on adversarial examples, Yoshua Bengio talked about having two systems competing\u00a0with each other.<\/p>\n System D is a discriminative system that aims to classify between real data and artificially generated data.<\/p>\n System G is a generative system, that tries to generate artificial data, which D would incorrectly classify as real.<\/p>\n As we train one, the other needs to get better as well. In practice this does work, although\u00a0the step needs to be quite small to make sure D can keep up with G. Below are some examples from “Deep Generative Image Models using a\u00a0Laplacian Pyramid of Adversarial Networks<\/a>” – a more advanced version of this model which tries to generate images of churches.<\/p>\n1. The need for distributed representations<\/h2>\n
![\"dlss-3aug2015\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/dlss-3aug2015.png\")
<\/a><\/p>\n
\n<\/p>\n2. Local minima are not a problem in high dimensions<\/h2>\n
![\"dlss-3aug20152\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/dlss-3aug20152.png\")
<\/a><\/p>\n3. Derivatives derivatives derivatives<\/h2>\n
![\"bottou1\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/bottou1.png\")
<\/a><\/p>\n![\"bottou2\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/bottou2.png\")
<\/a><\/p>\n4. Weight initialisation strategy<\/h2>\n
5. Neural net training tricks<\/h2>\n
\n
6. Gradient checking<\/h2>\n
7. Motion tracking<\/h2>\n
![\"gwtaylor_cvpr2010\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/gwtaylor_cvpr2010.png\")
<\/a><\/p>\n8. Syntax or no syntax? (aka, “is syntax a thing?”)<\/h2>\n
\nMethod, Accuracy
\nRNTN (Socher et al. 2013), 45.7
\nParagraph Vector (Le & Mikolov 2014),\u00a048.7
\nDRNN (Irsoy & Cardie 2014), 49.8
\nTree LSTM (Tai et al. 2015), 50.9
\n[\/table]<\/p>\n9. Distributed vs distributional<\/h2>\n
10. The state of dependency parsing<\/h2>\n
\nParser,\u00a0Unlabelled Accuracy, Labelled Acccuracy,\u00a0Speed (sent\/s)
\nMaltParser, 89.8, 87.2, 469
\nMSTParser, 91.4, 88.1, 10
\nTurboParser, 92.3, 89.6, 8
\nStanford Neural Dependency Parser<\/a>, 92.0, 89.7, 654
\nGoogle,\u00a094.3, 92.4, ?
\n[\/table]<\/p>\n11. Theano<\/h2>\n
![\"\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/14439811.png\")
<\/a><\/p>\n12. Nvidia Digits<\/h2>\n
13. Fuel<\/h2>\n
14. Multimodal linguistic regularities<\/h2>\n
![\"talk_Montreal_part2_pdf\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/talk_Montreal_part2_pdf.png\")
<\/a><\/p>\n15.\u00a0Taylor series approximation<\/h2>\n
\nf(x) = f(x_0) + (x – x_0)f'(x) + \\frac{1}{2}\u00a0(x – x_0)^2 f”(x) + …
\n\\)<\/p>\n
\nJ(\\theta)\u00a0=J(\\theta_0) + (\\theta – \\theta_0)^T g + \\frac{1}{2}\u00a0(\\theta – \\theta_0)^T\u00a0H(\\theta – \\theta_0) + …
\n\\)<\/p>\n16. Computational intensity<\/h2>\n
![\"dlss_systems_1\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/dlss_systems_1-300x209.png\")
<\/a><\/p>\n
\n= \\text{max}\\{ 7\\mu\u00a0s, 0.16\\mu s \\} \\)<\/p>\n![\"dlss_systems_2\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/dlss_systems_2.png\")
<\/a><\/p>\n17. Minibatches<\/h2>\n
![\"dlss_systems_3\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/dlss_systems_3.png\")
<\/a><\/p>\n18. Training on adversarial examples<\/h2>\n
![\"fish\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/fish.jpeg\")
<\/a><\/p>\n![\"goodfellow_adv\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/goodfellow_adv.png\")
<\/a>Finally, we can improve this further by penalising the KL-divergence between the original predicted distribution and the predicted distribution on the adversarial example. This optimises the network to be more robust, and to predict similar class distributions for similar (adversarial) images.<\/p>\n19. Everything is language modelling<\/h2>\n
20. SMT had a rough start<\/h2>\n
![\"blunsom-lm-mt2\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/blunsom-lm-mt2-270x300.png\")
<\/a><\/p>\n21. The state of Neural Machine Translation<\/h2>\n
![\"blunsom-lm-mt\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/blunsom-lm-mt.png\")
<\/a><\/p>\n
\nModel,\u00a0BLEU score
\nBaseline, 33.30
\nBest WMT’14 result,\u00a037.0
\nScoring with\u00a05 LSTMs,\u00a036.5
\nOracle (upper bound),\u00a0\u223c45
\n[\/table]<\/p>\n22. MetaMind classifier demo<\/h2>\n
![\"edison_vs_einstein\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/edison_vs_einstein.png\")
<\/a><\/p>\n23. Optimising gradient updates<\/h2>\n
![\"2015_DLSS_ConvexOptimization\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/2015_DLSS_ConvexOptimization-300x233.png\")
<\/a><\/p>\n24. Theano profiling<\/h2>\n
25.\u00a0Adversarial nets framework<\/h2>\n
![\"denton_generating_images\"](\"https:\/\/www.marekrei.com\/blog\/wp-content\/uploads\/2015\/08\/denton_generating_images.png\")
<\/a><\/p>\n