Jekyll2020-04-20T12:17:01+00:00https://jontysinai.github.io/feed.xmlJonty SinaiMathematician. Yogi. Runner. Dreamer.Jonty SinaiUnderstanding Bayesian Inference2020-04-19T00:00:00+00:002020-04-19T00:00:00+00:00https://jontysinai.github.io/jekyll/update/2020/04/19/understanding-bayesian-inference<p><em>What do we mean when we say “Bayesian inference”? More specifically, what does Bayesian inference mean for my machine learning or data modelling problem? In this blogpost I will introduce Bayesian inference and explain how it is a machine learning paradigm. More importantly I will attempt to intuitively bridge the gap between Bayesian inference as a theoretical framework and Bayesian inference as a machine learning approach.</em></p>
<h1 id="the-real-world-data-and-modelling">The Real World: Data and Modelling</h1>
<p>Let’s begin with the problem which we’re trying to solve in the first place: we have some acquired data and we would like to predict future instances. Using the same starting point as <a href="https://jontysinai.github.io/jekyll/update/2019/01/18/understanding-neural-odes.html">my post on neural ODE’s</a>, we can formalise this setting as follows:</p>
<ol>
<li>We have \(N\) observations of \(x, y\): \(\mathscr{D} = \big\{(x_1, y_1), (x_2, y_2), …, (x_N, y_N)\big\}\). \(\mathscr{X}\) is the input domain and \(\mathscr{Y}\) is the output domain.</li>
<li>Given a new input, \(x^{*}\), we would like to predict the unknown output: \(y^{*}\).</li>
<li>We assume that there is some function which maps the input and output domains: \(f:\mathscr{X} \to \mathscr{Y}\).</li>
</ol>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2019-01-18-understanding-neural-odes/function_x_to_y.png" /></p>
<blockquote>
<p>We model the problem as trying to find a function mapping from \(\mathscr{X}\) to \(\mathscr{Y}\).</p>
</blockquote>
<p>The machine learning approach is to choose a flexible class of functions, typically <em>neural networks</em>, which are described by parameters \(\theta\). We use a <em>learning algorithm</em>, such as stochastic gradient descent, to optimize the parameters \(\theta\) such that the function \(f\) is a good fit for the data. What we mean by a “good fit” is codified by a <em>cost</em> function which is minimized during the learning procedure. Traditionally this cost function measures the degree to which the data fits the model well. If not, then adjust the parameters of the model and measure again until the cost is sufficiently low.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/function_x_to_y_with_theta.png" /></p>
<p>We then make new predictions using the optimal parameter \(\theta^{*}\). One shortcoming of this approach is that once we have found the optimal \(\theta^{*}\), the prediction process is completely deterministic. This is a problem if we require uncertainty in our predictions:</p>
<ul>
<li>How certain are we that patient \(\mathscr{x}\) will have outcome \(\mathscr{y}\)?</li>
<li>How certain are we that image \(\mathscr{x}\) can be classified as \(\mathscr{y}\)?</li>
<li>How certain are we that measurement \(\mathscr{x}\) will yield signal \(\mathscr{y}\)?</li>
</ul>
<p>In general uncertainty is desirable if the variance in the data is large. The learning algorithm finds \(\theta^{*}\) which works for most of the data points which we’ve seen so far. How reliable will \(\theta^{*}\) be, when \(\mathscr{x}\) is an outlier - when it is not like most datapoints?</p>
<p>Uncertainty is also desirable when we don’t have a large number of samples. In this case, if the variance is large between each sample, then we may not have enough evidence to support any one particular value of \(\theta\). Instead we would like a way of making predictions which accounts for our uncertainty on the value of the parameter \(\theta\) itself.</p>
<blockquote>
<p>This is precisely what we will try to achieve with Bayesian inference.</p>
</blockquote>
<h1 id="an-old-favourite-bayes-theorem">An old favourite: Bayes’ Theorem</h1>
<p>As the name implies, Bayesian inference has a lot to do with <em>Bayes’ Theorem</em>. In fact, Bayes’ Theorem will form the bridge between the uncertainty in our data and the uncertainty in our predictions. You may already be familiar with Bayes’ Theorem (which I also covered in my post on <a href="https://jontysinai.github.io/jekyll/update/2018/12/23/probability-part-two-conditional-probability.html">conditional probability</a>), but as a recap here is what we mean by Bayes’ Theorem:</p>
<blockquote>
<p><em>Let \(A\) and \(B\) be random variables\(^{*}\) with probability measure \(\mathbb{P}\), such that \(\mathbb{P}(A) \neq 0\), then</em>
\[
\mathbb{P}(B|A) = \frac{ \mathbb{P}(A | B)\mathbb{P}(B) }{ \mathbb{P}(A) }.
\]</p>
</blockquote>
<blockquote>
<ul>
<li>\(\mathbb{P}(B)\) is known as the <strong>prior probability</strong> of \(B\) - it is what we know (assume) about \(B\) before we consider the effects \(A\).</li>
<li>\(\mathbb{P}(A)\) is known as the <strong>evidence</strong> of \(A\) - it is everything we know about \(A\).</li>
<li>\(\mathbb{P}(A | B)\) is known as the <strong>likelihood</strong> of \(A\), given \(B\) - it is an estimate of how likely the \(A\) that we observed is, given possible values of \(B\).</li>
<li>\(\mathbb{P}(B | A)\) is known as the <strong>posterior probability</strong> of \(B\), given \(A\) - it is the <em>conditional probability</em> of \(B\) <em>after</em> we have obtained evidence of \(A\).</li>
</ul>
</blockquote>
<p>Bayes’ Theorem is useful when it is difficult to calculate the posterior of \(B\) directly. We use Bayes’ Theorem when we have obtained data on \(A\) and we can measure the likelihood and prior with greater ease.</p>
<p>* <small> for more detail on random variables and probability measures, see my <a href="https://jontysinai.github.io/jekyll/update/2017/11/23/probability-for-everyone.html">introductory post on probability</a>.
</small></p>
<h1 id="bayes-theorem-meets-the-real-world">Bayes’ Theorem Meets the Real World</h1>
<p>Let’s go back to the fundamental modelling problem. We have a parametric function, described by \(\theta\), and we want to choose the right \(\theta\) so that our function describes the data well. We also want to compute our uncertainty over \(\theta\) so that we can make predictions with confidence, and know when this uncertainty is high.</p>
<blockquote>
<p>But what does it mean to include uncertainty into our modelling of \(f\) and \(\theta\)?</p>
</blockquote>
<p>This means precisely to use probabilities - the mathematical language of uncertainty - and in particular it means to use <em>probability distributions</em>. Instead of optimizing for a single value of \(\theta\), <em>we can treat \(\theta\) as a <strong>random variable</strong> which has <strong>highest probability</strong> at the optimal value</em>.</p>
<blockquote>
<p>But how do we ensure that the probability distribution over \(\theta\) has highest probability at the optimal value?</p>
</blockquote>
<p>We do this by using Bayes’ Theorem. In particular we can start with a best guess at what this distribution may be and call it our <strong>prior distribution</strong></p>
<p>\[
P(\; \theta \;)^{*}.
\]</p>
<blockquote>
<p>How do we know what a good prior distribution is?</p>
</blockquote>
<p>This is a difficult question but in <a href="https://en.wikipedia.org/wiki/Bernstein–von_Mises_theorem">theory</a> our choice on the prior doesn’t matter if are able to obtain enough data. In practice the choice of prior will affect the posterior and it helps to encode (and test!) our assumptions on \(\theta\). For example use a discrete probability distribution if \(\theta\) is discrete, or a positive distribution if \(\theta\) must be non-negative.</p>
<p>What’s important in Bayesian inference is that whatever choice of prior we choose, the additional data which we obtain will alter the shape of prior to inform our target posterior distribution. Going back to Bayes’ Theorem we can incorporate the data \(\mathscr{D}\) into the formula using the <strong>likelihood</strong></p>
<p>\[
P(\; \mathscr{D} \;|\; \theta \;).
\]</p>
<p>It is through the likelihood that we encode the optimality of \(\theta\) into the posterior, which is defined by the degree to which \(\theta\) is a good fit for the data. Strictly speaking the likelihood is a function of \(\theta\) - we assume the data has been observed and is fixed. With this point of view the likelihood can be interpreted as the <em>likelihood of the observed data occurring, given the possible values of \(\theta\)</em>. A higher likelihood indicates a better value of \(\theta\).</p>
<blockquote>
<ul>
<li>In fact many machine learning loss functions are derived from the likelihood, usually in the form of the negative log-likelihood (applying the log transformation to the likelihood helps for numerical stability purposes, especially over a large dataset).</li>
<li>The likelihood function does not need to strictly be a probability distribution.</li>
</ul>
</blockquote>
<p>Now we can use Bayes’ Theorem to write down a formula for the <strong>posterior distribution</strong> of \(\theta\):</p>
<p>\[
P(\; \theta \;|\; \mathscr{D} \;) = \frac{ P(\; \mathscr{D} \;|\; \theta \;)P(\; \theta \;) }{ P(\; \mathscr{D} \;)}.
\]</p>
<p>The posterior distribution takes into account our uncertainty over \(\theta\) (incorporated in the prior distribution), adjusted by the actual data that we observed (incorporated in the likelihood, and also the evidence of the data). The combination of the likelihood with the prior makes the posterior distribution suitable for estimating optimal values of \(\theta\) with the added measure of uncertainty.</p>
<blockquote>
<p>How does this posterior distribution help us?</p>
</blockquote>
<p>Instead of calculating a single estimate of the optimal \(\theta\) we look for a (posterior) distribution on \(\theta\). From this distribution we can calculate a sample mean for \(\theta\) and report our confidence over a credible interval - say the middle 95% of the distribution.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/posterior_over_theta.png" /></p>
<p>Going back to the original function estimation problem, this means that when we calculate predictions for \(\mathscr{y}\), we first estimate \(\theta\) from the posterior so that our predictions become conditional on \(\theta\):</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/posterior_estimation_with_theta.png" /></p>
<p>But it turns out that calculating this posterior can be <strong>hard</strong> with a capital <em>H</em>. For two reasons:</p>
<ol>
<li>The likelihood function is often highly nonlinear and must include every sample from the training dataset. In addition the complexity of the likelihood increases with the dimension of the data.</li>
<li>The evidence, \(P(\; \mathscr{D} \;)\), can be extremely hard to compute. It requires knowing how much probability to assign to every possible value which the data can take. In an uncertain world this is practically impossible.</li>
</ol>
<p>The algorithmic insight is to use what’s called <strong>approximate inference</strong>. Instead of calculating the posterior directly, we will try to approximate it.</p>
<p><small>*<strong>Note:</strong> that whereas before I used blackboard bold \(\mathbb{P}\) to denote the <em>probability measure</em> for individual probabilities, here I am using plain \(P\) to denote a <strong>probability distribution function</strong>. These are subtly different things conceptually and the change in notation is a pedantic way of expressing that. In short the <em>probability distribution function</em> is a function which describes all of the probabilities of a random variable over its domain.
</small></p>
<h1 id="approximate-inference">Approximate Inference</h1>
<p>Broadly speaking there are three main types of approximate inference algorithms, with the first two being the most popular. Briefly, they are:</p>
<ul>
<li>
<p><strong>Monte Carlo Sampling:</strong> this is a broad class of algorithms which approximates a target distribution by computing samples. In Bayesian inference we use a special type of Monte Carlo sampling algorithm known as <strong>Markov Chain Monte Carlo (MCMC)</strong> which iteratively computes samples using only information from the previous sample and the target distribution we are trying to approximate:
\[
\theta_{t} \leftarrow \theta_{t-1}, \ \theta_{t} \sim P(\; \theta \;|\; \mathscr{D} \;).
\]
With Monte Carlo algorithms, we do not try to explicitly compute the distribution, but rather try to compute candidate samples for \(\theta\) until eventually most samples are close to the optimal value of \(\theta\).</p>
</li>
<li>
<p><strong>Variational\(^{*}\) Inference:</strong> here we attempt to approximate the posterior distribution itself. We do this in much the same way that we use machine learning to approximate a function: choose a flexible function \(q\) parametrised by \(\phi\) and optimise \(\phi\) so that
\[
q(\theta) \approx P(\; \theta \;|\; \mathscr{D} \;).
\]</p>
</li>
<li>
<p><strong>Expectation Propagation:</strong> again we approximate the posterior with a tractable function \(q\), except we specifically choose this function to be factorisable into a product of distributions which are conditional on each other. We then use <em>message passing algorithms</em> to iteratively update the factors of \(q\) until it is a good fit for the posterior.</p>
</li>
</ul>
<p>While I won’t go into more detail into each of these algorithms - they are entire blogposts (and more!) unto themselves - I will try and unravel the basic fundamentals of approximate inference for the remainder of this blopost.</p>
<p><small>\(^{*}\)The name variational inference comes from <em>variational calculus</em> which is the calculus of finding functions by varying their parameters. The function \(q\) is sometimes known as the <em>guide</em>.
</small></p>
<h4 id="a-chicken-and-egg-problem">A Chicken and Egg problem</h4>
<p>You may have noticed that approximate inference has a chicken and egg problem. We want to approximate the posterior by drawing samples, but how can we draw samples from the posterior if we can’t compute it directly? Or we want to approximate the posterior with some parametrised guide distribution, but how do we know that it is a good match for the posterior without evaluating the posterior directly?</p>
<p>We get around this as follows:</p>
<ol>
<li>The denominator in Bayes’ Theorem is a <em>normalization constant</em>. If we ignore this constant (a constant function really), then the posterior distribution is <em>proportional</em> to the product of the likelihood and the prior (both of which are computable to us), which we write as follows:
\[
P(\; \theta \;|\; \mathscr{D} \;) \propto P(\; \mathscr{D} \;|\; \theta \;)P(\; \theta \;). <br />
\]</li>
<li>Furthermore in approximate inference we are more interested in the <em>argmax</em> of the posterior distribution - which \(\theta\) has maximum probability? - rather than the probability on \(\theta\) itself. Thus we can exploit the property of optimization where if \(g\) is our objective function and \(C\) is a constant then:
\[
argmax\{\; g \;\} = argmax\{\; \frac{1}{C}g \;\}.
\]
<blockquote>
<p>In the case of MCMC specifically, we use an algorithm which explicitly does not require knowing \(P(\; \mathscr{D} \;)\) in order to update samples of \(\theta\), doing so until we start sampling the \(\theta\) with maximum probability.</p>
</blockquote>
</li>
<li>Once we have the posterior up to a normalization constant, we simply need to normalize the posterior to turn it into a probability distribution.</li>
</ol>
<h2 id="unravelling-the-graph">Unravelling the graph</h2>
<p>We can express the cyclic nature of Bayesian inference graphically in the following way:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/bayesian_inference_graph_cyclic.png" /></p>
<blockquote>
<ul>
<li>Start with a prior distribution on \(\theta\), which we can use to sample \(\theta\).</li>
<li>What is the likelihood of \(\theta\), given the data?</li>
<li>We get the posterior probability of \(\theta\), which we can use to sample a better estimate of\(\theta\).</li>
<li>But we can still test the likelihood of this \(\theta\)…</li>
</ul>
</blockquote>
<p>Now we can see what we are trying to do with Bayesian inference a bit better. Start with a prior and likelihood (going forwards on the graph), and then go backwards to calculate a suitable posterior. When we can’t calculate this posterior exactly (feasible for only the simplest of models), then we resort to approximate inference. Approximate inference works by iteratively updating the posterior (forwards) and then checking to see if the samples for \(\theta\) are good (backwards). In general approximate inference algorithms work as follows:</p>
<p><em>Given a prior distribution and likelihood function:</em></p>
<ol>
<li>Choose an initial starting point for \(\theta\).</li>
<li>Use the likelihood function and prior to estimate a posterior (or posterior sample) from the current value of \(\theta\).</li>
<li>Suitably update the posterior based on how good (or bad) a fit we have so far.</li>
<li>Sample a new \(\theta\) value from the current posterior.</li>
<li>Repeat steps 2-4 until we get a suitable fit for the posterior.</li>
</ol>
<blockquote>
<p><strong>Note:</strong> For variational inference we instead update the parameter \(\phi\) to obtain a good functional approximation of the posterior distribution, and from this approximation we will sample \(\theta\).</p>
</blockquote>
<p>Going back to the graph, we are essentially unrolling the cyclic relationship between the posterior estimate and the sampling of \(\theta\) into a successive sequence of algorithm iterations:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="100%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/approximate_inference_graph.png" /></p>
<blockquote>
<p>Start with an initial value for \(\theta\). Successively update the posterior and then sample a new estimate of \(\theta\).</p>
</blockquote>
<p>The exact details of steps 2-4 and when we determine to terminate the algorithm will depend on the choice of algorithm used. Although these algorithms are beyond the scope of this introduction to approximate inference, I intend on covering them in future posts.</p>
<h1 id="tying-it-back-together">Tying it back together</h1>
<p>What we have seen so far is that when we want to express uncertainty over the modelling parameters in a machine learning problem, we can use Bayes’ Theorem to help us. Specifically</p>
<ol>
<li>When we have a dataset \(\mathscr{D} = \big\{(x_1, y_1), (x_2, y_2), …, (x_N, y_N)\big\}\),</li>
<li>And a model, \(f: \mathscr{X} \to \mathscr{Y}\), with parameters \(\theta\),</li>
<li>Then we can use Bayes’ Theorem to describe a <strong>posterior distribution</strong> over the modelling parameters \(\theta\), by specifying a <em>prior distribution</em> and a <em>likelihood function</em> so that:
\[
P(\; \theta \;|\; \mathscr{D} \;) \propto P(\; \mathscr{D} \;|\; \theta \;)P(\; \theta \;).
\]</li>
</ol>
<p>Once we have this posterior distribution, we can make a prediction \(y^{*}\) at a new point \(x^{*}\), using the so-called <em>predictive distribution</em> - the probability distribution over \(y^{*}\) given \(x^{*}\) <em>and</em> our <strong>uncertainty over the parameters \(\theta\)</strong>:</p>
<p>\[
P(\; y^{*} \;| x^{*}; \mathscr{D} \;) = \int_{\theta} P(\; y^{*} \;| x^{*}, \theta \;)P(\; \theta \;|\; \mathscr{D} \;)d\theta.
\]</p>
<p>In practice we approximate \(P(\; y^{*} \;|\; x^{*}, \theta \;)\) by the function \(f\) and we can use the posterior distribution to compute samples \(\big\{ \theta_1, \theta_2, … \theta_M \big\}\). Then we can calculate a <em>sample mean</em> for \(y^{*}\) using a <em>Monte Carlo</em> estimate:</p>
<p>\[
y^{*} \approx \; \widehat{y} \; = \; \frac{1}{M}\sum_{s=1}^{M}f(x^{*}; \theta_s).
\]</p>
<p>To measure the uncertainty of our prediction we can calculate a <em>sample variance</em>:</p>
<p>\[
Var(y^{*}) \approx \frac{1}{M}\sum_{s=1}^{M}\big( f(x^{*}; \theta_s) - \widehat{y} \big)^{2}.
\]</p>
<h3 id="bayesian-inference-is-a-modelling-paradigm">Bayesian Inference is a Modelling Paradigm</h3>
<p>In traditional machine learning we specify a model and try and find the parameters of the model which best fit the data. The cost function which we use, typically the <em>likelihood</em>, gives us a measure of <em>how well the parameters fit the data</em>.</p>
<p>Bayesian inference instead seeks to find the <em>most probable</em> parameters, given the data. By seeking <em>most probable</em>, we imbue the model with predictive uncertainty. In Bayesian inference we model a <em>posterior distribution</em> over the parameters.</p>
<h3 id="bayesian-inference-is-an-optimization-paradigm">Bayesian Inference is an Optimization Paradigm</h3>
<p>In the traditional machine learning approach we can only find a single point estimate of the optimal parameter \(\theta^{*}\). This is known as a <em>maximum likelihood estimate (MLE)</em> and can be interpreted as follows: <em>if true, we would likely observe the data which we have under this model</em>.</p>
<p>In the Bayesian inference approach, we use the likelihood to reshape the prior distribution into a posterior distribution which places <em>highest probability over the optimal \(\theta\)</em>. This \(\theta\) is sometimes known as the <em>maximum a posteriori estimate (MAP)</em>, and can be interpreted as the <em>parameter which is a best fit for the data, given our uncertainty</em>.</p>
<blockquote>
<p>Take note that when using the predictive distribution above we are not explicitly using the MAP estimate for \(\theta\) but rather taking many samples of \(\theta\). Of course if our uncertainty over \(\theta\) is not too large, than most of the \(\theta\) samples will be close to the MAP estimate.</p>
</blockquote>
<p>In the algorithmic approach, our aim is to optimize \(\theta\) based on the data. Except now we can optimize \(\theta\) according to the posterior distribution.</p>
<h3 id="bayesian-inference-is-a-machine-learning-paradigm">Bayesian Inference is a Machine Learning Paradigm</h3>
<p>In modern machine learning, Bayesian inference gives us a framework by which we can resolve the desire for an accurate algorithm with a measure of uncertainty. Using approximate inference we can scale Bayesian inference to large datasets and models with high dimensional parameters, such as neural networks.</p>
<p>Really this whole post has been about how we can rethink machine learning using Bayes’ Theorem, and that ultimately, is what Bayesian inference is.</p>
<h1 id="bonus-a-little-bit-of-information-theory">Bonus: A Little Bit of Information Theory</h1>
<p>In a way we can think of Bayesian inference (and machine learning in general) as an exercise in compression. We don’t have access to all the data in the world for our problem. Instead we use the data which we have to optimize the parameters \(\theta\) of a function which we hope can describe the relationship between \(\mathscr{X}\) and \(\mathscr{Y}\). In general the dimensionality of \(\theta\) is much lower than the data, so really we are <em>compressing</em> all the information in our dataset into the parameter \(\theta\).</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/compression_with_theta.png" /></p>
<p>When we go in the forward direction of Bayesian inference, we are finding the \(\theta\) which most <em>likely</em> describes the data. In other words, the <em>likelihood</em> helps us to <strong>encode</strong> the data into \(\theta\). When we go in the backward direction (posterior inference), we sample many values of \(\theta\) to account for our uncertainty. If our posterior distribution is good, then we will almost always sample the optimal value of \(\theta\) such that such that we should be able to recover original dataset. In other worlds the posterior helps us to <strong>decode</strong> the input to our target values.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="90%" src="/assets/article_images/2020-04-19-understanding-bayesian-inference/compression_with_posterior.png" /></p>
<hr />
<p><em>Photo by Dattatreya Patra on Unsplash</em></p>Jonty SinaiWhat do we mean when we say “Bayesian inference”? More specifically, what does Bayesian inference mean for my machine learning or data modelling problem? In this blogpost I will introduce Bayesian inference and explain how it is a machine learning paradigm. More importantly I will attempt to intuitively bridge the gap between Bayesian inference as a theoretical framework and Bayesian inference as a machine learning approach.Understanding Neural ODE’s2019-01-18T00:00:00+00:002019-01-18T00:00:00+00:00https://jontysinai.github.io/jekyll/update/2019/01/18/understanding-neural-odes<p><em>In this blogpost I explore how ODE’s can be used to solve data modelling problems. I take a deep dive into the data modelling problem at hand and present ODE’s (which model rates of change) as an alternative to regression (which attempts to model data directly). Later I introduce the extension to neural ODE’s. To keep the focus on neural ODE’s I’ll assume that you have knowledge of linear regression, deep learning and basic calculus.</em></p>
<p><em>This is a long post and I hope that you will take the time to read it in its entirety. I’ve split the post into 5 numbered parts, which I summarise at the end as 5 conceptual steps (leaps). If at times you feel that you are losing track of the bigger picture, then feel free to scroll to the end to see where each idea fits.</em></p>
<p>Many of you may have recently come across the concept of “Neural Ordinary Differential Equations”, or just “Neural ODE’s” for short. Based on a <a href="https://arxiv.org/abs/1806.07366">2018 paper</a> by Ricky Tian Qi Chen, Yulia Rubanova, Jesse Bettenourt and David Duvenaud from the University of Toronto, neural ODE’s became prominent after being named one of the <a href="https://nips.cc/Conferences/2018/Awards">best student papers</a> at NeurIPS 2018 in Montreal. Shortly afterwards a media feature in the <a href="https://www.technologyreview.com/s/612561/a-radical-new-neural-network-design-could-overcome-big-challenges-in-ai/">MIT Tech Review</a> and a front page appearance on <a href="https://news.ycombinator.com/item?id=18676986">Hacker News</a> helped propel neural ODE’s into the machine learning limelight.</p>
<blockquote>
<p>MIT Tech Review described the architechture as a “radical new design” with the “potential to shake up the field—in the same way that Ian Goodfellow did when he published his paper on GANs.”</p>
</blockquote>
<p>This may sound like hype, and perhaps it is, however I’m intrigued and excited by neural ODE’s for several reasons. At first glance they appear to have immediate practical advantages (this is where I believe the hype differs from GANs), for example in continuous time settings. Secondly ODE’s have a long history in applied and pure mathematics. They are well studied in physics, engineering and other sciences. Most modern scientific programmes have extremely well-tested and high performing differential equation libraries.</p>
<p>Finally neural ODE’s bring a powerful modelling tool out of the woodwork. When I was an undergrad in mathematics, we were taught that we solve applied maths problems using differential equations. Today we are taught that we solve applied maths problems using machine learning. I got into machine learning because I found neural networks to be the most promising tool for solving problems with maths. Neural ODE’s open up a different arena for solving problems using the muscle power of neural networks.</p>
<blockquote>
<p>In a word, they are a indeed a “radical” new paradigm in machine learning.</p>
</blockquote>
<p>In this blogpost I explore this new paradigm, starting with the initial data modelling problem. I’ll introduce ODE’s as an alternative approach to regression and explain why they may hold an advantage. I’ll give a brief perspective of the world of numerical ODE solvers. After that I’ll introduce neural ODE’s as they are described in the paper. I’ll briefly explain how backpropogation is implemented, which is the major contribution of the paper. Finally I will bring everything together.</p>
<h1 id="1-modelling-data">1. Modelling Data</h1>
<p><br /></p>
<p>In a classical data modelling setting we have a set of \(N\) pairs of data points, \(\mathscr{D} = \big\{(x_1, y_1), (x_2, y_2), …, (x_N, y_N)\big\}\). \(\mathscr{X}\) is the <em>input</em> domain and \(\mathscr{Y}\) is the <em>output</em> domain. Given a new data point, \(x^{*}\), we would like to make a <em>prediction</em> about it’s value \(y^{*}\). We can view this problem as:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/nature_black_box.png" /></p>
<blockquote>
<p><strong>Nature generates the data:</strong> we can try and describe the data using an algorithm and then use the resulting model to make new predictions. For more on this approach, see the seminal essay, <a href="https://projecteuclid.org/euclid.ss/1009213726"><strong>Two Cultures</strong></a>, on data modelling by the late Leo Breiman.</p>
</blockquote>
<p>Our original dataset is generated by nature (physical, social, economic or otherwise). We have no good sophisticated way of remodelling a reliable data generation process, so instead we treat nature as a black box and bypass it with an algorithm. The machine learning approach is to iteratively find a function which best describes the data. The process of finding this function is known as a <em>learning algorithm</em>. In short, machine learning can be thought of as repurposing the original data modelling problem into a <em>function approximation problem</em>. In modern machine learning, particularly deep learning, these functions are highly flexible <strong>neural networks</strong>. Thus the original data modelling problem becomes something like this:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/nn_bypassing_nature.png" /></p>
<blockquote>
<p><strong>Neural networks</strong> can be viewed as an algorithmic approximation which bypasses the generative process of the data. They are trained over many iterations of an optimisation loop.</p>
</blockquote>
<p>Part of the success of machine learning (which I will use interchangeably with deep learning) lies in the enormous flexibility of neural networks. They are high-dimensional, have millions of parameters over which we can compress patterns found in data, are highly nonlinear and can be implemented using highly optimised computing frameworks.</p>
<p>However, in order to understand where neural ODE’s fit in, it will be useful to abstract away from neural networks and return to the original function approximation perspective.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/function_x_to_y.png" /></p>
<blockquote>
<p><strong>Our ultimate goal</strong> will be to find a robust mapping from \(\mathscr{X}\) to \(\mathscr{Y}\).</p>
</blockquote>
<p>We are now at the fundamental mathematical problem where there is some function, which we would like to know, which sends points in \(\mathscr{X}\) to points in \(\mathscr{Y}\):</p>
<p>\[
f: \mathscr{X} \to \mathscr{Y}. <br />
\]</p>
<h3 id="two-basic-approches-odes-vs-regression">Two Basic Approches: ODE’s vs Regression</h3>
<p><br /></p>
<p>Given the dataset, \(\mathscr{D}\), there are two basic approaches for solving this problem. The first approach, known as <strong>regression</strong>, should be familiar to anyone working in machine learning. The other approach, which is the <em>dark horse</em> here, is to use an <strong>ordinary differential equation</strong>.</p>
<p>To explain these two approaches, let’s suppose that \(\mathscr{X}\) and \(\mathscr{Y}\) are both just the ordinary real number line, \(\mathbb{R}\), so that the problem can be visualised and intuitioned easily, and that we have the following arbitrary data points:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/toy_data_in_r2.png" /></p>
<blockquote>
<p><strong>Linear data</strong> in \(\mathbb{R}^2\).</p>
</blockquote>
<p>How should we go about finding the function</p>
<p>\[
f: \mathbb{R} \to \mathbb{R}. <br />
\]</p>
<p>which best describes the data?</p>
<h3 id="curve-fitting">Curve Fitting</h3>
<p><br /></p>
<p>I’ve kept this situation simple and low dimensional so that we can use ordinary <em>linear regression</em>. The data has a somewhat linear shape so we describe it using the <em>parametric form</em> of a line:</p>
<p>\[
\widehat{y} = ax + b,
\]</p>
<p>where \(y_i = \widehat y_i + \epsilon\), and \(\epsilon\) is an error term added to our model, representing random noise in the data. In linear regression we try to minimise the <em>mean square error</em></p>
<p>\[
\mathscr{L}(a, b) = \frac{1}{n}\sum_{i=1}^{N}\big(y_i - \widehat{y}_i\big)^2.
\]</p>
<p>The mean square error is a <strong>loss function</strong> which is a function of the <strong>parameters</strong> \(a\) and \(b\) - since we take our data as fixed and each \(\widehat y_i\) is evaluated at known \(x_i\) for unknown \(a\), \(b\). In machine learning we find the <em>optimal choice</em> of the parameters which <em>minimise</em> the loss function. Call them \(a^*\) and \(b^*\). The function which we’ve approximated is then the line</p>
<p>\[
f^*(x) = a^*x + b^*,
\]</p>
<p>which will look something like this:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/curve_fitting.png" /></p>
<blockquote>
<p><strong>The line of best fit</strong> produced by linear regression.</p>
</blockquote>
<p>In general this is known as curve fitting.</p>
<h2 id="2-modelling-rates-of-change">2. Modelling Rates of Change</h2>
<p><br /></p>
<p>In order to optimise the loss function we typically require the function which we’re approximating to be differentiable. In research and in practice, great care is taken to ensure that neural network architectures are indeed differentiable. Now in <em>Calculus I</em>, we learn that every differentiable function \(f\) has a <strong>derivative</strong> \(f’\), which is the continuous limit of the <em>rate of change</em> of \(f\):</p>
<p>\[
\frac{df}{dx} = f’.
\]</p>
<p>We also learn in calculus that if a function satisfies certain continuity properties then we can go the other way round and <strong>integrate</strong> it, so that if</p>
<p>\[
F(x) = \int f(x)dx,
\]</p>
<p>then</p>
<p>\[
F’(x) = f.
\]</p>
<p>How does this relate to our modelling problem? With regression, we assumed that there was a <em>continuous</em> and <em>differentiable</em> relationship between \(x\) and \(y\), described by the function \(f\):</p>
<p>\[
y = f(x).
\]</p>
<p>In regression we try to find \(f\) <em>directly</em>. But if \(f\) is differentiable, what if we tried to find its <em>derivative</em> instead? This amounts to searching for \(f\) <em>indirectly</em> by differentiating the regression relationship:</p>
<p>\[
\frac{dy}{dx} = f’(x).
\]</p>
<p>This is a basic form of an <strong>ordinary differential equation</strong>, or an <strong>ODE</strong>. Solving the ODE is equivalent to solving the integral</p>
<p>\[
f(x) = \int f’(x)dx,
\]</p>
<p>and can therefore be viewed as <em>function approximation</em>, only here we are <em>approximating the derivative</em> instead.</p>
<h3 id="odes-basic-form">ODE’s: Basic Form</h3>
<p><br /></p>
<p>In the basic form of an ODE we allow the derivative to depend not only on \(x\), but also on \(y\). This allows for greater modelling flexibility. Now in calculus you learnt that the integral depends on at least one unknown constant value. To solve for this constant we need a pair of points, \(x_0\) and \(y_0\). If there is more than one constant, we need more pairs of points. Luckily in the machine learning scenario we have an entire dataset, \(\mathscr{D}\), of \(N\) data points!</p>
<blockquote>
<p>The functional form of the integral will depend on the approximating function we choose for the derivative. The precise form of the integral will depend on the data.</p>
</blockquote>
<p>To keep the notation limited, we’ll use \(f\) to denote the function describing the derivative in the ODE. The setup is then,</p>
<p>\[
y’(x) = f(x, y), \ \ \ \ y(x_0) = y_0,
\]</p>
<p>where \(y(x)\) is interpreted as “the value of \(y\) at \(x\). From the most abstract point of view, nothing much has changed. We are still interested in finding some function - called \(f\). What has changed fundamentally, however, is that now this function describes the <strong>rate of change</strong> - how \(y\) changes as \(x\) changes - as opposed to the direct relationship.</p>
<p>Why is this useful? We’ll see that approximating derivatives reduces the number of parameters, and also the number of function evaluations (computal cost) required to find the optimal parameters.</p>
<h3 id="parametric-efficiency">Parametric Efficiency</h3>
<p><br /></p>
<p>Let’s go back to the dataset I showed earlier in \(R^2\). In the regression scenario, we made the assumption that the function we are trying to approximate is linear, i.e.</p>
<p>\[
y \approx f(x) = ax + b.
\]</p>
<p>In this case there are <em>two free parameters</em>, \(a\) and \(b\).</p>
<blockquote>
<p>The term <strong>free</strong> means that these parameters are allowed to <em>vary</em> as we try to minimise the loss. They are the parameters which we are interested in optimising.</p>
</blockquote>
<p>What if we used the ODE approach instead? We know the <em>parametric</em> form of the derivative, which we can write as:</p>
<p>\[
\frac{d\widehat y}{dx} = a.
\]</p>
<p>Now we are trying to approximate the function:</p>
<p>\[
\frac{dy}{dx} \approx f(x, y) = a.
\]</p>
<p>Immediately we can see that there is only <em>one free parameter</em>, \(a\).</p>
<blockquote>
<p>If we were to solve this problem analytically, we would still end up needing to solve for the parameter \(b\). However most interesting ODE problems can’t be solved analytically and require <strong>numerical methods</strong>.</p>
</blockquote>
<p>These numerical methods are implicit in that they don’t analytically give you the integral but rather a set of function evaluations at future points. Thus we don’t need to care about the extra parameter \(b\) at all. All we need is an initial point \((x_0, y_0)\) to get us started and any number of additional points from our data to tune the fit.</p>
<p>It turns out that optimising an ODE can be more computationally efficient than regression. In order to explain this, it will be helpful to look a bit more into numerical integration.</p>
<h3 id="numerical-methods-for-odes">Numerical Methods for ODE’s</h3>
<p><br /></p>
<p>When we can’t find the solution to an ODE analytically, and this is often the case in practical situations, we need to resort to numerical methods which approximate a solution at discrete evaluation points.</p>
<p>How do we go about finding the solution to an ODE numerically? For starters the gradient of the integral we are trying to solve is available to us - in our setup it is \(f(x,y)\) - and we know that the gradient traces the curvature of the function we’re trying to solve for. The picture below describes this for the parabola:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/gradients_tracing_parabola.png" /></p>
<blockquote>
<p><strong>Gradients</strong> tracing the parabola.</p>
</blockquote>
<p>So we can use precisely this logic to compute integrals numerically. We can start at an initial point and move in the direction of the gradient evaluated at the initial point to get to a new <em>evaluation point</em>. Starting at this second evaluation point we can repeat the same procedure to move on to a <em>third</em> evaluation point, and so on.</p>
<p>This is the basic idea behind Euler’s method. While it seems simple, Euler’s method is a starting point for advanced numerical methods, which build on this basic idea with more sophisticated updates at each step. Typically a single step will be composed of smaller steps, sometimes using higher order gradients where available. The more substeps that are taken, the higher the fidelity of the method.</p>
<blockquote>
<p>Euler’s method falls in to two larger classes of methods: <a href="https://en.wikipedia.org/wiki/Runge–Kutta_methods">Runge-Kutte methods</a> and <a href="https://en.wikipedia.org/wiki/Linear_multistep_method#Adams–Bashforth_methods">Adams-Bashforth</a> methods. In both cases, Euler’s method is the simplest order method.</p>
</blockquote>
<h3 id="eulers-method">Euler’s Method</h3>
<p><br /></p>
<p>To describe Euler’s method, we’ll replace the symbol for the input domain by \(t\). By convention we’ll interpret \(t\) as being a time element in the evolution of \(y\) as we iteratively use the method. This will also tie in better with the paper. Euler’s method is derived from the basic definition of the tangential approximation of the gradient at a point:</p>
<p>\[
\frac{dy}{dt} \approx \frac{y(t + \delta) - y(t)}{\delta},
\]</p>
<p>where \(\delta\) is a fixed <strong>step-size</strong>. We can rearrange this expression to get:</p>
<p>\[
y(t + \delta) = y(t) + \delta\frac{dy}{dt}.
\]</p>
<p>This is an explicit algebraic description of the stepwise procedure which I described and illustrated above. We can then plug in the functional formula for the derivative to get</p>
<p>\[
y(t + \delta) = y(t) + \delta f(t, y).
\]</p>
<p>Finally to compute approximations for \(y\) using Euler’s method, we need to discretize the domains. Starting from an initial point \((t_0, y_0)\), we define a <strong>computation trajectory</strong> recursively as follows:</p>
<p>\[
t_{n+1} = t_{n} + \delta (n+1), \ \ \ \ n = 0, 1, 2, …
\]
\[
y_{n+1} = y_{n} + \delta f(t_{n}, y_{n}), \ \ \ \ n = 0, 1, 2, …
\]</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/eulers_method_steps.png" /></p>
<blockquote>
<p><strong>Euler’s method illustrated:</strong> at each time point, we use the current value of \(y\) to calculate the next value. The gradient tells us which direction to move in and by how much.</p>
</blockquote>
<h3 id="optimising-the-ode">Optimising the ODE</h3>
<p><br /></p>
<p>At this point we have a modelling assumption that</p>
<p>\[
\frac{dy}{dx} = f(x, y) = a,
\]</p>
<p>and we have a numerical method for evaluating \(y\) at different values of \(x\). However to use Euler’s method we need to explicitly define \(f(x, y)\), which requires that we have a value for \(a\).</p>
<blockquote>
<p>The machine learning paradigm is to treat <strong>a</strong> as a free parameter which we can optimise.</p>
</blockquote>
<p>In particular we can make an initial guess for \(a\) and use Euler’s method to compute the <strong>forward pass</strong> on our data. Each numerical method is different, but for Euler’s method we will compute the forward pass as follows:</p>
<ol>
<li>Choose an initial value for \(a\) and a <em>fixed</em> stepsize \(\delta\)</li>
<li>Choose some \((x_0, y_0)\) to be the initial value. Choose \(k\) data points to evaluate at, and order your choices. Choosing \(k < N\) points can reduce the scaling complexity of ordering.</li>
<li>Add additional points from \(\mathscr{X}\) so that you can use Euler’s method at regular intervals. This is your computation trajectory.</li>
<li>Compute Euler’s method along your computation trajectory, also evaluating at your chosen points from your data.</li>
</ol>
<p>You can then calculate the loss by comparing Euler’s method evaluated at the \(k\) chosen datapoints with their actual values. In the backward pass you can calculate a derivative of your loss function with respect to \(a\), for each evaluation point, and adjust it as you would in gradient descent.</p>
<p>In summary, by using Euler’s method in the context of a machine learning <em>optimisation</em> problem, we treat \(a\) as a free parameter. Importantly \(a\) is the only free parameter and can be used to describe a linear relationship which typically requires two parameters.</p>
<h3 id="computational-cost">Computational Cost</h3>
<p><br /></p>
<p>This process is also more compuationally efficient. In the simple linear regression example we can solve for the free parameters analytically using well known formulas. However in more complex regression (and classification) problems we use gradient descent to solve for the parameters. Each step of gradient descent requires computing the loss at every datapoint. Even if we use stochastic gradient descent, we still require several epochs, passing through the whole dataset, in order to find the optimal parameters. Throughout training we ultimately rely on not only the entire dataset itself, but also on performing function evaluations on the entire dataset during the forward and backward pass.</p>
<p>With an ODE, we require only one initial datapoint to get started. We can then perform \(N\) function evaluations at each point in the forward pass. We then need to only evaluate the loss function at a handful of data points, before deciding how to update the free parameters. We can run this for several epochs until we’re satisfied. Ultimately, solving for an ODE requires fewer function evaluations, particularly in the backward pass. In fact, one of the contributions made in the paper is to show that this tends to be emperically true for neural ODE’s as well.</p>
<blockquote>
<p>If we use modern implementations in numerical ODE methods, then we’re in even better luck. These solvers are designed to adapt the amount of computation required so that the number of function evaluations can be decreased if it doesn’t impact error.</p>
</blockquote>
<p>To get an intuition for why we do not need the whole dataset, consider a situation where we are trying to fit a <em>parabola</em> which requires two parameters to describe the derivative:</p>
<p>\[
\frac{dy}{dx} = ax + b.
\]</p>
<p>After choosing an initial guess for \(a\) and \(b\) and an initial data point, we can run several iterations of Euler’s method and compare it to the true data:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/eulers_method_diverging.png" /></p>
<blockquote>
<p><strong>Euler’s method diverging:</strong> simple methods like Euler’s method can easily diverge. More advanced methods will be more robust.</p>
</blockquote>
<p>We only need to evaluate Euler’s method at the most recent function evaluations to determine if it is diverging. We can then use gradient descent to adjust the free parameters.</p>
<h1 id="3-neural-odes">3. Neural ODE’s</h1>
<p><br /></p>
<p>So far I’ve described how we can use an ODE to solve a data modelling problem. We approximate the relationship between points in our input and output domain by optimising the functional form of the rate of change:</p>
<p>\[
\frac{dy}{dx} = f(x, y).
\]</p>
<p>With <em>deep learning</em> we can go even further. We know that neural networks are <a href="http://neuralnetworksanddeeplearning.com/chap4.html"><em>universal function approximators</em></a>. So what if we used our neural network to approximate \(f\)? In theory if we can approximate the derivative of any differentiable function using a neural network, then we have a powerful modelling tool at hand. We also have a lot more flexibility in the data modelling process.</p>
<p>In particular, consider a neural network where the <em>hidden states</em> all have the same dimension (this will also be helpful for visualising their evolutionary trajectories later). Each hidden state depends on a <em>neural layer</em>, \(f\), which itself depends on <em>(free) parameters</em> \(\theta_t\), where \(t\) is the <em>layer depth</em>. Then</p>
<p>\[
h_{t+1} = f(h_{t}, \theta_{t}).
\]</p>
<p>If we have a <em>residual network</em>, then this looks like</p>
<p>\[
h_{t+1} = h_{t} + f(h_{t}, \theta_{t}).
\]</p>
<p>One of the intuitions which inspired the paper is that this has a similar form to Euler’s method which we described above. To understand this, remember that Euler’s method is a discretisation of the <em>continuous relationship</em> between the input and output domains of the data. Neural networks are also discretisations of this continuous relationship, only the discretisation is through hidden states in a latent space. Residual neural networks create a pathway through this latent space by allowing states to depend directly on each other, just like the updates in Euler’s method.</p>
<blockquote>
<p><em>Residual neural network</em> appears to follow the modelling pattern of an ODE: namely that the continuous relationship is modelled at the level of the derivative.</p>
</blockquote>
<p>To take this logic full circle, we consider the <strong>continuous limit</strong> of each discrete layer in the network. <em>This is the radical idea proposed by neural ODE’s</em>. Instead of a discrete number of layers between the input and output domains, we allow the progression of the hidden states to become continuous:</p>
<p>\[
\frac{dh(t)}{dt} = f(t, h(t), \theta_t),
\]</p>
<p>where \(h(t)\) is the value of the hidden state evaluated for some \(t\), which we understand as a <em>continuous parametrisation of layer depth</em>. The arena is opened up for solving the data problem as a Neural ODE. The next step is to explore this dynamic even further.</p>
<h1 id="4-continuous-hidden-state-dynamics">4. Continuous Hidden State Dynamics</h1>
<p><br /></p>
<p>In Euler’s method we defined a computation trajectory by starting from some initial point \((t_0, y_0)\) and recursively evaluating the ODE at fixed step sizes:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/euler_evals_through_time.png" /></p>
<blockquote>
<p><strong>Euler’s method</strong> traces an appoximate evolution of \(y\) through time.</p>
</blockquote>
<p>The fixed stepsizes represent a <em>time scale</em> in the evolution of the ODE as we solve it numerically. This defines a <em>dynamic</em> of the outcome \(y\) with respect to <em>time</em>. In fact the functional form of the ODE neatly represents this idea:</p>
<p>\[
\frac{dy}{dt} = f(t, y).
\]</p>
<p>Now the key feature of a neural network is that we add in <em>hidden states</em> between the input and outcome:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/hidden_state_evals.png" /></p>
<blockquote>
<p><strong>Hidden state evaluations in a neural network:</strong> we can view the transformation made by each layer as an evolution of the hidden state through time.</p>
</blockquote>
<p>This looks similar to the computation trajectory of Euler’s method. Only now the fixed stepsizes correspond to the layers in the neural network, which defines a <em>dynamic</em> of the hidden state with respect to <em>depth</em>. This dynamic can be visualised as a <em>discrete evolution</em> of the hidden state, evaluated at each layer:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/discrete_hidden_state.png" /></p>
<blockquote>
<p><strong>Discrete hidden state trajectory:</strong> we can plot how the hidden state evolves through each layer in a neural network.</p>
</blockquote>
<p>When we take the continuous limit of the hidden state with respect to depth, we smooth out this computation trajectory so that in theory, the hidden state can be evaluated at any “depth”, which we now consider to be continuous:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/cont_hidden_state.png" /></p>
<blockquote>
<p><strong>Hidden state dynamics:</strong> the continuous hiddens state trajectories will vary with the free parameters.</p>
</blockquote>
<p>In a neural ODE, we parametrise this <strong>hidden state dynamic</strong> by</p>
<p>\[
\frac{dh(t)}{dt} = f(t, h(t), \theta_t),
\]</p>
<p>where \(f(t, h(t), \theta_t)\) is a neural network layer parametrised by \(\theta_t\) at layer \(t\).</p>
<p>We can evaluate the value of the hidden state at any depth by solving the integral</p>
<p>\[
h(t) = \int f(t, h(t), \theta_t)dt.
\]</p>
<p>How do we solve this integral? Using a numerical ODE method of course. The final modelling assumption is to take the input as \(h(t_0) = x\) - i.e. the <em>initial value</em> of the ODE - and to let the output be evaluated at some time \(t_1\), so that \(h(t_1) = y\). The <em>function approximation</em> problem now takes place over a continuous hidden state dynamic:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/neural_ode_func_approx.png" /></p>
<blockquote>
<p><strong>Function approximation over hidden state dynamics:</strong> ultimately we are finding an implicit mapping \(F\) going from \(\mathscr{X}\) to \(\mathscr{Y}\).</p>
</blockquote>
<p>How do we decide what the values for \(t_0\) and \(t_1\) are? Well, since this is a machine learning approach we can treat them as two additional <strong>free parameters</strong> to be optimised. We can put everything together into a numerical ODE solver of our choice, which we can just call \(ODESolve\) in the spirit of the paper, so that:</p>
<p>\[
\widehat{y} = h(t_1) = ODESolve\big(h(t_0), t_0, t_1, \theta, f\big), <br />
\]</p>
<p>where \(f\) is a neural network.</p>
<h1 id="5-backpropogating-through-depth">5. Backpropogating Through Depth</h1>
<p><br /></p>
<p>The central innovation of the paper is an algorithm for <strong>backpropagating</strong> (reverse-mode differentiation) through the continuous hidden state dynamics. In the neural ODE, our parameters are not just \(\theta_t\), but also the evalutation times \(t_0\) and \(t_1\). As usual we can define any (differentiable) loss function of our choice on these (free) parameters:</p>
<p>\[
\mathscr{L}(t_0, t_1, \theta_t) = \mathscr{L}\Big(ODESolve\big(h(t_0), t_0, t_1, \theta, f\big)\Big).
\]</p>
<p>To optimise the loss, we require gradients with respect to the free parameters. As with the usual backpropagation algorithm for deep learning, the first step is to compute the gradient of the loss with respect to the hidden states:</p>
<p>\[
\frac{\partial\mathscr{L}}{\partial h(t)}. <br />
\]</p>
<p>But the hidden state itself is dependent on time (depth), so we can take a derivative with respect to time. Bear in mind that this time derivative will be a reverse traversal of the hidden states, which we will need to keep track of. This is where the <strong>adjoint method</strong> comes in, which is a decades old numerical technique for efficiently computing gradients.</p>
<blockquote>
<p>A proof for the adjoint method and how it is used in the paper can be found in Appendix B of the paper. For an accessible explanation of the adjoint method see this <a href="https://rkevingibson.github.io/blog/neural-networks-as-ordinary-differential-equations">blogpost</a> by Kevin Gibson.</p>
</blockquote>
<p>To keep track of the time dynamics, we’ll define the so-called <strong>adjoint state</strong></p>
<p>\[
a(t) = -\frac{\partial\mathscr{L}}{\partial h(t)}. <br />
\]</p>
<p>The adjoint state represents how the loss depends on the hidden state (remember that this is continuous along a trajectory) at any time \(t\). It’s time derivative is given by the following formula:</p>
<p>\[
\frac{da(t)}{dt} = -a(t)^{T}\frac{\partial f(t, h(t), \theta_t)}{\partial h(t)}.
\]</p>
<p>This derivative is computable since the loss, \(\mathscr{L}\), and \(f\), which is precisely a neural network, are both differentiable by design. This derivative also happens to be an ODE, so we can write down the solution of the adjoint state as the integral:</p>
<p>\[
a(t) = \int -a(t)^{T}\frac{\partial f(t, h(t), \theta_t)}{\partial h(t)} dt,
\]</p>
<p>or equivalently</p>
<p>\[
\frac{\partial\mathscr{L}}{\partial h(t)} = \int a(t)^{T}\frac{\partial f(t, h(t), \theta_t)}{\partial h(t)} dt,
\]</p>
<p>We can solve this integral by making a different call to an ODE solver. To get the gradient at \(t_0\), we can run this ODE solver <strong>backwards</strong> in time from the initial point which is known to us (just like usual backpropagation) at time \(t_1\):</p>
<p>\[
a(t_0) = \int_{t_1}^{t_0} -a(t)^{T}\frac{\partial f(t, h(t), \theta_t)}{\partial h(t)} dt,
\]</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2019-01-18-understanding-neural-odes/backprop_adjoint.png" /></p>
<blockquote>
<p><strong>Backpropagation using the adjoint method:</strong> the adjoint state is a vector with its own time dependent dynamics, only the trajectory runs backwards in time. We can solve for the gradients at \(t_0\) using an ODE solver for the adjoint time derivative, starting at \(t_1\).</p>
</blockquote>
<p>Now at this stage we have a way of computing the gradient with respect to \(t_1\) and \(t_0\). This covers two of the parameters. To compute gradients with respect to \(\theta\) we solve the ODE (also proven in Appendix B):</p>
<p>\[
\frac{\partial\mathscr{L}}{\partial\theta} = \int_{t_1}^{t_0} a(t)^{T}\frac{\partial f(t, h(t), \theta_t)}{\partial\theta} dt.
\]</p>
<p>Again this integral can be solved using an ODE solver. However the paper insists that we can do even better. All three integrals can be computed using <em>only one call to an ODE solver</em> by <strong>vectorising the problem</strong>. This is described in Appendix B.</p>
<h3 id="optional-augmented-state-dynamics">(Optional) Augmented State Dynamics</h3>
<p><br /></p>
<p><em>The rest of this section is based on Appendix B of the paper. It is more mathematically involved but worth going through to understand the computational implementation of backpropagation for neural ODE’s. Feel free to skip ahead to the final section and come back to this point at a later stage.</em></p>
<ol>
<li>Let \(\theta\) be “independent of time” such that
\[
\frac{d\theta}{dt} = 0,
\]
and observe that
\[
\frac{dt}{dt} = 1.
\]</li>
<li>Let \([h, \theta, t]\) represent an <em>augmented state</em>, then we can define the <em>augmented state function</em>:
<br /><br /><script type="math/tex">f_{aug}([h, \theta, t]) =
\begin{bmatrix}
f\big(t, h(t), \theta\big) \\
0 \\
1 \\
\end{bmatrix}.</script></li>
<li>Then let the <em>augmented state dynamics</em> be given by:
<br /><br /><script type="math/tex">\frac{d}{dt} \bigg[ \begin{smallmatrix}
h \\
\theta \\
t \\
\end{smallmatrix} \bigg] = f_{aug}([h, \theta, t]).</script></li>
<li>This is an ODE which has <em>augmented adjoint state</em>:
<br /><br /><script type="math/tex">a_{aug} =
\begin{bmatrix}
a \\
a_\theta \\
a_t \\
\end{bmatrix},</script></li>
</ol>
<p>where \(a\) is the adjoint for the hidden state described above:</p>
<p>\[
a = -\frac{\partial\mathscr{L}}{\partial h},
\]</p>
<p>and
\[
a_\theta = \frac{\partial\mathscr{L}}{\partial\theta}, \ a_t = \frac{\partial\mathscr{L}}{\partial t}.
\]</p>
<p>The time derivatives of this adjoint state can then be computed using the following formula (if you know anything about vector calculus, then the details can be found in Appendix B of the paper):
\[
\frac{da_{aug}}{dt} = - \biggl[ a\frac{\partial f}{dh}, \ a\frac{\partial f}{d\theta}, \ a\frac{\partial f}{dt} \biggr].
\]</p>
<p>The entire backpropagation algorithm can now be solved by making a call to an ODE solver on the <em>augmented state dynamics</em>.</p>
<h1 id="tying-everything-together">Tying Everything Together</h1>
<p><br /></p>
<p>Thank you for making it to this point. I hope that you now have a better understanding of how neural ODE’s can help solve your data modelling problem.</p>
<p>My intention is to explain intuitively how ODE’s can model a simple modelling problem, and how they can be optimised (in a simple scenario). The second half of this post extended that simple model to the neural ODE model as it’s presented in the paper. We covered how an ODE problem can be paramatrised by a neural network and how the neural network parameters can be optimised by backpropagating through the ODE using the adjoint method.</p>
<p>For now, it is worth reiterating the neural ODE approach to solving a data modelling problem.</p>
<ol>
<li>We have a set of \(N\) pairs of data points, \(\big\{(x_1, y_1), …, (x_N, y_N)\big\}\). Given a new data point, \(x^{*}\), we would like to make a prediction about it’s value \(y^{*}\). We seek a functional approximation to the relationship between the input and output domains.</li>
<li>Instead of modelling this relationship directly, we model the derivative:
\[
\frac{dy}{dx} = f(x, y).
\]</li>
<li>We paramatrise this approximation by a neural network with hidden states \(h(t)\), <em>depending continuously</em> on layer depth \(t\), with \(h(t_0) = x\) and \(h(t_1) = y\). The function approximation problem is now
\[
\frac{dh(t)}{dt} = f(t, h(t), \theta),
\]
where \(f\) is a neural network.</li>
<li>This is an ODE describing <em>continuous hidden state dynamics</em>. We can solve the data modelling problem by solving this ODE. This reduces to solving the integral
\[
h(t) = \int f(t, h(t), \theta) dt.
\]
To make predictions we solve the definite integral
\[
\widehat{y} = h(t_1) = \int_{t_0}^{t_1} f(t, h(t), \theta) dt.
\]
The analytical solution of this integral is not available to us. Instead we can use a numerical method (available to us using modern numerical computation software) to solve the integral at the required evaluation points:
\[
\widehat{y} = h(t_1) = ODESolve(h(t_0), t_0, t_1, \theta, f).
\]</li>
<li>The free parameters of this problem are \(t_0\), \(t_1\) and \(\theta\). We optimise our choice of these free parameters by backpropagating through the ODE solver using the method of adjoints.</li>
</ol>
<p><em>Photo by Randall Ruiz on Unsplash</em></p>Jonty SinaiIn this blogpost I explore how ODE’s can be used to solve data modelling problems. I take a deep dive into the data modelling problem at hand and present ODE’s (which model rates of change) as an alternative to regression (which attempts to model data directly). Later I introduce the extension to neural ODE’s. To keep the focus on neural ODE’s I’ll assume that you have knowledge of linear regression, deep learning and basic calculus.Probability Part 2: Conditional Probability2018-12-23T00:00:00+00:002018-12-23T00:00:00+00:00https://jontysinai.github.io/jekyll/update/2018/12/23/probability-part-two-conditional-probability<p><em>This is the second in a series of blogposts which I am writing about probability. In this post I introduce the fundamental concept of conditional probability, which allows us to include additional information into our probability calculations. The ideas behind conditional probability lead naturally to the most important idea in probability theory, known as Bayes Theorem. As with the first post, which you can read <a href="https://jontysinai.github.io/jekyll/update/2017/11/23/probability-for-everyone.html">here</a>, the approach is to explain conditional probability using mathematical ideas from measure theory. Again, as with the first post, this post is meant to be accesible to anyone, regardless of whether you’ve studied maths or not.</em></p>
<h1 id="a-game-of-dice">A Game of Dice</h1>
<p><br /></p>
<p>In my last post I discussed probability as a way of the measuring the uncertainty of an event. We thought of probability as assigning a weight to the chance that an event has a particular outcome, which we measured relative to other possible outcomes. We defined an event space, which we called \(\Omega\). On this space of events we defined a <em>random variable</em>, \(X\), as a function which maps each event to our observed outcomes, encoded as numbers on the real line, \(\mathbb{R}\):</p>
<p>\[
X:\Omega \to \mathbb{R}.
\]</p>
<p>The probability of an outcome is then the size of the number of possible events which are <em>mapped</em> to that outcome, <em>relative to the size of the whole event space</em>:</p>
<p>\[
\mathbb{P}:\mathscr{F} \to [0,1].
\]</p>
<p>We called the set of possible events the <a href="https://jontysinai.github.io/jekyll/update/2017/11/23/probability-for-everyone.html#the-preimage"><em>preimage</em></a> of the random variable. If \(B\) is an outcome, then the preimage of \(B\) is \(X^{-1}(B)\) and the probability of \(B\) is the relative size of the preimage, given by \(\mathbb{P} \big((X^{-1}(B)\big)\).</p>
<p>For instance, in the dice example from the last post where we considered the experiment of rolling two die and adding the numbers, if we observe an <em>outcome</em> \(9\), then there were four possible <em>events</em> which could add up to 9, namely \((3,6)\), \((4,5)\), \((5,4)\) and (\(6,3)\). We called this set \(A\). Since there are <a href="https://qph.fs.quoracdn.net/main-qimg-13d2e066e80c0ac1511e0477c6ffdcb4-c">\(36\)</a> possible outcomes in \(\Omega\), we can then measure the probability of getting a \(9\) by the relative size of \(A\) with respect to \(\Omega\). If \(X\) is the random variable representing our experiment, then \(A\) is the preimage of the observation \(X = 9\), and the probability of A is</p>
<p>\[
\mathbb{P}(A) = \frac{4}{36} = \frac{1}{9}.
\]</p>
<h1 id="thinking-about-the-past">Thinking About the Past</h1>
<p><br /></p>
<p>However, what if we were able to incorporate information about events that have happened before? For example, suppose that instead of rolling the two die at once, we roll them one at a time. In this case, depending on the outcome of the first roll, the probability of getting a \(9\) will be different. To see how, let’s consider an example where our first roll is a \(3\). In the first scenario there was a total of 36 possibile configurations <em>before</em> any roll was made. However, now that we’ve <em>observed</em> one of the die as a \(3\), the total number of configurations is diminished,namely the set:</p>
<p>\[
B = \big\{(3,1), (3,2), (3,3), (3,4), (3,5), (3,6) \big\}.
\]</p>
<p>In this case there is only one possible event leading to an outcome of \(9\): the event that we get a \(6\) on the second roll. How do we calculate the probability of \(9\) now? By the end of this blogpost we’ll see that calculating the <em>total uncertainty</em> will require a fundamental and important result in probability theory, known as <em>Bayes’ Theorem</em>. We’ll work our way towards that point, but for now we’ll start with the simplest scenario.</p>
<h2 id="using-what-we-already-know">Using What We Already Know</h2>
<p><br /></p>
<p>We’ve reduced the problem to the event that we have <em>already rolled</em> a \(3\), so <em>all of our uncertaintly only lies with the second roll</em>. In this case we’ve seen that there are only \(6\) possible events in total with only <em>one</em> of them leading to a \(9\). Using the same logic from the previous post, the <em>probability of rolling a \(9\), <strong>given</strong> that we have already rolled a \(3\)</em> is:</p>
<p>\[
\mathbb{P}(\text{observe a }9 | \text{rolled a }3) = \frac{1}{6}.
\]</p>
<p>The first thing to notice is that this probability is higher than the totally uncertain scenario, where \(\mathbb{P}(A) = 1/9\). This makes sense since rolling a \(3\) greatly increases our chances of getting a \(9\), even though there is only one valid configuration out of \(6\). To understand this better, suppose that we do not roll a \(\ 3\) on the first roll, then our space of possible events, which we’ll call \(\Theta\) (capital Greek letter “theta”) has \(30\) possible configurations. On this space there are \(3\) possible configurations leading to a 9, namely \((4, 5), (5, 4)\) and \((6, 3)\). So the probability of getting a \(9\) <strong>given</strong> that we <em>do not</em> roll a three on the first roll is:</p>
<p>\[
\mathbb{P}(\text{observe a }9 | \text{have not rolled a }3) = \frac{3}{30} = \frac{1}{10},
\]</p>
<p>which is less than the totally uncertain scenario, and even smaller than the scenario where we roll a \(3\) on the first throw. Clearly rolling a \(3\) on the first roll increases our chances of getting a \(9\).</p>
<blockquote>
<p>In fact, it is more general than that. Any roll that increases our chances of landing in \(A\), which is after all the preimage of \(X = 9\), will <em>increase</em> our chances. This fundamental idea can be explained graphically.</p>
</blockquote>
<h1 id="the-shape-of-probability">The Shape of Probability</h1>
<p><br /></p>
<p>Let’s go back to an earlier statement where I said that the probability of an outcome \(B\) is given by the relative size of it’s preimage:</p>
<p>\[
\mathbb{P}(B) = \frac{size\big(X^{-1}(B)\big)}{size(\Omega)}.
\]</p>
<p>Let’s unpack what this means graphically. We can visualise the events in \(\Omega\) as shaded regions - for example the set \(X^{-1}(B)\), which for simplicity we simply label as \(B\), is shown below:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2018-12-23-probability-part-two-conditional-probability/set_b.png" /></p>
<p>While these visualisations are just a heuristic and are too imprecise for measuring probabilities, they are helpful for visualising how different events interact, and hence how their probabilities may interact.</p>
<p>For example, if \(A\) is the event our rolls add up to \(9\), we can include that in our diagram and it is now clear that this event intersects \(B\) at exactly one point: the pair \((3, 6)\).</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2018-12-23-probability-part-two-conditional-probability/set_a_b.png" /></p>
<p>This was the one configuration leading to \(9\), with a \(3\) on the first roll. However, since this a point in \(\Omega\), why don’t we measure it’s probability as \(\mathbb{P}\big((3,6)\big)= 1/36\)? That would depend on what we were looking for. In particular we were interested in the <em>probability of \(9\), given that we have already rolled a \(3\)</em>. If we have already rolled a \(3\), then we are no longer in the whole space \(\Omega\).</p>
<blockquote>
<p>Configurations such as \((2, 4)\), \((1, 5)\) and \((4, 1)\) are no longer available to us, so clearly we have to redefine our space of interest.</p>
</blockquote>
<h3 id="revaluating-size">Revaluating Size</h3>
<p><br /></p>
<p>To see how to do so, let’s go back to the diagram of \(A\) and \(B\). If we have already rolled a \(3\), the possible configurations available to us have shrunk from the whole of \(\Omega\) to just the set \(B\)! This shrinking action also excludes some of the configurations in \(A\), all except for \((3, 6)\), which is in the <strong>intersection</strong> of \(A\) and \(B\). We can think of this visually as:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2018-12-23-probability-part-two-conditional-probability/condition_set.png" /></p>
<p>Now, when we take the probability of \(9\), <em>given</em> that we have rolled a \(3\), we have to revaluate how we measure size in context of what is now possible. The preimage of \(9\) is now just the intersection. Since the event \(B\) is assumed to have already happened (as visualised by the shrinking of the event space to \(B\)), we measure the size of the preimage relative to \(B\), so that</p>
<p>\[\begin{align}
\mathbb{P}(\text{observe a }9 | \text{rolled a }3) & = \mathbb{P}(A | B) \\
& \\
& = \dfrac{size(A\cap B)}{size(B)} \\
& \\
& = \dfrac{1}{6}.
\end{align}\]</p>
<h1 id="conditional-probability">Conditional Probability</h1>
<p><br /></p>
<p>What have just seen is known as the <strong>condtional probability</strong> of an event. It is the probability of the event \(A\), <em>conditional</em> on the event \(B\). By thinking of <em>conditioning</em> as a restriction on the size of the event space, we can measure the <em>conditional probability of \(A\) given \(B\)</em> as</p>
<p>\[
\mathbb{P}(A| B) = \frac{size(A\cap B)}{size(B)}.
\]</p>
<p>We can make this even more intuitive by remembering that the probability of any event is given by the size of that event relative to the whole set, namely:</p>
<p>\[
\mathbb{P}(B) = \frac{size(B)}{size(\Omega)}.
\]</p>
<p>A slight hand rearrangement gives:</p>
<p>\[
size(B) = size(\Omega)\mathbb{P}(B),
\]</p>
<p>which we can do since \(size(\Omega)\) is always positive. We can get a similar formula for the set \(A\cap B\). Plugging both formulas back into our expression for the conditional probability gives</p>
<p>\[
\mathbb{P}(A| B) = \frac{size(\Omega)\mathbb{P}(A\cap B)}{size(\Omega)\mathbb{P}(B)},
\]</p>
<p>which simplifies to</p>
<p>\[
\mathbb{P}(A| B) = \frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}.
\]</p>
<h3 id="tying-back-to-intuition">Tying Back to Intuition</h3>
<p><br /></p>
<p>Intuitively this says that the probability of \(A\) <em>given</em> \(B\) is the probability of \(A\) <em>and</em> \(B\) divided by the probability of <em>just</em> \(B\). By deriving this formula in the example, we understand this division as the shrinking of the event space. Mathematically this shrinking action is referred to as the <em><strong>projection</strong> of the event space onto the conditioning space</em>.</p>
<blockquote>
<p>Immediately we know from elementary mathematics that the probability of \(B\) cannot be zero to avoid a division error. Indeed this reinforces our intution since we cannot condition on an impossible event!</p>
</blockquote>
<p>Finally we can view the intersection as a restriction of our event of interest to the conditioning event. Indeed, the intersection of \(A\) and \(B\) can be thought of as the <em>projection of the set \(A\) onto the set \(B\)</em>, which is the conditioning space.</p>
<blockquote>
<p>All in all, we can think of <strong>conditional probability</strong> as <em>probability which is <strong>projected</strong> onto some (smaller) <strong>conditioning space</strong></em>.</p>
</blockquote>
<h1 id="bayes-theorem-the-fundamental-property-of-probability">Bayes’ Theorem: The Fundamental Property of Probability</h1>
<p><br /></p>
<p>It turns out that the intersection is symmetric: the projection of \(A\) onto \(B\) is identical to the projection of \(B\) onto \(A\). Indeed the diagrams from earlier reinforce this.</p>
<p>Based on the same procedure we could have just as easily derived the conditional probability</p>
<p>\[
\mathbb{P}(B| A) = \frac{\mathbb{P}(A\cap B)}{\mathbb{P}(A)}.
\]</p>
<p>Once again, since \(\mathbb{P}(A)\) is positive and cannot be zero, we can use a mathematical slight of hand to derive an expression for the probability of the intersection:</p>
<p>\[
\mathbb{P}(A\cap B) = \mathbb{P}(B| A)\mathbb{P}(A).
\]</p>
<p>This says that the probability of \(A\) <em>and</em> \(B\) is the <em>conditional probability of \(B\) given \(A\)</em> times the probability of <em>just</em> \(A\).</p>
<blockquote>
<p>Intuitively if we take the conditional probability of \(B\) given that \(A\) has already happened, and we factor in the probability of \(A\), we must be left with the probability of both.</p>
</blockquote>
<p>But we know from earlier that if we have the probability of the intersection, we can project the conditioning space of \(B\) to calculate the <strong>inverse probability</strong>: the <em>conditional probability of \(A\) given \(B\)</em>, which is given by:</p>
<p>\[\begin{align}
\mathbb{P}(A | B) & = \dfrac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)} \\
& \\
& = \dfrac{\mathbb{P}(B | A)\mathbb{P}(A)}{\mathbb{P}(B)}.
\end{align}\]</p>
<p>This final result is known as <strong>Bayes’ Theorem</strong>.</p>
<p>In this formulation, if \(A\) is our event of interest and \(B\) is the conditioning event, then the quantity</p>
<p>\[
\mathbb{P}(A \vert B)
\]</p>
<p>is known as the <strong>posterior probability</strong> of \(A\) given \(B\).</p>
<p>The quantity</p>
<p>\[
\mathbb{P}(B \vert A)
\]</p>
<p>is known as the <strong>likelihood</strong> of \(A\) given \(B\).</p>
<p>Finally the quantity</p>
<p>\[
\mathbb{P}(A)
\]</p>
<p>is simply known as the <strong>prior probability</strong> of \(A\).</p>
<blockquote>
<p>Intuitively the <em>prior</em> of \(A\) is the raw probability of \(A\) before any other event is taken into consideration. The <em>likelihood</em>, although taken as a conditional probability of \(B\), can be thought of as a measure of how likely it is that \(A\) depends on \(B\). To understand why, recall that we derived the likelihood’s place in the formula by considering the symmetry of the intersection. Finally the <em>posterior</em> is the final probability of \(A\) after conditioning on \(\ B\).</p>
</blockquote>
<h3 id="the-importance-of-bayes-theorem">The Importance of Bayes’ Theorem</h3>
<p><br /></p>
<p>Fundamentally, Bayes’ Theorem gives us a way of measuring conditional probability by taking into consideration our <em>total uncertainty</em>. This is the point I promised to get earlier this post. In the example, instead of considering each roll one at a time, then enumerating how the total event space changes after the first roll lands on a \(3\) we can measure \(\mathbb{P}(\text{observe a }9 \vert \text{rolled a }3)\) directly by considering all of the quantities in Bayes’ Theorem.</p>
<blockquote>
<p>I’ll leave this as an exercise but by thinking carefully about the likelihood (<em>hint</em>: recall how we derived the likelihood using the symmetry of the intersection), you can calculate the posterior and see that it yields the same result.</p>
</blockquote>
<p>Although it is not obvious from the elementary example which I used to derive conditional probability using <em>measure theoretic</em> intuition, there are situations where computing likelihoods is much easier than directly measuring the posterior of interest. This post was intended as an introduction to conditional probability, so I will not go into the details here. However, for those interested I suggest reading Chapter 3 of the late David MacKay’s book <em>Information Theory, Inference and Learning Algorithms</em>, which is available for free <a href="http://www.inference.org.uk/itila/book.html">here</a>.</p>
<blockquote>
<p>Although ITILA (as it is abbreviated to) has been superseded by more modern machine learning textbooks (which, considering the pace of research in machine learning, don’t have long either before they are out of date), it still remains a fundamental reference for learning algorithms, especially from a probabilistic point of view.</p>
</blockquote>
<h2 id="what-about-casuality">What About Casuality?</h2>
<p><br /></p>
<p>There is a natural tendency to think of conditional probability as specifying a causal relationship. If we condition \(A\) on \(B\), does this mean that \(B\) causes \(A\). In general this is not the case and it is dangerous to fall into this trap. Indeed, in the derivation I argued that the conditioning action is somewhat symmetric in the sense that if we can condition on \(B\), then we can also condition on \(A\). In most cases this is true. In fact Bayes’ Theorem demands that we think of conditional probabilities (the posterior) as depending on their <em>conditional inverse</em> (the likelihood).</p>
<p>Causality is one of the least well understood concepts in statistics. It requires that we rethink how statistical relationships are modelled. In the future, depending on how well I can understand the basic ideas, I hope to write a series of similar blogposts on causal inference.</p>
<h3 id="bonus">Bonus</h3>
<p><br /></p>
<p>The cover image for this post hints that conditional probability allows us to introduce a time element into our calculations. Indeed, given a sequence of events, we can take measurements at each event to get a <em>random process</em> \(\big\{X_1, X_2, … , X_t, …\big\}\). We can measure then measure the value of the random variable \(X_t\) at time \(t\) by considering what has already occured in the past. We are therefore interested in the conditional probability</p>
<p>\[
\mathbb{P}(X_t | \ X_1, X_2, …, X_{t-1}).
\]</p>
<p>Considering how many events in the past are relevant to the present time \(t\) can help simplify our computations of the process as it evolves in time. If \(X_t\) only depends on its last value at time \(t-1\), then the random process, \(\big\{X_t\big\}_{t \ > \ 0 \ }\), is known as a <strong>Markov Chain</strong>. We write this as</p>
<p>\[
\mathbb{P}(X_t | \ X_1, X_2, …, X_{t-1}) = \mathbb{P}(X_t | \ X_{t-1}).
\]</p>
<p>Usually the time element is taken to be discrete.</p>
<hr />
<p><em>Photo by Lukas Blazek on Unsplash</em></p>Jonty SinaiThis is the second in a series of blogposts which I am writing about probability. In this post I introduce the fundamental concept of conditional probability, which allows us to include additional information into our probability calculations. The ideas behind conditional probability lead naturally to the most important idea in probability theory, known as Bayes Theorem. As with the first post, which you can read here, the approach is to explain conditional probability using mathematical ideas from measure theory. Again, as with the first post, this post is meant to be accesible to anyone, regardless of whether you’ve studied maths or not.Probability Part 1: Probability for Everyone, a.s.2017-11-23T00:00:00+00:002017-11-23T00:00:00+00:00https://jontysinai.github.io/jekyll/update/2017/11/23/probability-for-everyone<p><em>Inspired by a course which I am taking in probability theory, this blogpost is an attempt to explain the fundamentals of the mathematical theory of probability at an intuitive level. As the title suggests, this post is pretty much intended for everyone, regardless of mathematical level or ability. There will be some mathematics, but feel free to skip through these sections. The important stuff comes in between the mathematics.</em></p>
<h1 id="chance-and-nature">Chance and Nature</h1>
<p><br /></p>
<p>If you’re reading this post, then at some point you’ve encountered chance. Chance is intertwined with nature, just as the planets revolve around the sun or as our genetic code unwinds and transforms from one generation to the next. But unlike the laws of motion of the universe, chance is uncertain. Every time you’ve played a game involving the roll of dice, or forgot to check the weather before stepping outside, or clicked next on your music streaming service, you’ve taken a bet against chance. All without knowing with certainty what hand the universe will play.</p>
<p>Chance, as we know it, is the possibility that something will happen - say that you roll 6 or that the sun will be shining on your way home*. Probability is the measure of the <em>likelihood</em> that something will happen - i.e. the probability that you roll a 6 is 1 in 6 (assuming of course that the dice is as you would expect).</p>
<p>In everyday terms, probability is our best guess at trying to predict what nature, and let’s face it, what humanity has in store. For example, you might be wondering what the probability is that today you will have three glorious looking donuts, coated in sprinkles (like the ones above).</p>
<blockquote>
<p>Probability theory is our attempt to measure uncertainty in the universe.</p>
</blockquote>
<p>* <small> This is a perfect example of how the universe plays its hand. When I started this post this morning, the sun was shining as bright as can be. The whole day was like this, except on my way home when lo and behold a runaway cloud let loose. Unexpected! <small></small></small></p>
<h1 id="measuring-the-universe">Measuring the Universe</h1>
<p><br /></p>
<p>Mathematical studies of chance and probability date back to 16th century Italy and France. During this time, the Renaissance mathematicians Geralamo Cardano, Pierre de Fermat and Blaise Pascal were mostly concerned with calculating their odds of winning games of dice. However, the modern theory of probability as we know it, was advanced during the 20th century following the development of <em>measure theory</em>.</p>
<p>The goal of measure theory was to construct a systematic method in which to compare the relative sizes of sets in mathematics. Since the probability of an event occuring is precisely a <em>measure of the likelihood</em> of that event occuring, measure theory became a natural framework for studying probability.</p>
<p>The problem with measure theory (for our purposes) is that it becomes gloriously abstract very quickly. Nonetheless, I will try and introduce the mathematical framework under which probability is studied, without going too deep into the measure theory. Let’s see how far we can get.</p>
<h3 id="a-motivating-example">A Motivating Example</h3>
<p><br /></p>
<p>Consider a game of throwing two fair dice. We would like to measure the likelihood that any roll of the dice will add up to a given number. For example, if we roll a \(3\) and \(6\), we get \(9\). Mathematically we can represent this as a function, \(X\), taking the pair \((3,6)\) to \(9\):</p>
<p>\[ <br />
X(3,6) \to 9
\]</p>
<p>We would like to consider the set of all such possible combinations of dice throws. We call this set \(\Omega\) (capital <em>omega</em>), which we can enumerate compactly as</p>
<p>\[
\Omega = \big\{(i,j): 1\leq i,j \leq6\big\}
\]</p>
<blockquote>
<p>The pair \((3,6)\) is then a member of \(\ \Omega\), which we write: \((3,6) \in \Omega\).</p>
</blockquote>
<p>However, if we consider the event that a given throw of dice sums to \(9\), then there is more than one possible combination, i.e. the pairs:</p>
<p>\[
A = \big\{(3,6), (4,5), (5,4), (6,3) \big\}
\]</p>
<p>Evaluating \(X\) (sampling) on any of the pairs in the above set will give us \(9\). Notice that the set A is not an element (member) of \(\Omega\), but it clearly contains elements of \(\Omega\)! So we need something based on \(\Omega\), only more expansive.</p>
<p>This motivates the concept of a <em>family of events*</em>, \(\mathscr{F}\), which contains all possible combinations of elements from \(\Omega\), including the whole of \(\Omega\) itself, and the empty set, denoted \(\emptyset\). In essence, \(\mathscr{F}\) is a set of sets, so that for consistency, a single roll takes the form: \(\{(3,6)\}\).</p>
<blockquote>
<p>The set \(A\) is then an element of \(\mathscr{F}\), i.e. \(A \in \mathscr{F}\).</p>
</blockquote>
<p>Now that we have a set of events, we can proceed to our goal of measuring the likelihood of these events. The mathematically precise manner in which we do this is fairly involved, but intuitively (and perhaps even realistically), we can think of the probability of an event as the likelihood of that event <em>relative</em> to the overall possibility of something happening. To this end, if \(F \in \mathscr{F}\) is <em>any event</em>, then one way of calculating its probability is:</p>
<p>\[
\mathbb{P}(F) = \frac{size(F)}{size(\Omega)}
\]</p>
<p>The ambiguity of course lies in what we mean precisely by the <em>size</em> of \(F\) or \(\Omega\), but this is a debate for another day! In our simple example, this is fairly easy:</p>
<ul>
<li>In the case of <em>one</em> die, there are 6 outcomes of a roll, let’s call them \(\Omega_1 = \{1,2,3,4,5,6\}\). There is only one way of rolling a \(1\), so
\[
\mathbb{P}(1) = \frac{size\big(\{1\}\big)}{size(\Omega_1)} = \frac{1}{6}
\]
On \(\Omega_1\), an example of an element of the <em>family of events</em> is \(\{1,2\}\), which we can interpret as rolling \(1\) <em>or</em> \(\ 2\), which has probability
\[
\mathbb{P}\big(\{1,2\}\big) = \frac{size\big(\{1,2\}\big)}{size(\Omega_1)} = \frac{2}{6} = \frac{1}{3}
\]</li>
<li>In the case of <em>two</em> dice, there are \(6^{2} = 36\) possible outcomes, with \(\Omega\) defined compactly as above. Then
\[
\mathbb{P}\big((1,2)\big) = \frac{size\big(\{(1,2)\}\big)}{size(\Omega)} = \frac{1}{36}
\]
and taking \(A\) as above:
\[
\mathbb{P}(A) = \frac{size(A)}{size(\Omega)} = \frac{4}{36} = \frac{1}{9}
\]</li>
</ul>
<p>Equipped with a set of outcomes, \(\Omega\), a family of events, \(\mathscr{F}\), and a measure of probability, \(\mathbb{P}\), we have almost everything we need to lay out the fundamental ideas of probability theory. The missing ingredient is the function we defined at the beginning of this section: \(X\). This function is a tricky object and handling it requires some care. In other words a bit more math, which we will cover shortly. For now, I will claim that we can use \(X\) to <em>encode the randomness</em> of the example game.</p>
<p>* <small> Technically \(\mathscr{F}\) is what we call a <em>\(\sigma\)-algebra</em> on \(\Omega\).
<small></small></small></p>
<h2 id="the-setting">The Setting</h2>
<p><br /></p>
<p>The motivating example has given us enough material to set the scene more generally.</p>
<p>The triple \(\big(\Omega, \mathscr{F}, \mathbb{P}\big)\) is known as a <strong>probability space</strong> if \(\ \mathbb{P}(\Omega) = 1\). In which case the map which sends elements of \(\mathscr{F}\) to numbers in the interval \([0,1]\),</p>
<p>\[
\mathbb{P}:\mathscr{F} \to [0,1] <br />
\]</p>
<p>is known as a <em>probability measure</em>.</p>
<blockquote>
<p>The requirement that \(\ \mathbb{P}(\Omega) = 1\) reflects the logic that in the world of \(\big(\Omega, \mathscr{F}, \mathbb{P}\big)\) something has got to happen from \(\Omega\).</p>
</blockquote>
<p>The elements \(A \in \mathscr{F}\) are known as <strong>events</strong>, while the elements \(\omega\) (lowercase <em>omega</em>) \(\in \Omega\) (equivalently \(\{\omega\} \in \mathscr{F}\)) are called <strong>elementary events</strong> (or <strong>realisations</strong>).</p>
<blockquote>
<p>Events in \(\mathscr{F}\) are made up of elementary events (or the lack thereof, e.g. the event that you <strong>don’t</strong> roll two 6’s).</p>
</blockquote>
<p>The next step is to define the notion of a <em>random variable</em>. Our intuition says that this must be some kind of variable that takes its values at random. Mathematically, this isn’t as obvious to define. The formal definition will follow our intuition in the following way:</p>
<blockquote>
<p>When we measure the probability of an event, we start by measuring how likely we are to <strong>observe</strong> an event.</p>
</blockquote>
<h3 id="the-preimage">The Preimage</h3>
<p><br /></p>
<p>In the case of the dice example, we can observe the sum of two dice being \(9\) in four ways, listed by the elements of the set \(A\). We will do the same thing in general by starting with a set of outcomes and measuring the size of the set of possibilities that could lead to that outcome.</p>
<p>We formalise this as follows: let \(X:\Omega \to \mathbb{R}\) be a function which sends the elementary events from \(\Omega\) to the <em>real line</em> (i.e. all the usuals numbers that we are familiar with). Then the function \(X\) will be a <strong>random variable</strong> if the <em>preimage</em> of any subset \(B \subset \mathbb{R}\) (the symbol \(\subset\) is used to indicate that \(B\) is a subset of \(\mathbb{R}\)) is contained in \(\mathscr{F}\).</p>
<blockquote>
<p>A function’s <strong>preimage</strong> of a given set, \(B\), is the set of elements which it sends to \(B\). For \(X:\Omega \to \mathbb{R}\), this is defined as:
\[
X^{-1}(B) = \{\omega \in \Omega : X(\omega) \subset B\}
\]
where \(B \subset \mathbb{R}\).</p>
</blockquote>
<hr />
<p>For instance, if we return to the function \(X\) from our working example of a game of two dice. Then the preimage of \(9\) (taking \(\{9\}\) as a subset of \(\mathbb{R}\)) is</p>
<p>\[\begin{align}
X^{-1}\big(\{9\}\big) & = \big\{(i,j) \in \Omega : X\big((i,j)\big) = 9\big\} \\
& = \big\{(i,j) \in \Omega : i + j = 9\big\} \\
& = \big\{(3,6), (4,5), (5,4), (6,3) \big\} \\
& = A
\end{align}\]</p>
<p>We know that \(A \in \mathscr{F}\). So far, so good. In fact, since we include the empty set, \(\emptyset\), as part of \(\mathscr{F}\), it turns out that the preimage of <em>any</em> subset of \(\mathbb{R}\) is in \(\mathscr{F}\). So \(X\) satisfies the condition* for it to be a random variable.</p>
<p>* <small>In reality this condition is a technical requirement which we don’t have to worry about in most (computational) settings. Rather we will be more concerned if the setting we end up using makes sense.<small></small></small></p>
<h2 id="random-variables-and-randomness">Random Variables and Randomness</h2>
<p><br /></p>
<p>Relating the maths back to our intuition, how do we interpret the formal condition that \(X:\Omega \to \mathbb{R}\) is a <strong>random variable</strong> if \(X^{-1}(B) \in \mathscr{F}\) for any subset \(B \subset \mathbb{R}\) of real numbers? We can think of this in the following way:</p>
<blockquote>
<p>If we make some observation, \(B\), which is <em>quantifiable</em> in \(\mathbb{R}\), then \(X^{-1}(B)\) is the set of possible events, through which the <strong>random variable \(X\) leads to \(B\)</strong>. In other words, the releative <strong>size</strong> \(X^{-1}(B)\) is the <strong>chance</strong> of \(B\) happening as governed by the <strong>random generating mechanism</strong> of \(X\).</p>
</blockquote>
<p>In fact, this is precisely what we do when we <em>measure</em> the likelihood of \(B\) using the probability measure:</p>
<p>\[
\mathbb{P}\big(X^{-1}(B)\big) = \mathbb{P}\big(\{\omega \in \Omega : X(\omega) \subset B\}\big)
\]</p>
<p>Motivated by this equivalence, as well as the desire for intuitive coherence (usually we don’t think of events in terms of preimages!), we can simply write</p>
<p>\[
\mathbb{P}\big(X \subset B\big) \ \ \text{in place of} \ \ \mathbb{P}\big(X^{-1}(B)\big)
\]</p>
<blockquote>
<p>We can read this representation as the <strong>probability that the random variable \(X\) lands in \(B\)</strong>.</p>
</blockquote>
<p>However, the set containment symbol looks a little unwieldy and is just there for technical completeness. We can make things easier to read by writing</p>
<p>\[
\mathbb{P}\big(X = B\big) \ \ \text{in place of} \ \ \mathbb{P}\big(X \subset B\big)
\]</p>
<blockquote>
<p>We read this as the <strong>probability that \(X\) equals \(B\)</strong>, which simply means the <em>probability that the random variable \(X\) takes on the value \(B\)</em>.</p>
</blockquote>
<p>Returning to our example, if we take \(B = \{9\}\), and then simply writing \(9\) instead of \(\{9\}\), we can recover the usual notation for the probability of rolling two numbers in a game of dice which sum to \(9\):</p>
<p>\[
\mathbb{P}\big(X = 9\big) = \frac{1}{9}
\]</p>
<p>Finally we can take a more direct approach to measuring probability by beginning with the event of an interest, say some \(A \in \mathscr{F}\) which generates some observation. To measure its likelihood we start by evaluating the random variable on this event:</p>
<p>\[
B = X(A) = \{X(\omega): \omega \in A\}
\]</p>
<p>Then we calculate:</p>
<p>\[
\mathbb{P}(A) = \mathbb{P}\big(X(A) = B\big) = \mathbb{P}(X = B) = \mathbb{P}(B)
\]</p>
<p>choosing whichever notation is more clear, depending on the context.</p>
<h3 id="bonus">Bonus</h3>
<p><br /></p>
<p>Given any set \(\mathscr{U}\), we say that \(\mathscr{U}\) happens <em>almost surely</em> (a.s.) if \(\mathbb{P}\big(\mathscr{U}\big) = 1 \). This explains the title.</p>
<hr />
<p><em>Photo by Patrick Fore on Unsplash</em></p>Jonty SinaiInspired by a course which I am taking in probability theory, this blogpost is an attempt to explain the fundamentals of the mathematical theory of probability at an intuitive level. As the title suggests, this post is pretty much intended for everyone, regardless of mathematical level or ability. There will be some mathematics, but feel free to skip through these sections. The important stuff comes in between the mathematics.The Perceptron2017-11-11T00:00:00+00:002017-11-11T00:00:00+00:00https://jontysinai.github.io/jekyll/update/2017/11/11/the-perceptron<p><em>This is the second post in a series dedicated to the history of Artificial Neural Networks. Read the first post on the MCP Neuron <a href="https://jontysinai.github.io/jekyll/update/2017/09/24/the-mcp-neuron.html">here</a>. For an accompanying Jupyter notebook with a Python implementation of the Perceptron, go <a href="https://github.com/JontySinai/PythonAI/blob/master/Notebooks/Sect1-2_Perceptron.ipynb">here</a></em></p>
<p>In my last post, I introduced the MCP Neuron, the first biologically inspired algorithm. The MCP Neuron was a significant first step in artificial intelligence as it could model the AND, OR and NOT logic gates using an algorithm inspired directly by the neurons found in the brain. This post is about the Perceptron, a natural evolution of the MCP Neuron, which incorporated an early version of a <em>learning algorithm</em>.</p>
<p>As a reminder, neurons in the brain are connected to each other to form a neural network. In this network, a single biological neuron can receive signals from other neurons. If the combined <em>intensity</em> of these signals is sufficient, the neuron will fire off another signal to other neurons in the network.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/bio-vs-MCP.png" /></p>
<p>The MCP Neuron modelled this dynamic by summing a vector of <em>input signals</em>, \([x_1, x_2, … , x_m]\) with values \(1\) or \(0\), together with a vector of corresponding <em>weights</em>, \([w_1, w_2, … , w_m]\) with values \(1\),\(-1\) or \(0\). If the weighted sum of the signals exceeded some <em>threshold value</em>, \(t\), the model outputs a positive signal, otherwise it outputs a null signal. I.e.</p>
<blockquote>
<p>\[
y = \begin{cases}
1, & \text{if} \ \sum_{i=1}^{m}w_{i}x_{i} \ \geq \ t, \\
0, & \text{otherwise}
\end{cases}
\]</p>
</blockquote>
<p>As an example, to model the AND gate for two input signals, we set the weights to \([1,1]\) and the threshold value to \(2\) so that all input signals must be \(1\) for the neuron to fire a positive output.</p>
<p>However, the problem with the MCP Neuron is that the weights had to be <em>pre-programmed</em> for each logic gate. Furthermore, its limited use of integers restricted the model to basic logic gates. This problem would be addressed in 1957, when the psychologist Frank Rosenblatt extended the MCP Neuron to include a learning algorithm which could automatically figure out the correct weights. He called this model the Perceptron.</p>
<h2 id="learning-algorithms-a-brief-primer">Learning Algorithms: A Brief Primer</h2>
<p><br /></p>
<p>The Perceptron is an example of a <em>supervised learning algorithm</em>. But before we look at the Perceptron, what is a learning algorithm, and what does it mean for a learning algorithm to be supervised?</p>
<blockquote>
<p>A <strong>learning algorithm</strong> is, roughly speaking, a method which adapts its computation units (for example weights in a sum) in order to achieve a desired behaviour. This is known as <strong>training</strong>.</p>
<p>Typically, learning algorithms are presented with examples of input data, with their correct output data, during training. This is known as <strong>supervised learning</strong>.</p>
</blockquote>
<p>The class of learning algorithms that the Perceptron belongs to is known as a <strong>binary classifier</strong>. Binary classifiers learn to group data into one of two classes, typically referred to as the <strong>positive class</strong> and the <strong>negative class</strong>. In the supervised learning setting, input data used during training is already <em>labelled</em> as positive (\(1\)), or negative (\(-1\)). During training, binary classifiers learn a <strong>decision boundary</strong> which separates the data into the two classes.</p>
<p>Below is an example of a 1D decision boundary (the simplest case) on the left, and a 2D decision boundary on the right.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/decision-boundaries.png" /></p>
<p>Linear decision boundaries like the examples above can be parametrised by a vector of weights, just like those in the MCP Neuron. But how does a learning algorithm figure out the correct vector of weights to fit the decision boundary?</p>
<p>The basic idea is illustrated below:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/learning.png" /></p>
<blockquote>
<ol>
<li>Initialise the weights either to 0 or small random numbers.</li>
<li>For each training example, \(x_n\), compute the predicted class, \(\hat{y_n}\), using the weighted sum:
\[
\hat{y_n} = \begin{cases}
1, & \ \mathbf{w}^{T}\mathbf{x}_{n} \ \geq \ 0, \\
-1, & \ \text{otherwise}
\end{cases}
\]</li>
<li>If a training example is misclassified, adjust the weights a little.</li>
<li>Repeat 2 - 3, until the number of misclassifications is reduced to a minimum.</li>
</ol>
</blockquote>
<p><em>Note the notation from linear algebra: \(\mathbf{w} = [w_1, w_2, … , w_m]\), \(\mathbf{x} = [x_1, x_2, … , x_m]\), and \(\mathbf{w}^{T}\mathbf{x} = \sum_{i=1}^{m}w_{i}x_{i}\).</em></p>
<p>We will now make this more concrete by looking at the Perceptron learning algorithm.</p>
<h2 id="the-perceptron">The Perceptron</h2>
<p><br /></p>
<p>The Perceptron algorithm works by comparing the true label, \(y_n\), with the predicted label, \(\hat{y_n}\), for <strong>every</strong> training example, and then updating the weights according to whether or not the weights are too small or too large.</p>
<p>How do we measure this comparison? Since the labels are either \(1\) or \(-1\), one intuitive way is to note that the difference of the predicted label and the true label is \(0\) when the label is predicted correctly, and \(\pm2\), when the label is predicted incorrectly:</p>
<blockquote>
<p>\[
y_{n} - \hat{y_n} = \begin{cases}
0 &, \ y_n \ = \ \hat{y_n}, \\
2 &, \ y_n \ = 1, \ \hat{y_n} = -1, \\
-2 &, \ y_n \ = -1, \ \hat{y_n} = 1
\end{cases}
\]</p>
</blockquote>
<p>Since the value for \(\hat{y_n}\), depends on how large (positive) or small (negative) the weights vector is, we need to reduce \(\mathbf{w}\) when it is too large and increase it when it is too small. Using the cases above, this motivates the update rule:</p>
<blockquote>
<p>\[
\mathbf{w} = \mathbf{w} + \alpha(y_n - \hat{y_n})\mathbf{x_n} <br />
\]</p>
</blockquote>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/Perceptron.png" /></p>
<p>There are three important factors in this update rule:</p>
<ul>
<li>The parameter, \(\alpha > 0\), is called the <em>learning rate</em>, which as the name suggests can be used to tune how quickly or slowly \(\mathbf{w}\) is updated.</li>
<li>The difference \((y_n - \hat{y_n})\) will increase the weights by a factor of \(2\) when we misclassify a positive example, and decrease the weights by a factor of \(-2\) when we misclassify a negative example.</li>
<li>Finally, notice that the weight update is proportional on the value of \(\mathbf{x_n}\), which ensures that we move the decision boundary to a lesser degree when \(\mathbf{w}^{T}\mathbf{x}\) is closer to zero.</li>
</ul>
<hr />
<p>For a visual explanation of why the update rule works, consider the simple 2D case given below, with initial weights vector \(\mathbf{w} = [1, 1]\) and learning rate \(\alpha = 0.2\), and only one misclassified example.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/misclassification.png" /></p>
<p>In this scenario, the training example, \(\mathbf{x^* } = [-1, 2]\), \(y^* = -1\), lies on the wrong side of the decision boundary
, and indeed, \(\mathbf{w}^{T}\mathbf{x}^* = 1 > 0\), so \(\hat{y^*} = 1\). Then the update is:</p>
<p>\[\begin{align}
\mathbf{w} & = \mathbf{w} - \alpha\big(y^* - \hat{y^* }\big)\mathbf{x}^* \\
& = \begin{bmatrix}1 \\ 1\end{bmatrix} - 0.2\big(-2\big)\begin{bmatrix}-1 \\ 2\end{bmatrix} \\
& = \begin{bmatrix}1,4 \\ 0,2\end{bmatrix}
\end{align}\]</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/alpha-just-right.png" /></p>
<p>With the new decision boundary, \(\mathbf{w}^{T}\mathbf{x}^* = -1 < 0\), so \(\hat{y^*} = -1\), as required.</p>
<h2 id="the-learning-rate">The Learning Rate</h2>
<p><br /></p>
<p>In the example above, the learning rate \(\alpha = 0.2\) was chosen so that the Perceptron correctly classified \(\mathbf{x^* }\) after just one update. To get an understanding of how this parameter affects convergence of the learning algorithm, we can look at what happens when we choose other values for the learning rate:</p>
<ul>
<li>If we set \(\alpha = 0.05\), then the updated weights vector would be \([1.1 ,0.8]\). The decision boundary is only rotated by a small amount, and thus the point \(\mathbf{x^* } = [-1,2]\) is still misclassified.</li>
</ul>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/alpha-too-small.png" /></p>
<ul>
<li>If we set \(\alpha = 0,5\), the updated weights vector is now \([2,-1]\). The decision boundary has rotated by such a large amount, that although it correctly classifies \(\mathbf{x^* }\), it incorrectly classifies other points.</li>
</ul>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-11-11-the-perceptron/alpha-too-large.png" /></p>
<p>In summary, if the learning rate is too small, the learning algorithm may fail to converge in a reasonable amount of time. If the learning rate is too large, the learning algorithm may fail to settle on a good decision boundary
at all! Thus we can see that the learning rate, \(\alpha\), is an important parameter in the learning algorithm, and is vital to the success of the Perceptron.</p>
<h2 id="conclusion">Conclusion</h2>
<p><br /></p>
<p>Introduced in 1957, the Perceptron introduced an important innovation to the field of artificial intelligence: learning algorithms. The basic premise of the Perceptron’s learning algorithm is as follows:</p>
<ul>
<li>If a point is classified correctly, do nothing.</li>
<li>If a point is misclassified, adjust the Perceptron’s decision boundary until the point is classified correctly.</li>
<li>Do this for all points, until settling on a decision boundary which minimises the number of misclassified points, possibly zero of them.</li>
</ul>
<p>However, the Perceptron suffered from two things:</p>
<ul>
<li>It could only fit simple, linear decision boundaries.</li>
<li>Its learning capabilities were notoriously oversold by Rosenblatt, with the New York Times reporting that it was</li>
</ul>
<blockquote>
<p>“the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.”</p>
</blockquote>
<p>The first weakness (at least) meant that it wasn’t long before the Perceptron was shown to be incapable of recognising simple non-linear patterns. Ultimately the Perceptron met its demise at the hands of the revered computer scientists Marvin Minksy and Seymour Papert in their infamous book, titled <em>Perceptrons</em> and released in 1969. It is believed that the criticism of Perceptrons in this book, and their extension to neural networks (explored in the next post) contributed to the so-called <em>AI Winter</em> - a period of reduced funding and activity in artificial intelligence researched that spanned the 1970’s, 1980’s and 1990’s.</p>
<p>However, as we shall explore in the next post, artificial intelligence was far from doomed, as during this period a small group of “rebel” scientists continued to work on a new type of classifier: the <em>artificial neural network</em>.</p>
<hr />
<p><em>Special thanks go to <a href="https://sebastianraschka.com">Sebastian Raschka</a> and <a href="http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/">Andrey Kurenkov</a> for the inspiration, and to <a href="https://www.coursera.org/specializations/deep-learning">Andrew Ng</a> for his passion and dedication to the field of Deep Learning. The style of this blogpost is intended to be conversational and informal. For a more formal treatment of the mathematics and code, checkout the Jupyter notebook version on Github <a href="https://github.com/JontySinai/PythonAI/blob/master/Notebooks/Sect1-2_Perceptron.ipynb">here</a>.</em></p>
<p><em>Photo by Len dela Cruz on Unsplash</em></p>Jonty SinaiThis is the second post in a series dedicated to the history of Artificial Neural Networks. Read the first post on the MCP Neuron here. For an accompanying Jupyter notebook with a Python implementation of the Perceptron, go hereThe MCP Neuron2017-09-24T00:00:00+00:002017-09-24T00:00:00+00:00https://jontysinai.github.io/jekyll/update/2017/09/24/the-mcp-neuron<p>Artificial intelligence is an incredibly exciting area of research and development which spans mathematics, statistics, computer science, engineering, philosophy, linguistics, information theory, biology, pyschology, neuroscience and others. It is also a fairly nascent area of science - what has been achieved so far are just the first steps in the journey to achieving <em>general artificial intelligence</em>.</p>
<p>The success, however, of <em>deep learning</em> in image recognition, natural language and games, has inspired the world to take note of artificial intelligence. This success has fueled a wave of media hype and attention, that has perhaps misinterpreted what AI <em>is</em> for what it <em>isn’t</em>. AI as we know it today is not capable of thought, has no consciousness and certainly does not have any sort of intelligence that can surpass our own. Rather, the “AI” that we experience in our mobile phones, the internet or read about in the news, is a collection of computational and statistical techniques known as deep learning (or machine learning, depending on the scope).</p>
<p>So what is deep learning? For the best explanation of deep learning and it’s limits, I recommend this excellent <a href="https://blog.keras.io/the-limitations-of-deep-learning.html">two-part post</a> by Francois Chollet, the creator of Keras. In summary, deep learning is a sequence of geometric transformations (linear and non-linear), that when applied to data, <em>may</em> be able to statistically model the relationships contained in that data. These geometric transformations are organised in a layered network, known as a <em>neural network</em>. This series is about these so called artificial neural networks, in which I will attempt to uncover what they are, how they work, where they come and why they are called “neural networks”.</p>
<h2 id="the-mcp-neuron">The MCP Neuron</h2>
<p><br /></p>
<p>Before there were any artificial neural networks, or even the perceptron (more on both in upcoming posts!), there was the MCP Neuron. First proposed in 1943 by the neurophysiologist Walter S. McCulloch and the logician Walter Pitts, the McCulloch-Pitts (MCP) neuron is a simple mathematical model of a <em>biological neuron</em>.</p>
<p>To understand how this model works, let’s begin with a very simplified (and certainly non-expert) explanation of a biological neuron. These are electrically excitable, interconnected nerve cells in the brain which process and transmit information through electrical and chemical signals. These neurons are all connected with each other to form a neural network in the brain. The connections between neurons are known as <em>synapses</em>. Now a single neuron in this simplified explanation consists of three parts:</p>
<ul>
<li>Soma: this is the main part of the neuron which processes signals.</li>
<li>Dendrites: these are branch-like shapes which receive signals from other neurons.</li>
<li>Axon: this a single nerve which sends signals to other neurons.</li>
</ul>
<p>The picture looks something like this:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-09-24-the-mcp-neuron/neuron.png" /></p>
<p>At its most basic, a single biological neuron may receive multiple signals from other neurons via its dendrites. These signals are then combined in the soma, and this combination <em>may</em> fire off another signal from the neuron to other neurons, which is propagated via the axon.</p>
<hr />
<p>The idea behind the MCP neuron is to abstract the biological neuron described above into a simple mathematical model. The neuron receives incoming signals as \(1\)’s and \(0\)’s, takes a weighted sum of these signals and outputs a \(1\) if the weighted sum is at least some threshold value or a \(0\) otherwise. Formally this mathematical model can be specified as follows:</p>
<blockquote>
<ul>
<li>Let \([x_1, x_2, … , x_m]\) be a vector of input signals where each \(x_i\) has a value of \(1\) or \(0\).</li>
<li>Let \([w_1, w_2, … , w_m]\) be a vector of weights corresponding to the input signals where each \(w_i\) has a value of \(1\), \(-1\) or \(0\).
<ul>
<li>Input signals with a weight of \(1\) are called <em>excitatory</em> since they contribute towards a positive output signal in the sum.</li>
<li>Input signals with a weight of \(-1\) are called <em>inhibitory</em> since they repress a positive output signal in the sum.</li>
<li>Input signals with a weight of \(0\) do not contribute at all to the neuron.</li>
</ul>
</li>
<li>Then for some threshold value \(t\), an integer, the output signal is determined by the following <em>activation function</em>:</li>
</ul>
<p>\[
y = \begin{cases}
1, & \text{if} \ \sum_{i=1}^{m}w_{i}x_{i} \ \geq \ t, \\
0, & \text{otherwise}
\end{cases}
\]</p>
<ul>
<li>The neuron is said to be “activated” when the weighted sum is greater than the threshold value.</li>
</ul>
</blockquote>
<p>The MCP Neuron is illustrated below. Note that the \(x_i\) input signals are analogous to the <em>dendrites</em>, the activation function is analogous to the <em>soma</em> and the output is analogous to the <em>axon</em>.</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-09-24-the-mcp-neuron/MCP-neuron.png" /></p>
<p>McCulloch’s and Pitt’s original experiment was to see if they could use this model to construct different <a href="http://www.ee.surrey.ac.uk/Projects/CAL/digital-logic/gatesfunc/"><em>logic gates</em></a> by simply specifying what the weights and threshold should be. In the next section, we’ll go through the basic logic gates and show how the MCP neuron can model them. For a corresponding Python implementation of these examples in action, checkout the corresponding Jupyter notebook on Github <a href="https://github.com/JontySinai/PythonAI/blob/master/Notebooks/Sec1-1_MCP_Neuron.ipynb">here</a>.</p>
<hr />
<h3 id="the-or-gate">The OR Gate</h3>
<p><br /></p>
<p>The first logic gate that we will go through is the OR gate. The OR gate indicates if there are <strong>any</strong> positive (as opposed to null) signals amongst the inputs. It will output a \(1\) if at least one of the input signals is a \(1\). For two input signals, the OR gate’s truth table looks like this:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-09-24-the-mcp-neuron/OR-gate.png" /></p>
<p>To reproduce the OR Gate using an MCP neuron, all of the weights should be \(1\), so that the neuron “considers” all inputs, and the threshold value should be \(1\), so that only one positive signal is required (at minimum) for the neuron to “activate”.</p>
<h3 id="the-and-gate">The AND Gate</h3>
<p><br /></p>
<p>The next logic gate is the AND gate. The AND gate indicates if <strong>all</strong> of the inputs signals are positive. It will output \(1\) only if all of its input signals are \(1\). It’s truth table looks this:</p>
<p><img
style=”display:block; margin-left:auto; margin-right:auto”
width=”700px”
src=/assets/article_images/2017-09-24-the-mcp-neuron/AND-gate.png”></p>
<p>To reproduce the AND Gate using an MCP neuron, all of the weights should be \(1\), again so that the neuron considers all inputs, but the threshold value should be equal to the number of inputs (eg. \(2\) for the example above), so that the neuron is activated only when all inputs are positive.</p>
<h3 id="the-not-gate">The NOT Gate</h3>
<p><br /></p>
<p>So far we have seen logic gates which consider all inputs - i.e. their MCP neuron weights were all \(1\). What about gates which ignore their inputs? This can be done using a NOT gate, which inverts the signal of its input, so that if the input is positive then the output will be null and vice-versa. In short, it <strong>negates</strong> its input signal. It’s truth table is shown below:</p>
<p><img style="display:block; margin-left:auto; margin-right:auto" width="700px" src="/assets/article_images/2017-09-24-the-mcp-neuron/NOT-gate.png" /></p>
<p>To specify a NOT Gate using an MCP neuron, set the input weights to \(-1\) and the threshold value to 0, so that the output is only ever positive when the input signal is null.</p>
<h2 id="conclusion">Conclusion</h2>
<p><br /></p>
<p>The MCP Neuron seems almost too simple to represent artificial intelligence of any kind, yet it is - and it isn’t. Formal logic is a fundamental component of intelligence. For any machine to have artificial intelligence, it surely should be able to comprehend logic gates. The idea being that logic gates can be stringed together to form logic circuits, capable of executing any kind of instruction. This is indeed what underpins modern computational processors. However, we know that CPU’s aren’t really “intelligent” - they’re just able to process any instruction given to them at lightning speed.</p>
<p>What makes the MCP Neuron different is the fact that it could reproduce logic gates using a <em>biologically inspired algorithm</em>. In the field of artificial intelligence, this was a promising achievement, since it almost surely makes sense that any kind of artificial intelligence should resemble the brain - which is after all the great stage of human intelligence.</p>
<p>The problem with the MCP Neuron is that every logic gate which it could model (and hence every logic circuit which a collection of neurons could model) had to be pre-programmed, something which is clear in the Jupyter notebook accompanying this post. This stands out as a massive contrast to the brain, which learns from experience. Nonetheless, it would take another 14 years before Frank Rosenblatt’s landmark debut of the <em>Perceptron</em> - the first <em>learning algorithm</em> of its kind. That the perceptron - and hence artificial neural networks - is a direct extension of the MCP Neuron, is what makes the MCP Neuron a cornerstone of artificial intelligence, and thus the beginning of our journey.</p>
<hr />
<p><em>This is the first post in a series dedicated to the history of Artificial Neural Networks. Special thanks go to <a href="https://sebastianraschka.com">Sebastian Raschka</a> and <a href="http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/">Andrey Kurenkov</a> for the inspiration, and to <a href="https://www.coursera.org/specializations/deep-learning">Andrew Ng</a> for his passion and dedication to the field of Deep Learning. The style of this blogpost is intended to be conversational and informal. For a more formal treatment of the mathematics and code, checkout the Jupyter notebook version on Github <a href="https://github.com/JontySinai/PythonAI/blob/master/Notebooks/Sec1-1_MCP_Neuron.ipynb">here</a>.</em></p>
<p><em>Photo by Meddy Huduti on Unsplash.</em></p>Jonty SinaiArtificial intelligence is an incredibly exciting area of research and development which spans mathematics, statistics, computer science, engineering, philosophy, linguistics, information theory, biology, pyschology, neuroscience and others. It is also a fairly nascent area of science - what has been achieved so far are just the first steps in the journey to achieving general artificial intelligence.