What is Deep Learning and what are Neural Networks?

Rekor makes use of the ‘deep learning’ technology, in turn closely related to ‘neural networks. Neural networks are sometimes known as “ANN”, artificial neural networks, to distinguish them from the neural network of the brain. But again, like artificial intelligence, there is nothing truly ‘artificial’ about ANN -- it’s just that the brain’s neural network is made of living cells, the computer’s ANN is made of non-living electrical components.

Let’s begin with the concept of a neural network. The human brain is a most incredible organ. The brain weighs about 1.2–1.4 kg or about 2% of the total body weight. The brain’s volume displaces around 1260 cc’s in men and around 1130 cc’s in women -- that’s about a liter, or a quart, roughly…although there is substantial individual variation. The brain actually uses about 20% of the body’s total consumption of energy – thinking is a big energy consumer.

There are about 100 billion neurons (brain cells) in a human brain – or so we thought. Until an enterprising researcher turned a brain into “soup” and recounted – turns out we ‘only’ have 86 billion neurons.

Our brain cells or neurons are connected by synapses – kind of tiny ‘cables’ that allow one brain cell to communicate with another by sending a very small electrical signal. The linkages of many brain cells are called a network, or neural network. The brain itself is a complex neural network, with many sub-networks comprising it. The more such connections, the better our brain is able to learn.

Deep learning is “deep” because the learning takes place through an (“artificial”) or man-made neural network, that has several layers. Just as the brain works by sending electrical signals from one neuron to another, many, many times, passing through a variety of locations in the brain and a variety of such sub-networks, so does deep learning ‘learn’ by employing several layers of neural networks, so that unsupervised learning starts at a low-level neural network, is passed on to a higher-level one, all the way up, until the learning is satisfactory or complete.

Deep learning is a relatively modern idea, dating to 2006, just a decade ago, and emerged from an academic paper by Hinton, Osindero and Teh (U. of Toronto and National University of Singapore)[1], though the idea of deep learning goes back long before that time. Hinton played a key role at Microsoft in developing modern speech recognition algorithms. In 2009, Nvidia, a Silicon Valley company that designs and sells graphics processing units (GPU’s) used deep learning together with its GPU’s to create powerful Deep Neural Networks capable of learning, 100 times faster than existing algorithms.

Neural networks today have at most a few million ‘units’ (comparable to neurons. They are similar in computing power to the brain of a worm, and many orders of magnitude smaller than human brains (86 b. neurons). Yet clever software enables them to perform way beyond that ability of a worm.

[1] Hinton, Osindero, & Teh, “A fast learning algorithm for deep belief nets”, Neural Computation, 18, 1527-1554, 2006.