Neural nets

Artificial Neural Nets

By some unknown reason people link neural nets to machine learning and vice versa. There are methods of machine learning which don't use neural nets and neural nets which can do useful job without training. Let's sort out the existing opportunities.

Formal neuron

A neuron is an elementary voting device. It has many inputs and only one output. A popular concept is that a neuron is similar to logical functions (AND, OR, NOT ...). The difference is the number of inputs (thousands in live nature) and that live neurons function on probabilistic principles. That is, neuron is an elementary unit of Fuzzy Logic. Nevertheless, in most models formal neurons are implemented as deterministic units of computing (which are a simple particular case) while probabilistic effects emerge when diverse input data flow through multiple connections of the net.

Simple perceptron

Was introduced by Rosenblatt in 1957. Since then, it was overhyped, then discredited by the reason that it can't solve the XOR problem, then rehabilitated again in the enhanced, multilayered variant. Despite the simple, 2-layered construct has certain, well reported limitations, it is very computationally efficient. If your application is within these constraints, why not use it?

Internal layers

Note that the concept of the layer is not the same in models and live stratified structures like the human neocortex. In the latter, one layer can contain neurons of different types with complicated local connections. In the former, one layer is one array of equal elements. If you want to model interaction between pyramidal and stellate neurons, you will probably create 2 arrays and define connections between them. That is, the number of layers in your model will be greater.

Convolution

What is called Convolutional Neural Nets (CNN) in many publications are indeed hybrid solutions which use this term as a brand name. Here, we will use it in the direct meaning. That is CNN is a 2-layered net with limited connectivity. Each neuron is linked only to some its neighbours in the next layer. If you know the properties of this building block, you can always understand how it works in more complicated, multilayered solutions.

Feed-forward nets

They may contain several layers, but information flows in one direction. They implement one step of image transformation.

Recurrent nets

In this variant, the output loops back to input. The corresponding function usually has the number of cycles as its parameter. RNNs are used in methods of successive approximation. As usual, there is a tradeoff: quality versus time. If you set more cycles, the result will be better, but the function will run longer.

Internal structure

The main difference of the brain from our computers is that it has no separation between storage and processing. The neocortex may be regarded as both operative and long-term memory, but it also performs important functions of data processing. These functions are hard-wired in the microstructure of intracortical connections. When a computer processes some data, this involves permanent swapping between the memory and processor. When you think, the process may be triggered by some external event but happens completely within your cortical fields.

Plasticity

Human memory is implemented as changing the efficiency of synapses, but how specifically does it work? Physiology undertook substantial efforts in this direction. The overall result is that there are many different mechanisms which operate in parallel. For theoretical models and practical implementation, it is needed to choose some algorithm which represents the sum effect. One popular variant was the Hebb synapse which is enhanced when active input coincides in time with active output. This principle is no more than a brave hypothesis that transfers Pavlovian learning to the level of microstructure. Meanwhile for practical purposes, it is better not to reproduce that mechanisms 1:1 but to try and understand the general idea implemented by Nature. This idea is long-term adaptation. That is, when you train your artificial neural net, just change synapses in the way which produces the desired effect.

Multi-net architecture

For useful projects, it is necessary to link several homogeneous neural nets. Adding more layers is not equivalent. In such constructs, internal connections between neurons define processing functions; connections between nets - the architecture of the computing system. Most interesting, links between various fields of the neocortex correspond to software of common computers.

Hybrid solutions

In practice, several aforementioned features are used in combination. For example, you may use CNN and add some biofeedback that is RNN.