Neural nets

Artificial Neural Nets

By some unknown reason people link neural nets to machine learning and vice versa. There are methods of machine learning which don't use neural nets and neural nets which can do useful job without training. Let's sort out the existing opportunities.

Formal neuron

A neuron is an elementary voting device. It has many inputs and only one output. A popular concept is that a neuron is similar to logical functions (AND, OR, NOT ...). The difference is the number of inputs (thousands in live nature) and that live neurons function on probabilistic principles. That is, neuron is an elementary unit of Fuzzy Logic. Nevertheless, in most models formal neurons are implemented as deterministic units of computing (which are a simple particular case) while probabilistic effects emerge when diverse input data flow through multiple connections of the net.

Simple perceptron

Was introduced by Rosenblatt in 1957. Since then, it was overhyped, then discredited by the reason that it can't solve the XOR problem, then rehabilitated again in the enhanced, multilayered variant. Despite the simple, 2-layered construct has certain, well reported limitations, it is very computationally efficient. If your application is within these constraints, why not use it?

Internal layers

Note that the concept of the layer is not the same in models and live stratified structures like the human neocortex. In the latter, one layer can contain neurons of different types with complicated local connections. In the former, one layer is one array of equal elements. If you want to model interaction between pyramidal and stellate neurons, you will probably create 2 arrays and define connections between them. That is, the number of layers in your model will be greater.

Convolution

What is called Convolutional Neural Nets (CNN) in many publications are indeed hybrid solutions which use this term as a brand name. Here, we will use it in the direct meaning. That is CNN is a 2-layered net with limited connectivity. Each neuron is linked only to some its neighbours in the next layer. If you know the properties of this building block, you can always understand how it works in more complicated, multilayered solutions.

Feed-forward nets

They may contain several layers, but information flows in one direction. They implement one step of image transformation.

Recurrent nets

In this variant, the output loops back to input. The corresponding function usually has the number of cycles as its parameter. RNNs are used in methods of successive approximation. As usual, there is a tradeoff: quality versus time. If you set more cycles, the result will be better, but the function will run longer.

Internal structure

Plasticity

Multi-net architecture

Hybrid solutions