Universal approximation theorem

https://en.wikipedia.org/wiki/Universal_approximation_theorem

     In themathematicaltheory ofartificial neural networks, theuniversal approximation theoremstates[1]that afeed-forwardnetwork with a single hidden layer containing a finite number ofneurons(i.e., amultilayer perceptron), can approximatecontinuous functionsoncompact subsetsofRn, under mild assumptions on the activation function. The theorem thus states that simple neural networks canrepresenta wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmiclearnabilityof those parameters.

One of the first versions of thetheoremwas proved byGeorge Cybenkoin 1989 forsigmoidactivation functions.[2]

Kurt Hornik showed in 1991[3]that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of beinguniversal approximators. The output units are always assumed to be linear. For notational convenience, only the single output case will be shown. The general case can easily be deduced from the single output case.