On Sharpness of Error Bounds for Single Hidden Layer Feedforward Neural Networks

A new non-linear variant of a quantitative extension of the uniform boundedness principle is used to show sharpness of error bounds for approximation by sums of sigmoid and ReLU functions. Single hidden layer feedforward neural networks perform such operations. Errors of best approximation can be expressed using moduli of smoothness of the function to be learned. In this context, the quantitative extension of the uniform boundedness principle indeed allows to construct counter examples that show approximation rates to be best possible. Approximation errors do not belong to the little-o class of given bounds. By choosing piecewise linear activation functions, the discussed problem becomes free knot spline approximation. But results also hold for non-polynomial (and not piecewise defined) activation functions like inverse tangent.