Data-Driven Learning of Feedforward Neural Networks with Different Activation Functions

This work contributes to the development of a new data-driven method (D-DM)\nof feedforward neural networks (FNNs) learning. This method was proposed\nrecently as a way of improving randomized learning of FNNs by adjusting the\nnetwork parameters to the target function fluctuations. The method employs\nlogistic sigmoid activation functions for hidden nodes. In this study, we\nintroduce other activation functions, such as bipolar sigmoid, sine function,\nsaturating linear functions, reLU, and softplus. We derive formulas for their\nparameters, i.e. weights and biases. In the simulation study, we evaluate the\nperformance of FNN data-driven learning with different activation functions.\nThe results indicate that the sigmoid activation functions perform much better\nthan others in the approximation of complex, fluctuated target functions.\n