Source: Deep Learning on Medium
Derivative of Neural Activation Function
Derivative are fundamental to optimization of neural network. Activation functions allow for non-linearity in the fundamentally linear model, which nothing but a sequence of linear operations.
There are various type of activation functions: linear, ReLU, LReLU, PReLU, step, sigmoid, tank, softplus, softmax and many other.
In this particular story, we will focus on the first order derivative of these activation functions as they are critical to the optimization of the neural network to learn a high performing network weights (parameters).
- relu(x) — Rectified Linear Unit
- lrelu(x) — Leaky Rectified Linear Unit
- sigmoid(x) — logistic function
- tanh(x) — hyperbolic tangent
For more details on the activation function and the intuition behind them, review my other activation story here