Derivative of Neural Activation Function

Source: Deep Learning on Medium

Derivative of Neural Activation Function

Derivative are fundamental to optimization of neural network. Activation functions allow for non-linearity in the fundamentally linear model, which nothing but a sequence of linear operations.

There are various type of activation functions: linear, ReLU, LReLU, PReLU, step, sigmoid, tank, softplus, softmax and many other.

In this particular story, we will focus on the first order derivative of these activation functions as they are critical to the optimization of the neural network to learn a high performing network weights (parameters).

  1. relu(x) — Rectified Linear Unit
  2. lrelu(x) — Leaky Rectified Linear Unit
  3. sigmoid(x) — logistic function
  4. tanh(x) — hyperbolic tangent

For more details on the activation function and the intuition behind them, review my other activation story here