Loss Function For Regression Problem in Machine Learning — Python Implementation Using Numpy and…

Original article was published on Deep Learning on Medium

Loss Function For Regression Problem in Machine Learning — Python Implementation Using Numpy and Tensorflow

Photo by ThisisEngineering RAEng on Unsplash

Regression Analysis:

Regression Analysis is basically a statistical approach to find the relationship between variables. In machine learning, this is used to predict the outcome of an event based on the relationship between variables obtained from the data-set.

Different types of Regression Algorithm used in Machine Learning. Some are:

  1. Linear regression
  2. Polynomial regression
  3. Logistic regression
  4. Stepwise Regression
  5. Ridge Regression

Cost Function or Loss Function or Error

In Machine Learning, the Cost function tells you that your learning model is good or not or you can say that it used to estimate how badly learning models are performing on your problem. This is typically expressed as a difference or distance between the predicted value and the actual value.

There are many types of Cost Function area present in Machine Learning. These are the following some examples:

  1. Cross-entropy cost
  2. Mean Squared Error
  3. Root Mean Square Error
  4. Hellinger Distance

Here are I am mentioned some Loss Function that is commonly used in Machine Learning for Regression Problems.

Let’s Get Start

Before I get started let’s see some notation that is commonly used in Machine Learning:

Summation

Summation: It is just a Greek Symbol to tell you to add up a whole list of numbers. For example, summation of [1, 2, 4, 2] is denoted 1 + 2 + 4 + 2, and results in 9, that is, 1 + 2 + 4 + 2 = 9.

y

Y: Actual Value or Real Value

y-hat

Y-hat: In Machine Learning, we y-hat as the predicted value

1. Mean Square Error or L2 Loss:

Mean Square Error or L2 Loss

Equation:

  • n = number of data points
  • y = actual value or real value (labels)
  • y-hat = predicted value

The average squared difference or distance between the estimated values (predicted value) and the actual value. Mean Square Error is almost always strictly positive (and not zero) is because of randomness or because the estimator does not account for information that could produce a more accurate estimate.

Python Implementation using Numpy:

import numpy as np 
Y_true = [1,1,2,2,4] # Y_true = Y (actual values)
Y_pred = [0.6,1.29,1.99,2.69,3.4] #Y_pred=Y-hat (predicted value)
# Mean Squared Error
MSE = np.square(np.subtract(Y_true,Y_pred)).mean()
print(MSE) # .21606

2. Mean Absolute Error or L1 Loss:

Mean Absolute Error or L1 Loss

Equation:

  • n = number of data points
  • x = actual value or real value (labels)
  • y = predicted value

Mean Absolute Error is the sum of absolute differences between our target and predicted variables. It measures the average magnitude of errors in a set of predictions, without considering their directions. It is a common measure of forecast error in time series analysis.

Python Implementation using Numpy:

import numpy as np 
Y_true = [1,1,2,2,4] # Y_true = Y (actual values)
Y_pred = [0.6,1.29,1.99,2.69,3.4] #Y_pred=Y-hat (predicted value)
# Mean Absolute Error
MAE = np.abs(np.subtract(Y_true,Y_pred)).mean()
print(MAE) # .398

3. Huber Loss or Smooth Mean Absolute Error:

The Huber loss can be used to balance between the MAE (Mean Absolute Error), and the MSE (Mean Squared Error). It is therefore a good loss function for when you have varied data or only a few outliers. It is more robust to outliers than MSE.

Python Implementation using Numpy and Tensorflow:

import tensorflow, numpyy_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
h = tf.keras.losses.Huber()
h(y_true, y_pred).numpy()
# output 0.155

4. Log-Cosh Loss:

From TensorFlow docs: log(cosh(x)) is approximately equal to (x ** 2) / 2 for small x and to abs(x) — log(2) for large x. This means that ‘logcosh’ works mostly like the mean squared error, but will not be so strongly affected by the occasional wildly incorrect prediction.

Python Implementation using Numpy and Tensorflow:

import tensorflow, numpyy_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
l = tf.keras.losses.LogCosh()
l(y_true, y_pred).numpy()
# output 0.108

Some Extra Loss Function:

Root Mean Squared Error: It is just a Root of MSE.

Python Implementation using Numpy:

import numpy as np 
Y_true = [1,1,2,2,4] # Y_true = Y (actual values)
Y_pred = [0.6,1.29,1.99,2.69,3.4] #Y_pred=Y-hat (predicted value)
# Root Mean Squared Error
RMSE = np.sqrt(np.square(np.subtract(Y_true,Y_pred))).mean()
print(RMSE) # 0.398

Mean Absolute Percentage Error: It is just a percentage of MAE.

Python Implementation using Numpy:

import numpy as np 
Y_true = [1,1,2,2,4] # Y_true = Y (actual values)
Y_pred = [0.6,1.29,1.99,2.69,3.4] #Y_pred=Y-hat (predicted value)
# Mean Absolute Percentage Error
MAPE = 100 * np.abs(np.subtract(Y_true,Y_pred)).mean()
print(MAPE) # 31.800000

Mean Squared Logarithmic Error (MSLE): It can be interpreted as a measure of the ratio between the true and predicted values. As the name suggests, it is a variation of the Mean Squared Error.

Python Implementation using Numpy and Tensorflow:

import tensorflow, numpyy_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
msle = tf.keras.losses.MeanSquaredLogarithmicError()
msle(y_true, y_pred).numpy()
# output 0.240

References