Classify Machine Learning algorithms

Source: Deep Learning on Medium

Classify Machine Learning algorithms

There are two common ways of grouping Machine learning algorithms. One is based on the learning style, the other is based on the function (of each algorithm).

1. Grouping based on learning style

According to the learning style, Machine Learning algorithms are usually divided into 4 groups: Supervise learning, Unsupervised learning, Semi-supervised learning, and Reinforcement learning. There are several subgroups that do not have Semi-supervised learning or Reinforcement learning.

Supervised Learning (Supervised Learning)

Supervised learning is an algorithm that predicts the outcome of a new input based on known pairs (input, outcome). This data pair is also known as (data, label), ie (data, label). Supervised learning is the most common group in Machine Learning algorithms.

Mathematically, supervised learning is when we have a set of input variables X={x1,x2,…,xN} and a corresponding set of labels Y={y1,y2,…,yN}, with
xi, yi are vectors. Pairs of known data are called training data sets. From this training data set, we need to create a mapping function for each element from the set X to a corresponding (approximate) element of the set Y

The goal is to approximate the function f so well that when there is a new data x, we can compute its corresponding label y = f (x).

Example 1: in handwriting recognition, we have pictures of thousands of examples of each digit written by different people. We put these images into an algorithm and show it which digit each image represents. After the algorithm creates a model, that is, a function whose input is a photograph and the output is a digit, when it receives a new image the model has never seen, it will predict the picture. What digit does that image contain?

Example 1:

This example is quite similar to the way people learned in childhood. We give the alphabet to a child and show them this is the letter A, this is the letter B. After a few times of teaching, they can recognize what is A and which is B in a book they have never seen.

Example 2: The algorithm to detect faces in an image has been developed for a long time. At first, Facebook used this algorithm to identify the faces in a photo and ask users to tag friends — ie assign a label to each face. The greater the number of data pairs (faces, names), the greater the accuracy at subsequent auto-tagging.

Example 3: The algorithm to detect faces in an image is also a Supervised learning algorithm with training data of thousands of pairs (photos, faces) and (photos, not faces) ) is included. Note that this data only distinguishes human and non-human faces without distinguishing the faces of different people.

The supervised learning algorithm is further broken down into two main categories:

A problem is called classification if the labels of input data are divided into a finite group number. For example, Gmail determines whether an email is a spam or not; Credit firms determine whether a customer is able to pay off debt. The above three examples are divided into this category.


If the label is not divided into groups but rather a specific real value. For example, How much for a house with x (m ^ 2), y bedroom(s) and a distance of z km from the city center?

Microsoft recently had a face and age prediction app. The gender prediction part can be considered as the Classification algorithm, the age prediction part can be considered the Regression algorithm. Note that the age prediction can also be considered Classification if we consider age as a positive integer not greater than 150, we will have 150 different classes.

Unsupervised Learning (Unattended Learning)

In this algorithm, we do not know the outcome or label but only the input data. The unsupervised learning algorithm will rely on the structure of the data to perform a certain task, such as clustering or dimension reduction to facilitate storage and calculation.

Mathematically, Unsupervised learning is when we only have input to X without knowing the corresponding Y label.

Such algorithms are called Unsupervised learning because unlike Supervised learning, we don’t know the correct answer for each input. Like when we study, no teacher will tell us whether it is an A or a B. An unattended cluster is named in this sense.

Unsupervised learning problems are further broken down into two categories:


A problem that divides all X data into small groups based on the relationship between the data in each group. For example, grouping customers based on buying behavior. This is like giving a child lots of pieces of different shapes and colors, such as triangles, squares, and circles with blue and red, then asking them to divide them into groups… Although it is not possible for a child to tell which piece corresponds to which image or color, it is more likely that they can still classify the pieces by color or shape.


The problem when we want to discover a rule based on a lot of given data. For example, male customers who buy clothes tend to buy more watches or belts; Spider-Man moviegoers tend to watch more Bat Man movies, which in turn creates a Recommendation System, which drives shopping demand.

Semi-Supervised Learning

The problem when we have a large amount of X data but only part of them is labeled is called Semi-Supervised Learning. The problems in this group lie between the two groups listed above.

A typical example of this group is that only a portion of the photo or text is labeled (for example, photographs of people, animals or scientific or political texts) and most other photos/texts. Unlabelled labels are collected from the internet. In fact, many Machine Learning problems fall into this category because collecting branded data is time-consuming and expensive. Many types of data even require specialist labeling. In contrast, unlabeled data can be collected at low cost from the internet.

Reinforcement Learning

Reinforcement learning is the math that helps a system to automatically identify behavior based on circumstances to maximize the benefits (maximizing the performance). Currently, Reinforcement learning is mainly applied to Game Theory, the algorithms need to determine the next move to achieve the highest score.

AlphaGo plays Go with Lee Sedol. AlphaGo is an example of Reinforcement learning.
(Source: AlphaGo AI Defeats Sedol Again, With ‘Near Perfect Game’)

Example 1: AlphaGo has recently gained a reputation for playing Go against people. Go is considered to be extremely complex with a total of approximately 10 ^ 761 moves, compared to chess 10 ^ 120 and the total number of atoms in the entire universe is about 10 ^ 80 !! Therefore, the algorithm has to choose an optimal move out of billions of billions of choices, and of course, it is impossible to apply the same algorithm as IBM Deep Blue (IBM Deep Blue won against humans in chess). 20 years ago). Basically, AlphaGo includes algorithms for both Supervised learning and Reinforcement learning. In the Supervised learning section, data from human-played games are included for training. However, the ultimate goal of AlphaGo is not to play like a human being but to even win over a human being. So, after completing human games, AlphaGo plays itself with millions of games to find new optimal moves. The algorithm in this game is classified as Reinforcement learning.

Example 2: Training a computer to play Mario games. This is an interesting program that teaches computers to play Mario games. This game is simpler than Go because at one point, players only have to press a small number of buttons (move, jump, fire) or no buttons are needed. At the same time, the reaction of the machine is also simpler and repeats at every play (at a specific time there will appear a fixed obstacle in a fixed position). The algorithm’s input is the diagram of the screen at the present time, the task of the algorithm is with that input, which key combination should be pressed. This training is based on the score for how long the game travels, the farther and faster the game is, the higher the reward (this is not a game score but a point because the programmer creates). Through training, the algorithm will find an optimal way to maximize the number of points, thereby achieving the ultimate goal of saving the princess.

Training a computer to play Mario games.

2. Classify based on function

There is a second way of grouping based on the function of algorithms. In this section, I would only list the algorithms. Specific information will be presented in other posts on this publication(I will update asap). During the writing process, I might add or remove some algorithms.

Regression Algorithms

  1. Linear Regression
  2. Logistic Regression
  3. Stepwise Regression

Classification Algorithms

  1. Linear Classifier
  2. Support Vector Machine (SVM)
  3. Kernel SVM
  4. Sparse Representation-based classification (SRC)

Instance-based Algorithms

  1. k-Nearest Neighbor (kNN)
  2. Learning Vector Quantization (LVQ)

Regularization Algorithms

  1. Ridge Regression
  2. Least Absolute Shrinkage and Selection Operator (LASSO)
  3. Least-Angle Regression (LARS)

Bayesian Algorithms

  1. Naive Bayes
  2. Gaussian Naive Bayes

Clustering Algorithms

  1. k-Means clustering
  2. k-Medians
  3. Expectation-Maximization (EM)

Artificial Neural Network Algorithms

  1. Perceptron
  2. Softmax Regression
  3. Multi-layer Perceptron
  4. Back-Propagation

Dimensionality Reduction Algorithms

  1. Principal Component Analysis (PCA)
  2. Linear Discriminant Analysis (LDA)

Ensemble Algorithms

  1. Boosting
  2. AdaBoost
  3. Random Forest

And there are many other algorithms.

3. References

A Tour of Machine Learning Algorithms