Automatic Timeline Generation Using Learning To Rank

Original article was published on Deep Learning on Medium


Pairwise Approach of Learning To Rank

First we need to sub sample our original dataset for this technique (reason will follow soon). We have taken 34 document vectors at random from whole dataset. We divided them into training and testing data with the ratio of 75:25 approximately which results into 25 training document vectors. Now, in order to establish pairwise approach, we created the pairs of documents vectors out of training data i.e. 25 C 2 combination are possible and that is the reason we have sub sampled the dataset.

How to define training points (X_train) and ground truths (Y_train)?

Well, each pair of document vector is defined as [di,dj] which itself serves as a single training point. All such 300 pairs combined to be called X_train. For each training point, the ground truth is given as follows: if di is published before dj, ground truth is 1, 0 otherwise. So, all ground truth corresponding to every training point are combined to form Y_train.

What is testing points (x_test) and ground truth (y_test)?

After splitting the sub sampled dataset, we are left with set 9 document vectors as testing data. The correct chronological order of these testing document vectors is the ground truth for calculating how good model’s accuracy in ordering these vectors.

Let’s dive into the process now….

1. We define a standard machine learning model from Sklearn library and feed it with our training points for learning.For Baseline, we have used Logistic Regression.For our actual approach, we have used various models such as SVM, Random Forest, KNN etc. and drew their comparison over accuracy.

2. Then we will use this learned model as a comparator to sort out testing points. For this, we take two testing point at a time and the learned model will predict whether the order in which the pairs are presented is correct or not. This information at each iteration is used for chronological ordering as we have passed model prediction as comparator to the sorting function.

3. Once we have predicted chronological order and correct chronological order, we calculated the NDCG to evaluate the correctness of the predicted chronological ordering.