Recurrent Neural Network for Prediction Acceptance

Source: Deep Learning on Medium


Introduction

Today, I’m going to introduce how to use recurrent neural network(RNN) to predict whether the paper will be accept in conference. As we know, RNN is good at dealing with sequential data, by inputting sequential data step by step, the RNN model can catch the temporal information from several times step ago. In this article, I’m going to use pytorch to implement RNN and LSTM architecture to do the prediction acceptance.


Collecting Data

First of all, we need to collect data. In this implementation, we are going to use paper’s title to predict whether the paper will be accept in conference(i.e accept/reject). Our data is come from ICLR 2018 and ICLR 2017. To collect the data efficiently, we will use BeautifulSoup to scrape all the paper’s title from these two websites. Want to see more detail information on scraping website(web-crawler), you can see my previous tutorial on how to use the BeautifulSoup package. Here’s the link to the tutorial and code.


Data Preprocessing

After we have the data, we need to preprocess them first. That’s because we can’t input a word into our RNN. So we’ll build a dictionary for the data first. For example, “How are you”. We can build our dictionary in this form. {“How”: 1, “are”: 2, “you”: 3}. Data need to be preprocessed in order to put them into the recurrent neural network, and the preprocess method is below. In order to have a build test data, we choose the first 50 titles to be our test data, and the others are the training data.

We import numpy and pandas to read our data.

In the second cell, we split “space” to get all the words from paper’s title

In the third and forth cell, we will use sklearn package to build the dictionary.

In the fifth and sixth cell, we build the target label and the input, and make it as a np.array.


Recurrent Neural Network

After we have preprocess our dataset, we can go to build our neural network. Because we are going to predict whether the paper will be accept or not, we only need the final output for our prediction. That is, we only need the output in the last time step. Before we put our data into RNN model, we need to use word embedding technique to deal with our data. Because some words might have relationship between each other. Take animal and plants for example, dogs and cats should be as closer as possible, in contrast to the plants. If we don’t use word embedding, neural network cannot find the relationship. It’s very easy to do the word embedding, we just need to use nn.Embedding function. Figure below is the example of the word embedding

Word Embedding

After we have the RNN model, we can put our data and start training our model. There are some issues we need to focus, if I don’t copy b_y to b_z, it would have the wrong type even if you use method(.to(, dtype=torch.float)). Here’s the stackover flow’s issues.

At the end, the learning curve and the accuracy rate is below. Want to see more detailed code or vanilla rnn implementation, you can check the link. The link will be available until 2019/5/19(So sorry for that!!!).