A Crash Course in Sequential Data Prediction using RNN and LSTM

Source: Deep Learning on Medium

A Crash Course in Sequential Data Prediction using RNN and LSTM

Learn data skills and how to apply them in the competitive finance industry to boost your portfolio with hands-on experience.

Statistics, machine learning, and deep learning skills have been growing in demand in the finance field over the past decade ⁠— ever since the global financial crisis of 2008. Firms actively seek employees with high mathematical and computing ability to model market risk, predict financial asset prices, and therefore mitigate their exposure to an economic downturn. Some well-known applications of data science in the finance industry include fraud detection, risk management, price prediction, and algorithmic trading.

Two tools that are imperative to these data science applications are recurrent neural networks (RNNs) and long short-term memory models (LSTMs), which are used to predict sequential data. The finance industry is going through a major revolution with the advent of deep learning and artificial intelligence ⁠— with competition getting tougher and companies scrambling to generate larger profits, the industry is shifting to using the latest technologies in AI to predict asset prices and manage the risk of their investment portfolios.

One way for you to try out RNNs and LSTMs and add them to your resume is by participating in bitgrit’s latest AI competition. bitgrit competitions are great to hone your skills in data prediction, and for this competition, the top three winners will be awarded cash prizes totaling $10,000. The competition draws data from the foreign exchange market, providing participants with economic index data and news data, which are prime candidates to practice using RNNs and LSTMs. For more information, you can sign up and check out the forex competition here.

Recurrent Neural Networks (RNNs)

In a neural network, information passes through different layers where each layer contains nodes that process the data in different ways. This information eventually reaches the last layer to create a final output value. The number of layers in a neural network can vary the final output, and there is no heuristic as to what the optimum number of layers should be. These networks are quite adaptive and tend to learn and perform better than most machine learning algorithms.

Recurrent neural networks are a special type of neural network in which the output of a layer is fed back to the input layer multiple times in order to learn from the past data. Basically, the neural network is trying to learn data that follows a sequence. The three best examples for this would be:

  • Text-to-speech recognition
  • Predicting the next word in a sentence
  • Time series prediction

Basically, RNNs are great for natural language processing and time series tasks.

Currency price prediction naturally follows a time dimension. Financial time series show us patterns in the past that may help us to predict a future state, the same way that bitgrit’s latest AI competition asks its participants to predict the foreign exchange rate a month in the future. Instead of using machine learning, data scientists have used econometrics and time series analysis for predicting prices of financial products. While these techniques have been useful, modern computing power and the ability to gather high volumes of data have enabled the use of deep learning techniques for predicting prices.

RNN Function

Output of RNN at time t as a function of the state of the hidden layer (h) at time t given the weights and biases (theta)

h at time t is equal to a function of h at the previous time, the input at time t (x) given theta

There are many notations out there on how to mathematically represent the above, so do be aware without getting confused.

Source

Recurrent neural networks remember what happened in the past, which influences what it will predict for the future. RNNs run in a loop when reaching the hidden layer until they learn the underlying relationship of the data.

What to watch out for in an RNN

The part of an RNN that may frustrate even the most savvy data scientist is the vanishing/exploding gradient problem. Simply put, an RNN faces this problem when given a long input-output sequence. Although powerful and accurate, recurrent neural networks have a problem of identifying the relationship between inputs in a sequence that are separated by a large space. In time series applications, RNNs may have a hard time modeling the relationship of two points in time that are far apart from each other.

This phenomenon is due to the chain rule of the backpropagation algorithm. Unless the partial derivatives of when computing backpropagation are all close to 1, their product will be either very small (i.e. vanishing) when partial derivatives are less than 1, or very large (i.e. exploding) when partial derivatives are more than 1.

To account for this phenomenon, long short-term memory models have risen in popularity.

Long Short Term Memory Models (LSTMs)

Unlike RNNs, LSTMs can remember longer periods of time. They are one of many variants of RNNs, but LSTMs have caught mainstream attention.

LSTMs allow the neural network to selectively remember and forget information and introduces gates that regulate the flow of information (namely, adding and removing information).

Source
Source

The resources below will help you implement and understand RNNs and LSTMs more in depth.

Also, both RNNs and LSTMs are computationally expensive and require more power than the standard user possesses. The following platforms may assist in analysis:

The finance industry is one of the pioneers in statistics, machine learning, and deep learning. Artificial intelligence will bring about new avenues for firms to increase their profitability and provide improved user experience. Within the realm of AI, RNNs and LSTMs are the de-facto algorithms for using deep learning to predict asset prices. As such, they are essential skills for any data scientist looking to work in the competitive finance field.

To learn more about bitgrit and their latest forex AI competition (ending soon on December 31), click here.