GSoC 2018: Performance of LSTM Network — Part V



Performance of LSTM features: Forward propagation and backward propagation.


In this blogpost, I’ll be sharing results obtained during testing session of forward propagation and backward propagation. This blogpost will be a short one since, it focuses more on result.

Forward Propagation Test Results:

For TestLSTMForwardPass.cxx : timeSteps = 1, batchSize = 2, stateSize = 3, inputSize = 2. Inputs and weights (input and state) are initialised randomly.

Values of random input and weights
Values of each weight matrix corresponding to each gate
Values of biases during forward pass
Hidden state and cell state initially zero.
Error during forward pass when input gate values are scaled. The error has been documented in the code.
Result of new cell state calculation.
New hidden state values is not providing expected results with dimensions of batchSize x stateSize x timeSteps

For TestLSTMForwardPassCpu.cxx : timeSteps = 1, batchSize = 2, stateSize = 3, inputSize = 2. Inputs and weights (input and state) are initialised randomly.

Getting error as 1.0 during forward pass LSTM-CPU. Running second or third time gives error ~ 0.90

Backpropagation Test Results:

For TestLSTMBackpropagation.cxx : timeSteps = 1, batchSize = 2, stateSize = 1, inputSize= 10, learningRate = 1e-5.

The current implementation when tested, goes into infinite loop. It is difficult to debug where the problem is occurring.

For TestLSTMBackpropagationCpu.cxx : timeSteps = 1, batchSize = 2, stateSize = 1, inputSize= 10, learningRate = 1e-5 and using random inputs.

Maximum relative error = 0.689
Other values

The error with random inputs and random weights is currently high. The debugging process is a bit hard. It may require changes backpropagation design. Using same weight matrices and random inputs gives maximum relative error as 1 and maximum absolute error as 0.

In my next story, I’ll be summarising my project work of LSTM networks during Google Summer Of Code’18.

Source: Deep Learning on Medium