Original article was published by Bob Rupak Roy on Deep Learning on Medium

## What are AutoEncoders?

AutoEncoder is an artificial neural network model that seeks to learn from a compressed representation of an input.

There are various types of autoencoders available suited for different types of scenarios, however the commonly used autoencoder is for feature extraction.

Combining feature extraction models with different types of models have a wide variety of applications.

*Feature Extraction Autoencoders models for prediction sequence problems are quite challenging not because of the length of the input can vary, its because machine learning algorithms and neural networks are designed to work with fixed length inputs.*

**Another problem with sequence prediction is the temporal ordering of the observations can make it challenging to extract features**. Therefore special predictive models where developed to overcome such challenges. These are called Sequence-to-sequence, or seq2seq. and the widely used we already have heard of are the LSTM models.

## LSTM

*Recurrent neural networks such as the LSTM or Long Short-Term Memory *network are specially designed to support the sequential data.

They are capable of learning the complex dynamics within the temporal ordering of input sequences as well as use an internal memory to remember or use information across long input sequences.

N*OW combing Autoencoders with LSTM will allow us to understand the pattern of sequential data with LSTM then extract the features with Autoencoders to recreate the input sequence.*

**In other words,** for a given dataset of sequences, an encoder-decoder LSTM is configured to read the input sequence, encode it and recreate it. The performance of the model is evaluated based on the model’s ability to recreate the input sequence.

Once the model achieves a desired level of performance in recreating the sequence. The decoder part of the model can be removed, leaving just the encoder model. Now further this model can be used to encode input sequences.

Let’s understand this with the help of an example

## Simple LSTM Autoencoder

To reconstruct each input sequence.

First we will import all the required libraries.

`from numpy import array`

from keras.models import Sequential

from keras.layers import LSTM

from keras.layers import Dense

from keras.layers import RepeatVector

from keras.layers import TimeDistributed

Then we will create a sample of simple sequential data for input and reshaping it into the preferred LSTM input data format/shape of [samples,timesteps,features]

#define input sequence

sequence = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])#reshape input into [samples, timesteps, features]

n_in = len(sequence) #i.e. 9

sequence = sequence.reshape((1, n_in, 1))

Next we will define the encoder-decoder LSTM architecture that expects input sequences with 9 time steps and on feature and outputs a sequence with 9 time steps and 1 feature.

`model = Sequential()`

model.add(LSTM(100, activation='relu', input_shape=(n_in,1)))

model.add(RepeatVector(n_in))

#RepeatVector layer repeats the incoming inputs a specific number of time

‘’’ For encoder-decoder, input is squashed into a single feature vector, if we want our output to regenerate the same dimension as the original input,we need to convert this single feature vector 1D to 2D by replicating it using `RepeatVector()`

.’’’

model.add(LSTM(100, activation='relu', return_sequences=True))

model.add(TimeDistributed(Dense(1))) #This wrapper allows to apply a layer to every temporal slice of an input.

model.compile(optimizer='adam', loss='mse')#fit model

model.fit(sequence, sequence, epochs=300, verbose=0)#Reconstruct the input sequence

p = model.predict(sequence, verbose=0)

print(p[0,:,0])

done that’s it….

Now the above will give the reconstructed input sequence.

`[0.1100185 0.20737442 0.3037837 0.40000474 0.4967959 0.59493166`

0.69522375 0.7985466 0.9058684 ]

Let me put all of the codes together.

from numpy import array

from keras.models import Sequential

from keras.layers import LSTM

from keras.layers import Dense

from keras.layers import RepeatVector

from keras.layers import TimeDistributed#define input sequence

sequence = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])#reshape input into [samples, timesteps, features]

n_in = len(sequence) #i.e. 9

sequence = sequence.reshape((1, n_in, 1))

model = Sequential()

model.add(LSTM(100, activation='relu', input_shape=(n_in,1)))

model.add(RepeatVector(n_in))#RepeatVector layer repeats the incoming inputs a specific number of time

model.add(LSTM(100, activation='relu', return_sequences=True))

model.add(TimeDistributed(Dense(1))) '''#This wrapper allows to apply a layer to every temporal slice of an input.'''

model.compile(optimizer='adam', loss='mse')#fit model

model.fit(sequence, sequence, epochs=300, verbose=1)#Reconstruct the input sequence

p = model.predict(sequence, verbose=0)

print(p[0,:,0])

Now lets try something a bit complex having one encoder and 2 decoders. We will name it as **Composite LSTM AutoEncoders where 1 decoder will be used for reconstruction** and **another decoder will be used for prediction.**

First the encoder is defined

#define the encoder

visible = Input(shape=(n_in,1))

encoder = LSTM(100, activation='relu')(visible)#Decoder 1 for reconstructiondecoder1 = RepeatVector(n_in)(encoder)

decoder1 = LSTM(100, activation='relu', return_sequences=True)(decoder1)

decoder1 = TimeDistributed(Dense(1))(decoder1)#Decoder 2 for prediction

decoder2 = RepeatVector(n_out)(encoder)

decoder2 = LSTM(100, activation='relu', return_sequences=True)(decoder2)

decoder2 = TimeDistributed(Dense(1))(decoder2)

then we will tie up the encoder and the decoders.

`model = Model(inputs=visible, outputs=[decoder1, decoder2])`

The work flow of the composite encoder will be something like this.

#LSTM-AutoEncoderfrom keras.layers import Dense

from keras.layers import RepeatVector

from keras.layers import TimeDistributed

from keras.utils import plot_model

from numpy import array

from keras.layers import Input

from keras.models import Model#define input sequence

seq_in = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])#reshape input into [samples, timesteps, features]

n_in = len(seq_in)

seq_in = seq_in.reshape((1, n_in, 1))#prepare output sequence

seq_out = seq_in[:, 1:, :]

n_out = n_in - 1#define encoder

visible = Input(shape=(n_in,1))

encoder = LSTM(100, activation='relu')(visible)#Decoder 1 for prediction

decoder1 = RepeatVector(n_in)(encoder)

decoder1 = LSTM(100, activation='relu', return_sequences=True)(decoder1)

decoder1 = TimeDistributed(Dense(1))(decoder1)#Decoder 2 for prediction

decoder2 = RepeatVector(n_out)(encoder)

decoder2 = LSTM(100, activation='relu', return_sequences=True)(decoder2)

decoder2 = TimeDistributed(Dense(1))(decoder2)#tie up the encoder and the decoders

model = Model(inputs=visible, outputs=[decoder1, decoder2])

model.compile(optimizer='adam', loss='mse')#fit model

model.fit(seq_in, [seq_in,seq_out], epochs=300, verbose=0)#demonstrate prediction

results = model.predict(seq_in, verbose=1)

print(results)

**Now once the autoencoder has been lift the decoder is no longer needed and can be removed and we can keep the encoder as a standalone model.**

The encoder can be then used to transform input sequences to a fixed length encoded vector. We can perform this by creating a new model that has the same inputs as the original mode. And outputs directly from the end of encoder model, before the RepeatVector Layer.

`#the encoder LSTM as the output layer`

model = Model(inputs=model.inputs, outputs=model.layers[0].output)

We can view the each layer using model summary()

Here’s what we have in our case

from numpy import array

from keras.models import Sequential

from keras.models import Model

from keras.layers import LSTM

from keras.layers import Dense

from keras.layers import RepeatVector

from keras.layers import TimeDistributed

from keras.utils import plot_model#define input sequence

sequence = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])#reshape input into [samples, timesteps, features]

n_in = len(sequence)

sequence = sequence.reshape((1, n_in, 1))#define model

model = Sequential()

model.add(LSTM(100, activation='relu', input_shape=(n_in,1)))

model.add(RepeatVector(n_in))

model.add(LSTM(100, activation='relu', return_sequences=True))

model.add(TimeDistributed(Dense(1)))

model.compile(optimizer='adam', loss='mse')#fit model

model.fit(sequence, sequence, epochs=300, verbose=1)#the encoder LSTM as the output layer

#We can view each layer using model.summary()

model.summary()

model = Model(inputs=model.inputs, outputs=model.layers[0].output)#get the feature vector for the input sequence

results = model.predict(sequence)

print(results)

print(results.shape)model.summary()

Now this standalone encoder model can be used later. However if we need more than 9 sequence then we have to increase our sequence length/time steps else it will over kill the 9 sequence prediction model.

Thanks your time I tried my best to keep it short and simple keeping in mind to use this code in our daily life. This article is based on ‘machinelearningmastry’ blog where I tried to recreate simpler and an easier version of ‘How to harness the high accuracy predictions with LSTM-AutoEncoders’

I hope you enjoyed.

Feel Free to ask because “Curiosity Leads To Perfection”

My alternative internet presences, Facebook, Blogger, Linkedin, Medium, Instagram, ISSUU and my very own Data2Dimensions

Also available on Quora @ https://www.quora.com/profile/Bob-Rupak-Roy

Stay tuned for more updates.! have a good day….