LSTM Neural Network Architecture

The core components of an LSTM neural network are a sequence input layer and an LSTM layer. A sequence input layer inputs sequence or time series data into the neural network. An LSTM layer learns long-term dependencies between time steps of sequence data.

This diagram illustrates the architecture of a simple LSTM neural network for classification. The neural network starts with a sequence input layer followed by an LSTM layer. To predict class labels, the neural network ends with a fully connected layer, a softmax layer, and a classification output layer.

This diagram illustrates the architecture of a simple LSTM neural network for regression. The neural network starts with a sequence input layer followed by an LSTM layer. The neural network ends with a fully connected layer and a regression output layer.

This diagram illustrates the architecture of a neural network for video classification. To input image sequences to the neural network, use a sequence input layer. To use convolutional layers to extract features, that is, to apply the convolutional operations to each frame of the videos independently, use a sequence folding layer followed by the convolutional layers, and then a sequence unfolding layer. To use the LSTM layers to learn from sequences of vectors, use a flatten layer followed by the LSTM and output layers.

numFeatures = 12;
numHiddenUnits = 100;
numClasses = 9;
layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits,'OutputMode','last')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

numFeatures = 12;
numHiddenUnits = 100;
numClasses = 9;
layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits,'OutputMode','sequence')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

numFeatures = 12;
numHiddenUnits = 125;
numResponses = 1;

layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits,'OutputMode','last')
    fullyConnectedLayer(numResponses)
    regressionLayer];

numFeatures = 12;
numHiddenUnits = 125;
numResponses = 1;

layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits,'OutputMode','sequence')
    fullyConnectedLayer(numResponses)
    regressionLayer];

inputSize = [28 28 1];
filterSize = 5;
numFilters = 20;
numHiddenUnits = 200;
numClasses = 10;

layers = [ ...
    sequenceInputLayer(inputSize,'Name','input')
    
    sequenceFoldingLayer('Name','fold')
    
    convolution2dLayer(filterSize,numFilters,'Name','conv')
    batchNormalizationLayer('Name','bn')
    reluLayer('Name','relu')
    
    sequenceUnfoldingLayer('Name','unfold')
    flattenLayer('Name','flatten')
    
    lstmLayer(numHiddenUnits,'OutputMode','last','Name','lstm')
    
    fullyConnectedLayer(numClasses, 'Name','fc')
    softmaxLayer('Name','softmax')
    classificationLayer('Name','classification')];

lgraph = layerGraph(layers);
lgraph = connectLayers(lgraph,'fold/miniBatchSize','unfold/miniBatchSize');

numFeatures = 12;
numHiddenUnits1 = 125;
numHiddenUnits2 = 100;
numClasses = 9;
layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits1,'OutputMode','sequence')
    dropoutLayer(0.2)
    lstmLayer(numHiddenUnits2,'OutputMode','last')
    dropoutLayer(0.2)
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

numFeatures = 12;
numHiddenUnits1 = 125;
numHiddenUnits2 = 100;
numClasses = 9;
layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits1,'OutputMode','sequence')
    dropoutLayer(0.2)
    lstmLayer(numHiddenUnits2,'OutputMode','sequence')
    dropoutLayer(0.2)
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

Layers

Classification, Prediction, and Forecasting

To classify or make predictions on new data, use classify and predict.

LSTM neural networks can remember the state of the neural network between predictions. The RNN state is useful when you do not have the complete time series in advance, or if you want to make multiple predictions on a long time series.

To predict and classify on parts of a time series and update the RNN state, use predictAndUpdateState and classifyAndUpdateState. To reset the RNN state between predictions, use resetState.

For an example showing how to forecast future time steps of a sequence, see Time Series Forecasting Using Deep Learning.

Sequence Padding, Truncation, and Splitting

LSTM neural networks support input data with varying sequence lengths. When passing data through the neural network, the software pads, truncates, or splits sequences so that all the sequences in each mini-batch have the specified length. You can specify the sequence lengths and the value used to pad the sequences using the SequenceLength and SequencePaddingValue name-value pair arguments in trainingOptions.

After training the neural network, use the same mini-batch size and padding options when using the classify, predict, classifyAndUpdateState, predictAndUpdateState, and activations functions.

sequenceLengths = cellfun(@(X) size(X,2), XTrain);
[sequenceLengthsSorted,idx] = sort(sequenceLengths);
XTrain = XTrain(idx);

Normalize Sequence Data

To recenter training data automatically at training time using zero-center normalization, set the Normalization option of sequenceInputLayer to 'zerocenter'. Alternatively, you can normalize sequence data by first calculating the per-feature mean and standard deviation of all the sequences. Then, for each training observation, subtract the mean value and divide by the standard deviation.

mu = mean([XTrain{:}],2);
sigma = std([XTrain{:}],0,2);
XTrain = cellfun(@(X) (X-mu)./sigma,XTrain,'UniformOutput',false);

Out-of-Memory Data

Use datastores for sequence, time series, and signal data when data is too large to fit in memory or to perform specific operations when reading batches of data.

To learn more, see Train Network Using Out-of-Memory Sequence Data and Classify Out-of-Memory Text Data Using Deep Learning.

Visualization

Investigate and visualize the features learned by LSTM neural networks from sequence and time series data by extracting the activations using the activations function. To learn more, see Visualize Activations of LSTM Network.

LSTM Layer Architecture

This diagram illustrates the flow of data through an LSTM layer with input $$x$$ and output $$y$$ with T time steps. In the diagram, $$h_t$$ denotes the output (also known as the hidden state) and $$c_t$$ denotes the cell state at time step t.

If the layer outputs the full sequence, then it outputs $$y_1$$, …, $$y_T$$, which is equivalent to $$h_1$$, …, $$h_T$$. If the layer outputs the last time step only, then the layer outputs $$y_T$$, which is equivalent to $$h_T$$. The number of channels in the output matches the number of hidden units of the LSTM layer.

The first LSTM operation uses the initial state of the RNN and the first time step of the sequence to compute the first output and the updated cell state. At time step t, the operation uses the current state of the RNN $$(c_{t-1},h_{t-1})$$ and the next time step of the sequence to compute the output and the updated cell state $$c_t$$.

The state of the layer consists of the hidden state (also known as the output state) and the cell state. The hidden state at time step t contains the output of the LSTM layer for this time step. The cell state contains information learned from the previous time steps. At each time step, the layer adds information to or removes information from the cell state. The layer controls these updates using gates.

These components control the cell state and hidden state of the layer.

This diagram illustrates the flow of data at time step t. This diagram shows how the gates forget, update, and output the cell and hidden states.

The learnable weights of an LSTM layer are the input weights W (InputWeights), the recurrent weights R (RecurrentWeights), and the bias b (Bias). The matrices W, R, and b are concatenations of the input weights, the recurrent weights, and the bias of each component, respectively. The layer concatenates the matrices according to these equations:

where i, f, g, and o denote the input gate, forget gate, cell candidate, and output gate, respectively.

The cell state at time step t is given by

where $$\odot$$ denotes the Hadamard product (element-wise multiplication of vectors).

The hidden state at time step t is given by

where $${\sigma }_c$$ denotes the state activation function. By default, the lstmLayer function uses the hyperbolic tangent function (tanh) to compute the state activation function.

These formulas describe the components at time step t.

In these calculations, $${\sigma }_g$$ denotes the gate activation function. By default, the lstmLayer function, uses the sigmoid function, given by $$\sigma (x)={(1+e^{-x})}^{-1}$$, to compute the gate activation function.