NARX Network

In the first type of time series problem, you would like to predict future values of a time series y(t) from past values of that time series and past values of a second time series x(t). This form of prediction is called nonlinear autoregressive with exogenous (external) input, or NARX (see Design Time Series NARX Feedback Neural Networks), and can be written as follows:

Use this model to predict future values of a stock or bond, based on such economic variables as unemployment rates, GDP, etc. You can also use this model for system identification, in which models are developed to represent dynamic systems, such as chemical processes, manufacturing systems, robotics, aerospace vehicles, etc.

NAR Network

In the second type of time series problem, there is only one series involved. The future values of a time series y(t) are predicted only from past values of that series. This form of prediction is called nonlinear autoregressive, or NAR, and can be written as follows:

You can use this model to predict financial instruments, but without the use of a companion series.

Nonlinear Input-Output Network

The third time series problem is similar to the first type, in that two series are involved, an input series x(t) and an output series y(t). Here you want to predict values of y(t) from previous values of x(t), but without knowledge of previous values of y(t). This input/output model can be written as follows:

The NARX model will provide better predictions than this input-output model, because it uses the additional information contained in the previous values of y(t). However, there may be some applications in which the previous values of y(t) would not be available. Those are the only cases where you would want to use the input-output model instead of the NARX model.

Select Data

The script assumes that the predictor and response vectors are already loaded into the workspace. If the data is not loaded, you can load it as follows:

load ph_dataset

This data set is one of the sample data sets that is part of the toolbox. For information about the data sets available, see Sample Data Sets for Shallow Neural Networks. You can also see a list of all available data sets by entering the command help nndatasets. You can load the variables from any of these data sets using your own variable names. For example, the command

[X,T] = ph_dataset;

Choose Training Algorithm

Define training algorithm. The network uses the default Levenberg-Marquardt algorithm (trainlm) for training.

trainFcn = 'trainlm';  % Levenberg-Marquardt backpropagation.

For problems in which Levenberg-Marquardt does not produce as accurate results as desired, or for large data problems, consider setting the network training function to Bayesian Regularization (trainbr) or Scaled Conjugate Gradient (trainscg), respectively, with either

net.trainFcn = 'trainbr';
net.trainFcn = 'trainscg';

Create Network

Create a network. The NARX network, narxnet, is a feedforward network with the default tan-sigmoid transfer function in the hidden layer and linear transfer function in the output layer. This network has two inputs. One is an external input, and the other is a feedback connection from the network output. After the network has been trained, this feedback connection can be closed, as you will see at a later step. For each of these inputs, there is a tapped delay line to store previous values. To assign the network architecture for a NARX network, you must select the delays associated with each tapped delay line, and also the number of hidden layer neurons. In the following steps, you assign the input delays and the feedback delays to range from 1 to 4 and the number of hidden neurons to be 10.

inputDelays = 1:4;
feedbackDelays = 1:4;
hiddenLayerSize = 10;
net = narxnet(inputDelays,feedbackDelays,hiddenLayerSize,'open',trainFcn);

Prepare Data for Training

Prepare the data for training. When training a network containing tapped delay lines, it is necessary to fill the delays with initial values of the predictors and responses of the network. There is a toolbox command that facilitates this process - preparets. This function has three input arguments: the network, the predictors, and the responses. The function returns the initial conditions that are needed to fill the tapped delay lines in the network, and modified predictor and response sequences, where the initial conditions have been removed. You can call the function as follows:

[x,xi,ai,t] = preparets(net,X,{},T);

Divide Data

Set up the division of data.

net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;

With these settings, the data will be randomly divided, with 70% used for training, 15% for validation, and 15% for testing.

Train Network

Train the network.

[net,tr] = train(net,x,t,xi,ai);

During training, the following training window opens. This window displays the training progress and allows you to interrupt training at any point by clicking the stop button .

This training stopped when the validation error increased consecutively for six iterations.

Test Network

Test the network. After the network has been trained, you can use it to compute the network outputs. The following code calculates the network outputs, errors, and overall performance. Note that to simulate a network with tapped delay lines, you need to assign the initial values for these delayed signals. This is done with the input states (xi) and the layer states (ai) provided by preparets at an earlier stage.

y = net(x,xi,ai);
e = gsubtract(t,y);
performance = perform(net,t,y)

performance =

    0.0042

View Network

View the network diagram.

view(net)

Analyze Results

Plot the performance training record to check for potential overfitting.

figure, plotperform(tr)

This figure shows that training and validation errors decrease until the highlighted epoch. It does not appear that any overfitting has occurred, because the validation error does not increase before this epoch.

All of the training is done in open loop (also called series-parallel architecture), including the validation and testing steps. The typical workflow is to fully create the network in open loop, and only when it has been trained (which includes validation and testing steps) it is transformed to closed loop for multistep-ahead prediction. Likewise, the R values in the Neural Net Times Series app are computed based on the open-loop training results.

Closed Loop Network

Close the loop on the NARX network. When the feedback loop is open on the NARX network, it is performing a one-step-ahead prediction. It is predicting the next value of y(t) from previous values of y(t) and x(t). With the feedback loop closed, it can be used to perform multi-step-ahead predictions. This is because predictions of y(t) will be used in place of actual future values of y(t). The following commands can be used to close the loop and calculate closed-loop performance

netc = closeloop(net);
netc.name = [net.name ' - Closed Loop'];
view(netc)
[xc,xic,aic,tc] = preparets(netc,X,{},T);
yc = netc(xc,xic,aic);
closedLoopPerformance = perform(net,tc,yc)

closedLoopPerformance =

    0.4014

Step-Ahead Prediction Network

Remove a delay from the network, to get the prediction one time step early.

nets = removedelay(net);
nets.name = [net.name ' - Predict One Step Ahead'];
view(nets)
[xs,xis,ais,ts] = preparets(nets,X,{},T);
ys = nets(xs,xis,ais);
stepAheadPerformance = perform(nets,ts,ys)

stepAheadPerformance =

    0.0042

From this figure, you can see that the network is identical to the previous open-loop network, except that one delay has been removed from each of the tapped delay lines. The output of the network is then y(t + 1) instead of y(t). This may sometimes be helpful when a network is deployed for certain applications.

Next Steps

If the network performance is not satisfactory, you could try any of these approaches:

To get more experience in command-line operations, try some of these tasks:

Each time a neural network is trained can result in a different solution due to random initial weight and bias values and different divisions of data into training, validation, and test sets. As a result, different neural networks trained on the same problem can give different outputs for the same input. To ensure that a neural network of good accuracy has been found, retrain several times.

There are several other techniques for improving upon initial solutions if higher accuracy is desired. For more information, see Improve Shallow Neural Network Generalization and Avoid Overfitting.