To implement a custom training loop for your network, first convert it to a dlnetwork object. Do not include output layers in a dlnetwork object. Instead, you must specify the loss function in the custom training loop.

Load a pretrained GoogLeNet model using the googlenet function. This function requires the Deep Learning Toolbox™ Model for GoogLeNet Network support package. If this support package is not installed, then the function provides a download link.

net = googlenet;

Convert the network to a layer graph and remove the layers used for classification using removeLayers.

lgraph = layerGraph(net);
lgraph = removeLayers(lgraph,["prob" "output"]);

Convert the network to a dlnetwork object.

dlnet = dlnetwork(lgraph)

dlnet = 
  dlnetwork with properties:

         Layers: [142x1 nnet.cnn.layer.Layer]
    Connections: [168x2 table]
     Learnables: [116x3 table]
          State: [0x3 table]
     InputNames: {'data'}
    OutputNames: {'loss3-classifier'}
    Initialized: 1

  View summary with summary.

Use network data layout objects to create a multi-input dlnetwork that is ready for training. The software uses the size and format information to determine the appropriate sizes and formats of the learnable and state parameters of the dlnetwork.

Define the network architecture. Construct a network with two branches. The network takes two inputs, with one input per branch. Connect the branches using an addition layer.

numFilters = 24;

layersBranch1 = [
    convolution2dLayer(3,6*numFilters,Padding="same",Stride=2)
    groupNormalizationLayer("all-channels")
    reluLayer
    convolution2dLayer(3,numFilters,Padding="same")
    groupNormalizationLayer("channel-wise")
    additionLayer(2,Name="add")
    reluLayer
    fullyConnectedLayer(10)
    softmaxLayer];

layersBranch2 = [
    convolution2dLayer(1,numFilters)
    groupNormalizationLayer("all-channels",Name="gnBranch2")];

lgraph = layerGraph(layersBranch1);
lgraph = addLayers(lgraph,layersBranch2);
lgraph = connectLayers(lgraph,"gnBranch2","add/in2");

Create network data layout objects that represent the size and format of typical network inputs. For both inputs, use a batch size of 32. Use an input of size 64-by-64 with three channels for the convolution layer in the first branch. Use an input of size 32-by-32 with 18 channels for the convolution layer in the second branch.

X1 = dlarray(rand([64 64 3 32]),"SSCB");
X2 = dlarray(rand([32 32 18 32]),"SSCB");

Create the dlnetwork. Provide the inputs in the same order that the unconnected layers appear in the Layers property of lgraph.

net = dlnetwork(lgraph,X1,X2);

Check that the network is initialized and ready for training by inspecting the Initialized property of the network.

net.Initialized

ans = logical
   1

This example shows how to train a network that classifies handwritten digits with a custom learning rate schedule.

You can train most types of neural networks using the trainNetwork and trainingOptions functions. If the trainingOptions function does not provide the options you need (for example, a custom learning rate schedule), then you can define your own custom training loop using dlarray and dlnetwork objects for automatic differentiation. For an example showing how to retrain a pretrained deep learning network using the trainNetwork function, see Transfer Learning Using Pretrained Network.

Training a deep neural network is an optimization task. By considering a neural network as a function $$f(X;\theta )$$, where $$X$$ is the network input, and $$\theta$$ is the set of learnable parameters, you can optimize $$\theta$$ so that it minimizes some loss value based on the training data. For example, optimize the learnable parameters $$\theta$$ such that for a given inputs $$X$$ with a corresponding targets $$T$$, they minimize the error between the predictions $$Y=f(X;\theta )$$ and $$T$$.

The loss function used depends on the type of task. For example:

You can optimize the objective using gradient descent: minimize the loss $$L$$ by iteratively updating the learnable parameters $$\theta$$ by taking steps towards the minimum using the gradients of the loss with respect to the learnable parameters. Gradient descent algorithms typically update the learnable parameters by using a variant of an update step of the form $${\theta }_{t+1}={\theta }_t-\rho \nabla L$$, where $$t$$ is the iteration number, $$\rho$$ is the learning rate, and $$\nabla L$$ denotes the gradients (the derivatives of the loss with respect to the learnable parameters).

This example trains a network to classify handwritten digits with the time-based decay learning rate schedule: for each iteration, the solver uses the learning rate given by ${\rho }_{\mathit{t}}=\frac{{\rho }_0}{1+\mathit{k}\mathit{t}}$, where t is the iteration number, $${\rho }_0$$ is the initial learning rate, and k is the decay.

unzip("DigitsData.zip")

imds = imageDatastore("DigitsData", ...
    IncludeSubfolders=true, ...
    LabelSource="foldernames");

[imdsTrain,imdsValidation] = splitEachLabel(imds,0.9,"randomize");

inputSize = [28 28 1];
pixelRange = [-5 5];

imageAugmenter = imageDataAugmenter( ...
    RandXTranslation=pixelRange, ...
    RandYTranslation=pixelRange);

augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain,DataAugmentation=imageAugmenter);

augimdsValidation = augmentedImageDatastore(inputSize(1:2),imdsValidation);

classes = categories(imdsTrain.Labels);
numClasses = numel(classes);

layers = [
    imageInputLayer(inputSize,Normalization="none")
    convolution2dLayer(5,20,Padding="same")
    batchNormalizationLayer
    reluLayer
    convolution2dLayer(3,20,Padding="same")
    batchNormalizationLayer
    reluLayer
    convolution2dLayer(3,20,Padding="same")
    batchNormalizationLayer
    reluLayer
    fullyConnectedLayer(numClasses)
    softmaxLayer];

net = dlnetwork(layers)

net = 
  dlnetwork with properties:

         Layers: [12×1 nnet.cnn.layer.Layer]
    Connections: [11×2 table]
     Learnables: [14×3 table]
          State: [6×3 table]
     InputNames: {'imageinput'}
    OutputNames: {'softmax'}
    Initialized: 1

  View summary with summary.

numEpochs = 10;
miniBatchSize = 128;

initialLearnRate = 0.01;
decay = 0.01;
momentum = 0.9;

mbq = minibatchqueue(augimdsTrain,...
    MiniBatchSize=miniBatchSize,...
    MiniBatchFcn=@preprocessMiniBatch,...
    MiniBatchFormat=["SSCB" ""], ...
    PartialMiniBatch="discard");

velocity = [];

numObservationsTrain = numel(imdsTrain.Files);
numIterationsPerEpoch = floor(numObservationsTrain / miniBatchSize);
numIterations = numEpochs * numIterationsPerEpoch;

monitor = trainingProgressMonitor( ...
    Metrics="Loss", ...
    Info=["Epoch" "LearnRate"], ...
    XLabel="Iteration");

epoch = 0;
iteration = 0;

% Loop over epochs.
while epoch < numEpochs && ~monitor.Stop
    
    epoch = epoch + 1;

    % Shuffle data.
    shuffle(mbq);
    
    % Loop over mini-batches.
    while hasdata(mbq) && ~monitor.Stop

        iteration = iteration + 1;
        
        % Read mini-batch of data.
        [X,T] = next(mbq);
        
        % Evaluate the model gradients, state, and loss using dlfeval and the
        % modelLoss function and update the network state.
        [loss,gradients,state] = dlfeval(@modelLoss,net,X,T);
        net.State = state;
        
        % Determine learning rate for time-based decay learning rate schedule.
        learnRate = initialLearnRate/(1 + decay*iteration);
        
        % Update the network parameters using the SGDM optimizer.
        [net,velocity] = sgdmupdate(net,gradients,velocity,learnRate,momentum);
        
        % Update the training progress monitor.
        recordMetrics(monitor,iteration,Loss=loss);
        updateInfo(monitor,Epoch=epoch,LearnRate=learnRate);
        monitor.Progress = 100 * iteration/numIterations;
    end
end

numOutputs = 1;

mbqTest = minibatchqueue(augimdsValidation,numOutputs, ...
    MiniBatchSize=miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatchPredictors, ...
    MiniBatchFormat="SSCB");

YTest = modelPredictions(net,mbqTest,classes);

TTest = imdsValidation.Labels;
accuracy = mean(TTest == YTest)

accuracy = 0.9220

figure
confusionchart(TTest,YTest)

function [loss,gradients,state] = modelLoss(net,X,T)

% Forward data through network.
[Y,state] = forward(net,X);

% Calculate cross-entropy loss.
loss = crossentropy(Y,T);

% Calculate gradients of loss with respect to learnable parameters.
gradients = dlgradient(loss,net.Learnables);

end

function Y = modelPredictions(net,mbq,classes)

Y = [];

% Loop over mini-batches.
while hasdata(mbq)
    X = next(mbq);

    % Make prediction.
    scores = predict(net,X);

    % Decode labels and append to output.
    labels = onehotdecode(scores,classes,1)';
    Y = [Y; labels];
end

end

function [X,T] = preprocessMiniBatch(dataX,dataT)

% Preprocess predictors.
X = preprocessMiniBatchPredictors(dataX);

% Extract label data from cell and concatenate.
T = cat(2,dataT{1:end});

% One-hot encode labels.
T = onehotencode(T,1);

end

function X = preprocessMiniBatchPredictors(dataX)

% Concatenate.
X = cat(4,dataX{1:end});

end

Load a pretrained network.

net = squeezenet;

Convert the network to a layer graph, remove the output layer, and convert it to a dlnetwork object.

lgraph = layerGraph(net);
lgraph = removeLayers(lgraph,'ClassificationLayer_predictions');
dlnet = dlnetwork(lgraph);

The Learnables property of the dlnetwork object is a table that contains the learnable parameters of the network. The table includes parameters of nested layers in separate rows. View the first few rows of the learnables table.

learnables = dlnet.Learnables;
head(learnables)

          Layer           Parameter           Value       
    __________________    _________    ___________________

    "conv1"               "Weights"    {3x3x3x64  dlarray}
    "conv1"               "Bias"       {1x1x64    dlarray}
    "fire2-squeeze1x1"    "Weights"    {1x1x64x16 dlarray}
    "fire2-squeeze1x1"    "Bias"       {1x1x16    dlarray}
    "fire2-expand1x1"     "Weights"    {1x1x16x64 dlarray}
    "fire2-expand1x1"     "Bias"       {1x1x64    dlarray}
    "fire2-expand3x3"     "Weights"    {3x3x16x64 dlarray}
    "fire2-expand3x3"     "Bias"       {1x1x64    dlarray}

To freeze the learnable parameters of the network, loop over the learnable parameters and set the learn rate to 0 using the setLearnRateFactor function.

factor = 0;

numLearnables = size(learnables,1);
for i = 1:numLearnables
    layerName = learnables.Layer(i);
    parameterName = learnables.Parameter(i);
    
    dlnet = setLearnRateFactor(dlnet,layerName,parameterName,factor);
end

To use the updated learn rate factors when training, you must pass the dlnetwork object to the update function in the custom training loop. For example, use the command

[dlnet,velocity] = sgdmupdate(dlnet,gradients,velocity);

Create an uninitialized dlnetwork object without an input layer. Creating an uninitialized dlnetwork is useful when you do not yet know the size and format of the network inputs, for example, when the dlnetwork is nested inside a custom layer.

Define the network layers. This network has a single input, which is not connected to an input layer.

layers = [
    convolution2dLayer(5,20)
    batchNormalizationLayer
    reluLayer
    fullyConnectedLayer(10)
    softmaxLayer];

Create an uninitialized dlnetwork. Set the Initialize option to false.

dlnet = dlnetwork(layers,'Initialize',false);

Check that the network is not initialized.

dlnet.Initialized

ans = logical
   0

The learnable and state parameters of this network are not initialized for training. To initialize the network, use the initialize function.

If you want to use dlnet directly in a custom training loop, then you can initialize it by using the initialize function and providing an example input.

If you want to use dlnet inside a custom layer, then you can take advantage of automatic initialization. If you use the custom layer inside a dlnetwork, then dlnet is initialized when the parent dlnetwork is constructed (or when the parent network is initialized if it is constructed as an uninitialized dlnetwork). If you use the custom layer inside a network that is trained using the trainNetwork function, then dlnet is automatically initialized at training time. For more information, see Deep Learning Network Composition.