Main Content

Workflow for Expected Shortfall (ES) Backtesting by Du and Escanciano

Open Live Script

This example shows the workflow for using the Du-Escanciano (DE) expected shortfall (ES) backtests and demonstrates a fixed test window for a single DE model with multiple VaR levels.

Load Data

The data in the ESBacktestDistributionData.mat file has returns, VaR and ES data, and distribution information for three models: normal, and t with 5 degrees of freedom and t with 10 degrees of freedom. The data spans multiple years from January 1996 to July 2003 and includes a total of 1966 observations.

This example uses a t distribution with 10 degrees of freedom and focuses on one year of data to show the difference between the critical value methods for large-sample approximation and simulation supported by the esbacktestbyde class.

load ESBacktestDistributionData.mat

TargetYear = 1998; % Change to test other calendar years
Ind = year(Dates)==TargetYear;
Dates = Dates(Ind);
Returns = Returns(Ind);
VaR = T10VaR(Ind,:);
ES = T10ES(Ind,:);
Mu = 0; % Always 0 in this data set
Sigma = T10Scale(Ind);

Plot Data

Plot the data for a VaR level of 0.975.

% Plot data
TargetVaRLevel = 0.975;
VaRInd = VaRLevel==TargetVaRLevel;

FailureInd = Returns<-VaR(:,VaRInd);

bar(Dates,Returns)
hold on
plot(Dates,-VaR(:,VaRInd),Dates,-ES(:,VaRInd))
plot(Dates(FailureInd),Returns(FailureInd),'.')
hold off
legend('Returns','VaR','ES','Location','best')
title(['Test Data, VaR Level ' num2str(TargetVaRLevel*100) '%'])
ylabel('Returns')
grid on

Figure contains an axes object. The axes object with title Test Data, VaR Level 97.5%, ylabel Returns contains 4 objects of type bar, line. One or more of the lines displays its values using only markers These objects represent Returns, VaR, ES.

Create an esbacktestbyde Object

Create an esbacktestbyde object to run the DE tests. Note that VaR and ES data are not required inputs because the DE tests work on "mapped returns" or "ranks" and perform mapping by using the distribution information. However, for convenience, the esbacktestbyde object computes the VaR and ES data internally using the distribution information and stores the data in the VaRData and ESData properties of the esbacktestbyde object. The VaR and ES data is used only to estimate the severity ratios reported by the summary function and are not used for any of the DE tests.

By default, when you create a esbacktestbyde object, a simulation runs and large-sample and simulation-based critical values are available immediately. Although the simulation processing is efficient, if you verify that large-sample approximation is appropriate for the sample size and VaR level under consideration, you can turn the simulation off to increase processing speed. To turn off the simulation, when using esbacktestbyde to create an esbacktestbtde object, set the name-value pair argument 'Simulate' to false. 

rng('default'); % For reproducibility
tic;
ebtde = esbacktestbyde(Returns,"t",...
'DegreesOfFreedom',10,...
'Location',Mu,...
'Scale',Sigma,...
'VaRLevel',VaRLevel,...
'PortfolioID',"S&P",...
'VaRID',"t(10)");
toc;
Elapsed time is 0.146348 seconds.

disp(ebtde)
  esbacktestbyde with properties:

    PortfolioData: [261x1 double]
          VaRData: [261x3 double]
           ESData: [261x3 double]
     Distribution: [1x1 struct]
      PortfolioID: "S&P"
            VaRID: ["t(10)"    "t(10)"    "t(10)"]
         VaRLevel: [0.9500 0.9750 0.9900]

disp(ebtde.Distribution)
                Name: "t"
    DegreesOfFreedom: 10
            Location: 0
               Scale: [261x1 double]

Summary Statistics

Use summary to return a basic expected shortfall (ES) report on failures and severity. This is the same summary output as the other ES backtesting classes esbacktest and esbacktestbysim. When the esbacktestbyde object is created, the VaR and ES data are computed using the distribution information. This information is stored in the VaRData and ESData properties. The summary function uses the VaRData and ESData properties to compute the observed severity ratio.

disp(summary(ebtde))
    PortfolioID     VaRID     VaRLevel    ObservedLevel    ExpectedSeverity    ObservedSeverity    Observations    Failures    Expected    Ratio     Missing
    ___________    _______    ________    _____________    ________________    ________________    ____________    ________    ________    ______    _______

       "S&P"       "t(10)"      0.95         0.94253            1.3288              1.5295             261            15        13.05      1.1494       0   
       "S&P"       "t(10)"     0.975         0.96935            1.2652              1.5269             261             8        6.525      1.2261       0   
       "S&P"       "t(10)"      0.99         0.98467            1.2169              1.5786             261             4         2.61      1.5326       0

Run Tests

Use runtests to run all expected shortfall (ES) backtests for esbacktestbyde object. The default critical value method is 'large-sample' or asymptotic approximation.

disp(runtests(ebtde))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    UnconditionalDE
    ___________    _______    ________    _____________    _______________

       "S&P"       "t(10)"      0.95         accept            accept     
       "S&P"       "t(10)"     0.975         accept            accept     
       "S&P"       "t(10)"      0.99         accept            accept

Run the tests with 'simulation' or finite-sample critical values.

disp(runtests(ebtde,'CriticalValueMethod','simulation'))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    UnconditionalDE
    ___________    _______    ________    _____________    _______________

       "S&P"       "t(10)"      0.95         accept            accept     
       "S&P"       "t(10)"     0.975         accept            accept     
       "S&P"       "t(10)"      0.99         accept            accept

The runtests function accepts the name-value pair argument 'ShowDetails' which includes extra columns in the output. Specifically, this output includes the critical value method used, number of lags, and test confidence level.

disp(runtests(ebtde,'CriticalValueMethod','simulation','ShowDetails',true))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    UnconditionalDE    CriticalValueMethod    NumLags    TestLevel
    ___________    _______    ________    _____________    _______________    ___________________    _______    _________

       "S&P"       "t(10)"      0.95         accept            accept            "simulation"           1         0.95   
       "S&P"       "t(10)"     0.975         accept            accept            "simulation"           1         0.95   
       "S&P"       "t(10)"      0.99         accept            accept            "simulation"           1         0.95

Unconditional DE Test Details

The unconditional DE test assesses the severity of the violations based on an evaluation of the observed average tail loss and determines whether the severity is consistent with the model assumptions. All the tests supported in the related classes esbacktest and esbacktestbysim are also severity tests.

To view the unconditional DE test details, use the unconditionalDE function. By default, this function uses the 'large-sample' critical value method.

disp(unconditionalDE(ebtde))
    PortfolioID     VaRID     VaRLevel    UnconditionalDE     PValue     TestStatistic     LowerCI     UpperCI     Observations    CriticalValueMethod    MeanLS      StdLS      Scenarios    TestLevel
    ___________    _______    ________    _______________    ________    _____________    _________    ________    ____________    ___________________    ______    _________    _________    _________

       "S&P"       "t(10)"      0.95          accept          0.31715      0.032842       0.0096343    0.040366        261           "large-sample"        0.025    0.0078398       NaN         0.95   
       "S&P"       "t(10)"     0.975          accept          0.32497      0.018009       0.0015295    0.023471        261           "large-sample"       0.0125    0.0055973       NaN         0.95   
       "S&P"       "t(10)"      0.99          accept         0.076391      0.011309               0    0.011978        261           "large-sample"        0.005    0.0035603       NaN         0.95

To compare the results of 'large-sample' to simulation-based critical values, use the name-value pair argument 'CriticalValueMethod'. In this example, the results of both critical value methods, including the confidence interval and the p-values, look similar.

disp(unconditionalDE(ebtde,'CriticalValueMethod','simulation'))
    PortfolioID     VaRID     VaRLevel    UnconditionalDE    PValue    TestStatistic     LowerCI     UpperCI     Observations    CriticalValueMethod    MeanLS    StdLS    Scenarios    TestLevel
    ___________    _______    ________    _______________    ______    _____________    _________    ________    ____________    ___________________    ______    _____    _________    _________

       "S&P"       "t(10)"      0.95          accept         0.326       0.032842        0.010859    0.041709        261            "simulation"         NaN       NaN       1000         0.95   
       "S&P"       "t(10)"     0.975          accept         0.336       0.018009       0.0032446    0.024657        261            "simulation"         NaN       NaN       1000         0.95   
       "S&P"       "t(10)"      0.99          accept         0.126       0.011309               0    0.013311        261            "simulation"         NaN       NaN       1000         0.95

You can visualize the 'simulation' and 'large-sample' distributions to assess whether the 'large-sample' approximation is accurate enough for the sample size and VaR level under consideration. The unconditionalDE function returns the 'simulated' test statistics as an optional output.

In this example, higher VaR levels cause a noticeable mismatch between the 'large-sample' and 'simulation' distributions. However, the confidence intervals and p-values are comparable.

% Choose VaR level
TargetVaRLevel = 0.975;
VaRInd = VaRLevel==TargetVaRLevel;

[~,s] = unconditionalDE(ebtde,'CriticalValueMethod','simulation');
histogram(s(VaRInd,:),'Normalization',"pdf")
hold on
t = unconditionalDE(ebtde,'CriticalValueMethod','large-sample');
Mu = t.MeanLS(VaRInd);
Sigma = t.StdLS(VaRInd);
MinValPlot = min(s(VaRInd,:))-0.001;
MaxValPlot = max(s(VaRInd,:))+0.001;
xLS = linspace(MinValPlot,MaxValPlot,101);
pdfLS = normpdf(xLS,Mu,Sigma);
plot(xLS,pdfLS)
hold off
legend({'Simulation','Large-Sample'})
Title = sprintf('UnconditionalDE Test Distribution\nVaR Level: %g%%, Sample Size = %d',VaRLevel(VaRInd)*100,t.Observations(VaRInd));
title(Title)

Figure contains an axes object. The axes object with title UnconditionalDE Test Distribution VaR Level: 97.5%, Sample Size = 261 contains 2 objects of type histogram, line. These objects represent Simulation, Large-Sample.

Conditional DE Test Details

The conditional DE test assesses whether there is evidence of autocorrelation in the tail losses.

Although the names are similar, the conditional DE test and the conditional test supported in esbacktestbysim are qualitatively different tests. The conditional Acerbi-Szekely test supported in esbacktestbysim tests the severity of the ES, conditional on whether the model passes a VaR test. The Acerbi-Szekely conditional test is a severity test, comparable to the tests supported in esbacktest, esbacktestbysim, and the unconditionalDE test.

However, the conditional DE test in esbacktestbyde is a test for independence across time periods.

To see the details of the conditional DE test results, use the conditionalDE function. By default, this function uses the 'large-sample' critical value method and tests for one lag (correlation with the previous time period).

disp(conditionalDE(ebtde))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    PValue     TestStatistic    CriticalValue    AutoCorrelation    Observations    CriticalValueMethod    NumLags    Scenarios    TestLevel
    ___________    _______    ________    _____________    _______    _____________    _____________    _______________    ____________    ___________________    _______    _________    _________

       "S&P"       "t(10)"      0.95         accept        0.45361        0.5616          3.8415            0.046387           261           "large-sample"          1          NaN         0.95   
       "S&P"       "t(10)"     0.975         accept        0.54189       0.37205          3.8415            0.037755           261           "large-sample"          1          NaN         0.95   
       "S&P"       "t(10)"      0.99         accept        0.87949      0.022989          3.8415          -0.0093851           261           "large-sample"          1          NaN         0.95

The results of the 'large-sample' critical value method, particularly the simulation critical values and p-values, differ substantially from the results of the 'simulation' critical value method.

The critical value is similar for a 95% VaR level, but the simulation-based critical value is much larger for higher VaR levels, especially for a 99% VaR. The autocorrelation is 1 for any sample without VaR failures. Therefore, the test statistic equals the number of observations for any scenario without VaR failures. For a 99% VaR level, scenarios without VaR failures are like; consequently, there is a mass point at the number of observations which appears as a long, heavy tail in the simulated distribution of the test statistic.

disp(conditionalDE(ebtde,'CriticalValueMethod','simulation'))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    PValue    TestStatistic    CriticalValue    AutoCorrelation    Observations    CriticalValueMethod    NumLags    Scenarios    TestLevel
    ___________    _______    ________    _____________    ______    _____________    _____________    _______________    ____________    ___________________    _______    _________    _________

       "S&P"       "t(10)"      0.95         accept        0.257         0.5616          3.6876            0.046387           261            "simulation"           1         1000         0.95   
       "S&P"       "t(10)"     0.975         accept        0.141        0.37205          5.3504            0.037755           261            "simulation"           1         1000         0.95   
       "S&P"       "t(10)"      0.99         accept        0.502       0.022989             261          -0.0093851           261            "simulation"           1         1000         0.95

You can visually compare the 'large-sample' and 'simulation' distributions. The conditionalDE function also returns the simulated test statistics as an optional output.

Notice that the tail of the distribution gets heavier as the VaR level increases.

% Choose VaR level
TargetVaRLevel = 0.975;
VaRInd = VaRLevel==TargetVaRLevel;

[t,s] = conditionalDE(ebtde,'CriticalValueMethod','simulation');
xLS = 0:0.01:20;
pdfLS = chi2pdf(xLS,t.NumLags(1));
histogram(s(VaRInd,:),'Normalization',"pdf")
hold on
plot(xLS,pdfLS)
hold off
ylim([0 0.01])
legend({'Simulation','Large-Sample'})
Title = sprintf('ConditionalDE Test Distribution\nVaR Level: %g%%, Sample Size = %d',VaRLevel(VaRInd)*100,t.Observations(VaRInd));
title(Title)

Figure contains an axes object. The axes object with title ConditionalDE Test Distribution VaR Level: 97.5%, Sample Size = 261 contains 2 objects of type histogram, line. These objects represent Simulation, Large-Sample.

Because the conditional DE test is based on autocorrelations, you can run the test for differing numbers of lags.

Run the conditional DE test for 2 lags. At a VaR level of 99%, the 'large-sample' critical value method rejects the model but the 'simulation' critical value method does not reject the model, with a p-value close to 10%. This shows that the 'simulation' distribution and the 'large-sample' approximation can lead to different results, depending on the sample size and VaR level.

disp(conditionalDE(ebtde,'NumLags',2,'CriticalValueMethod','large-sample'))
    PortfolioID     VaRID     VaRLevel    ConditionalDE      PValue      TestStatistic    CriticalValue    AutoCorrelation    Observations    CriticalValueMethod    NumLags    Scenarios    TestLevel
    ___________    _______    ________    _____________    __________    _____________    _____________    _______________    ____________    ___________________    _______    _________    _________

       "S&P"       "t(10)"      0.95         reject          0.015812        8.294           5.9915            0.17212            261           "large-sample"          2          NaN         0.95   
       "S&P"       "t(10)"     0.975         reject        0.00045758       15.379           5.9915            0.23979            261           "large-sample"          2          NaN         0.95   
       "S&P"       "t(10)"      0.99         reject        2.5771e-07       30.343           5.9915            0.34083            261           "large-sample"          2          NaN         0.95   

disp(conditionalDE(ebtde,'NumLags',2,'CriticalValueMethod','simulation'))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    PValue    TestStatistic    CriticalValue    AutoCorrelation    Observations    CriticalValueMethod    NumLags    Scenarios    TestLevel
    ___________    _______    ________    _____________    ______    _____________    _____________    _______________    ____________    ___________________    _______    _________    _________

       "S&P"       "t(10)"      0.95         reject         0.03         8.294           6.1397            0.17212            261            "simulation"           2         1000         0.95   
       "S&P"       "t(10)"     0.975         reject        0.019        15.379           9.3364            0.23979            261            "simulation"           2         1000         0.95   
       "S&P"       "t(10)"      0.99         accept        0.098        30.343              522            0.34083            261            "simulation"           2         1000         0.95

Running a New Simulation with simulate

If a p-value is near a rejection boundary, you can run a new simulation to request more scenarios to reduce a simulation error.

You can also run a new simulation to request a higher number of lags. By default, creating an esbacktestbyde object causes the simulation to run so that the simulation test results are available immediately. However, to avoid extra storage, only 5 lags are simulated. If you request more than 5 lags with the simulate function, the conditionalDE test function displays the following message:

No simulation results available for the number of lags requested. Call 'simulate' with the desired number of lags.

You first need to run a new simulation using esbacktestbyde and specify the number of lags to use for that simulation. Displaying the size of the esbacktestbyde object before and after the new simulation illustrates how simulating with more lags increases the amount of data stored in the esbacktestbyde object, as more simulated test statistics are stored with more lags.

% See bytes before new simulation, 5 lags stored
whos ebtde
  Name       Size             Bytes  Class             Attributes

  ebtde      1x1             164883  esbacktestbyde              

% Simulate 6 lags
rng('default'); % for reproducibility
ebtde = simulate(ebtde,'NumLags',6);

% See bytes after new simulation, 6 lags stored
whos ebtde
  Name       Size             Bytes  Class             Attributes

  ebtde      1x1             188891  esbacktestbyde

After you run a new simulation with esbacktestbyde that increases the number of lags to 6, the test results for conditionalDE are available for the 'simulation' method using 6 lags.

disp(conditionalDE(ebtde,'NumLags',6,'CriticalValueMethod','simulation'))
    PortfolioID     VaRID     VaRLevel    ConditionalDE    PValue    TestStatistic    CriticalValue    AutoCorrelation    Observations    CriticalValueMethod    NumLags    Scenarios    TestLevel
    ___________    _______    ________    _____________    ______    _____________    _____________    _______________    ____________    ___________________    _______    _________    _________

       "S&P"       "t(10)"      0.95         accept        0.136        9.5173           16.412           -0.022881           261            "simulation"           6         1000         0.95   
       "S&P"       "t(10)"     0.975         accept        0.086        15.854           21.299           -0.021864           261            "simulation"           6         1000         0.95   
       "S&P"       "t(10)"      0.99         accept        0.128        30.438             1566          -0.0096211           261            "simulation"           6         1000         0.95

Alternatively, the conditionalDE test results are always available for the 'large-sample' method for any number of lags.

disp(conditionalDE(ebtde,'NumLags',10,'CriticalValueMethod','large-sample'))
    PortfolioID     VaRID     VaRLevel    ConditionalDE      PValue      TestStatistic    CriticalValue    AutoCorrelation    Observations    CriticalValueMethod    NumLags    Scenarios    TestLevel
    ___________    _______    ________    _____________    __________    _____________    _____________    _______________    ____________    ___________________    _______    _________    _________

       "S&P"       "t(10)"      0.95         reject          0.018711       21.361           18.307             0.15415           261           "large-sample"         10          NaN         0.95   
       "S&P"       "t(10)"     0.975         accept          0.088587       16.406           18.307            0.027955           261           "large-sample"         10          NaN         0.95   
       "S&P"       "t(10)"      0.99         reject        0.00070234       30.526           18.307          -0.0092432           261           "large-sample"         10          NaN         0.95

See Also

| | |

Related Topics