Work with Big Data for Simulations
Simulation of models with many time steps and signals can involve big data that is too large to fit into the RAM of your computer. Such situations include:
Logging simulation data (signal logging, output port logging, and state logging)
Loading input signal data for simulating a model
Running multiple or parallel simulations
To work with big data for simulations, store the data to persistent storage in a MAT-file. Using big data techniques for simulations requires additional steps beyond what you do when the data is small enough to fit in workspace memory. As you develop a model, consider logging and loading simulation data without using persistent storage unless you discover that your model has big data requirements that overload memory.
Big Data Workflow
This example is a high-level workflow for handling big data that one simulation produces and that another simulation uses as input. For more detailed information about the major workflow tasks, see:
Tip
This example uses a SimulationDatastore
object for streaming data
into a model. Alternatively, you can stream a DatasetRef
object
directly into a model.
Configure two models to log several signals.
Simulate the models, logging the data to persistent storage for each model.
sim(mdl1,'LoggingToFile','on','LoggingFileName','data1.mat'); sim(mdl2,'LoggingToFile','on','LoggingFileName','data2.mat');
Logging that involves big data requires saving the data to persistent storage as a v7.3 MAT-file. Only the data logged in
Dataset
format is saved to the file. Data logged in other formats, such asStructure with time
, is saved in memory, in the base workspace.The data that you log to persistent storage is streamed during the simulation in small chunks, to minimize memory requirements. The data is stored in a file that contains
Dataset
objects for each set of logged data (for example,logsout
andxout
).Create
DatasetRef
objects (dsr1
anddsr2
) for specific sets of logged signals. Then createSimulationDatastore
objects (dst1
anddst2
) for values of elements of theDatasetRef
objects. This example code creates aSimulationDatastore
for the 12th element oflogsout
for the first simulation. For the second simulation, the example code creates a signal with values being aSimulationDatastore
object for the seventh element oflogsout
. You can use curly braces for indexing.dsr1 = Simulink.SimulationData.DatasetRef('data1.mat','logsout'); dsr2 = Simulink.SimulationData.DatasetRef('data2.mat','logsout'); dst1 = dsr1{12}; dst2 = dsr2{7};
Use
SimulationDatastore
objects as an external input for another simulation. To load theSimulationDatastore
data, include it in aDataset
object. The datastore input is incrementally loaded from the MAT-file. The third input is atimeseries
object, which is loaded into memory as a whole, not incrementally.input = Simulink.SimulationData.Dataset; input{1} = dst1; input{2} = dst2; ts = timeseries(rand(5,1),1,'Name','RandomSignals'); input{3} = ts; sim(mdl3,'ExternalInput','input');
Use MATLAB® big data analysis to work with the
SimulationDatastore
objects. Create atimetable
object by reading the values of aSimulationDatastore
object. Theread
function reads a portion of the data. Thereadall
function reads all the data.tt = dst1.Values.read;
Set the MATLAB session as the global execution environment (
mapreducer
) for working with the talltimetable
. Create a talltimetable
from aSimulationDatastore
object and read atimetable
object with in-memory data.mapreducer(0); ttt = tall(dst1.Values);
Tip
For another example showing how to work with big simulation data, see Working with Big Data.
See Also
Functions
Simulink.SimulationData.Dataset
|timeseries
|Simulink.SimulationData.DatasetRef
|matlab.io.datastore.SimulationDatastore