Set Up Access

To work with remote data in Amazon S3, you must set up access first:

If your Amazon S3 location is authorized for public access, you do not need to set environment variables or configure authentication. For more information on how to configure public access, see Blocking public access to your Amazon S3 storage.

Amazon S3 also allows multiple users to access one account. For more information on Amazon S3 access, see AWS Identity and Access Management (IAM).

If you are using Parallel Computing Toolbox™, you must ensure the cluster has been configured to access S3 services. You can copy your client environment variables to the workers on a cluster by setting EnvironmentVariables in parpool, batch, createJob, or in the Cluster Profile Manager.

Read Data from Amazon S3

The following example shows how to use an ImageDatastore object to read a specified image from Amazon S3, and then display the image to screen.

setenv('AWS_ACCESS_KEY_ID', 'YOUR_AWS_ACCESS_KEY_ID'); 
setenv('AWS_SECRET_ACCESS_KEY', 'YOUR_AWS_SECRET_ACCESS_KEY');

ds = imageDatastore('s3://bucketname/image_datastore/jpegfiles', ...
  'IncludeSubfolders', true, 'LabelSource', 'foldernames'); 
img = ds.readimage(1);
imshow(img)

Write Data to Amazon S3

The following example shows how to use a tabularTextDatastore object to read tabular data from Amazon S3 into a tall array, preprocess it by removing missing entries and sorting, and then write it back to Amazon S3.

setenv('AWS_ACCESS_KEY_ID', 'YOUR_AWS_ACCESS_KEY_ID');
setenv('AWS_SECRET_ACCESS_KEY', 'YOUR_AWS_SECRET_ACCESS_KEY');

ds = tabularTextDatastore('s3://bucketname/dataset/airlinesmall.csv', ...
  'TreatAsMissing', 'NA', 'SelectedVariableNames', {'ArrDelay'});
tt = tall(ds);
tt = sortrows(rmmissing(tt));
write('s3://bucketname/preprocessedData/',tt);

To read your tall data back, use the datastore function.

ds = datastore('s3://bucketname/preprocessedData/');
tt = tall(ds);

Set Up Access

To work with remote data in Azure storage, you must set up access first:

If your Azure storage location is authorized for public access, you do not need to set environment variables or configure authentication. For more information on public access to Azure storage, see Configure anonymous public read access.

Azure storage also allows multiple users to access one account. For more information on managing multiple users, see Azure Active Directory Identity and access management operations reference guide.

If you are using Parallel Computing Toolbox, you must copy your client environment variables to the workers on a cluster by setting EnvironmentVariables in parpool, batch, createJob, or in the Cluster Profile Manager.

For more information, see Use Azure storage with Azure HDInsight clusters.

Read Data from Azure

To read data from an Azure Blob Storage location, specify the location using the following syntax:

wasbs://container@account/path_to_file/file.ext

container@account is the name of the container and path_to_file is the path to the file or folders.

For example, if you have a file airlinesmall.csv in a folder /airline on a test storage account wasbs://blobContainer@storageAccount.blob.core.windows.net/, then you can create a datastore by using:

location = 'wasbs://blobContainer@storageAccount.blob.core.windows.net/airline/airlinesmall.csv';

ds = tabularTextDatastore(location, 'TreatAsMissing', 'NA', ...
  'SelectedVariableNames', {'ArrDelay'});

You can use Azure for all calculations datastores support, including direct reading, mapreduce, tall arrays and deep learning. For example, create an ImageDatastore object, read a specified image from the datastore, and then display the image to screen.

setenv('MW_WASB_SAS_TOKEN', 'YOUR_WASB_SAS_TOKEN');  
ds = imageDatastore('wasbs://YourContainer@YourAccount.blob.core.windows.net/', ...
  'IncludeSubfolders', true, 'LabelSource', 'foldernames'); 
img = ds.readimage(1);
imshow(img)

Write Data to Azure

This example shows how to read tabular data from Azure into a tall array using a tabularTextDatastore object, preprocess it by removing missing entries and sorting, and then write it back to Azure.

setenv('MW_WASB_SAS_TOKEN', 'YOUR_WASB_SAS_TOKEN');  

ds = tabularTextDatastore('wasbs://YourContainer@YourAccount.blob.core.windows.net/dataset/airlinesmall.csv', ...
  'TreatAsMissing', 'NA', 'SelectedVariableNames', {'ArrDelay'});
tt = tall(ds);
tt = sortrows(rmmissing(tt));
write('wasbs://YourContainer@YourAccount.blob.core.windows.net/preprocessedData/',tt);

To read your tall data back, use the datastore function.

ds = datastore('wasbs://YourContainer@YourAccount.blob.core.windows.net/preprocessedData/');
tt = tall(ds);

Specify Location of Data

MATLAB lets you use Hadoop Distributed File System (HDFS) as an online file storage web service. When you specify the location of the data, you must specify the full path to the files or folders using a uniform resource locator (URL) of one of these forms:

hdfs:/path_to_file

hdfs:///path_to_file

hdfs://hostname/path_to_file

hostname is the name of the host or server and path_to_file is the path to the file or folders. Specifying the hostname is optional. When you do not specify the hostname, Hadoop uses the default host name associated with the Hadoop Distributed File System (HDFS) installation in MATLAB.

For example, you can use either of these commands to create a datastore for the file, file1.txt, in a folder named data located at a host named myserver:

If hostname is specified, it must correspond to the namenode defined by the fs.default.name property in the Hadoop XML configuration files for your Hadoop cluster.

Optionally, you can include the port number. For example, this location specifies a host named myserver with port 7867, containing the file file1.txt in a folder named data:

'hdfs://myserver:7867/data/file1.txt'

The specified port number must match the port number set in your HDFS configuration.

Set Hadoop Environment Variable

Before reading from HDFS, use the setenv function to set the appropriate environment variable to the folder where Hadoop is installed. This folder must be accessible from the current machine.

For example, use this command to set the HADOOP_HOME environment variable. hadoop-folder is the folder where Hadoop is installed, and /mypath/ is the path to that folder.

setenv('HADOOP_HOME','/mypath/hadoop-folder');

HDFS data on Hortonworks or CLOUDERA

If your current machine has access to HDFS data on Hortonworks or CLOUDERA^®, then you do not have to set the HADOOP_HOME or HADOOP_PREFIX environment variables. MATLAB automatically assigns these environment variables when using Hortonworks or CLOUDERA application edge nodes.

Prevent Clearing Code from Memory

When reading from HDFS or when reading Sequence files locally, the datastore function calls the javaaddpath command. This command does the following:

To prevent persistent variables, code files, or MEX-files from being cleared, use the mlock function.

Write Data to HDFS

This example shows how to use a tabularTextDatastore object to write data to an HDFS location. Use the write function to write your tall and distributed arrays to a Hadoop Distributed File System. When you call this function on a distributed or tall array, you must specify the full path to a HDFS folder. The following example shows how to read tabular data from HDFS into a tall array, preprocess it by removing missing entries and sorting, and then write it back to HDFS.

ds = tabularTextDatastore('hdfs://myserver/some/path/dataset/airlinesmall.csv', ...
  'TreatAsMissing', 'NA', 'SelectedVariableNames', {'ArrDelay'});
tt = tall(ds);
tt = sortrows(rmmissing(tt));
write('hdfs://myserver/some/path/preprocessedData/',tt);

To read your tall data back, use the datastore function.

ds = datastore('hdfs://myserver/some/path/preprocessedData/');
tt = tall(ds);