Freightos Performs Big Data Analytics for Online Freight Logistics with MATLAB and Google BigQuery
Challenge
Operate and market a software-as-a-service solution for automating online freight sales
Solution
Use MATLAB to gain operational and marketing insights by performing advanced analytics on big data stored in the cloud
Results
- Analyses completed in minutes instead of hours
- BigQuery integration rapidly implemented
- Insights acquired 20 times faster
“With MATLAB and BigQuery integration, we’re not downloading our full database or overwhelming Microsoft® Excel® with complex operations. That means better reporting, better analytics, and better insights for our operations and marketing teams, all with less time wasted and higher accuracy.”
Treemap generated by Statistics and Machine Learning Toolbox, showing the most common ports of origin for U.S. imports.
Freightos has developed an online freight marketplace, backed by a powerful online freight routing and pricing system that replaces cumbersome manual processes and tools, eliminating many of the inefficiencies and errors that contribute to an estimated $650 million in losses each year for the industry. The Freightos software-as-a-service (SaaS) system uses Google® BigQuery to manage and store multiple databases for thousands of freight contracts, millions of freight quotes, and a wide array of other shipping data from some of the world’s largest logistics providers.
To gain operational and marketing insights from its BigQuery data, Freightos relies on big data analytics with MATLAB®.
“MATLAB enables us to slice and dice our data to gain insights that guide business decisions, help our customers, and drive sales,” says Eytan Buchman, marketing manager at Freightos. “With MATLAB, we are maximizing the benefits of the large amount of data we have in BigQuery.”
Challenge
With millions of rows of data spanning multiple databases in the cloud, Freightos analysts found running analytics to be cumbersome, making key insights that were critical for both operational and marketing tasks difficult to identify. Although they could run queries on the data in the cloud for static reports, to gain insights they needed to interactively explore and analyze real-time data. For example, to understand user behavior, they wanted to quantify users’ level of engagement and patterns of usage with the software. Like most big data sets, the Freightos data set was all but impossible to download for local processing.
Within the freight industry, routes and prices can change on an hourly basis, as supply and demand fluctuate and fuel costs vary. To keep pace with these changes, Freightos needed near-real-time performance. In addition, data from external sources varies in format and quality. Freightos needed to efficiently access subsets of their BigQuery data, and then rapidly clean the data, run advanced analytics, and visualize the results.
Solution
Freightos analysts worked with bat365 Consulting Services to integrate MATLAB with Google BigQuery for big data analytics.
Together they developed an automated approach for extracting data from BigQuery and importing it into MATLAB. In this approach, the results from queries performed on BigQuery are exported to Google Cloud Storage and then downloaded and accessed in MATLAB.
Because Freightos relies on external sources, including external data providers, third-party carriers, and freight forwarders, for much of its data, analysts must clean the query results before analyzing them. Working in MATLAB, the team developed scripts that identify and correct outliers in the data.
They used Statistics and Machine Learning Toolbox™ to perform complex statistical analysis of the cleaned data. One analysis evaluated 120,000 rows of freight quotes, each with more than 30 columns, to identify fluctuations in freight pricing based on different sales teams, companies, and shipping modes. Another analysis, driven by the marketing department, explored the most common origins and destinations, automatically plotting them into a treemap.
Working in MATLAB, Freightos and bat365 consultants generated scatter plots, treemaps, and other visualizations to better understand the results of the analysis. They also used the analytics results to create reports that help Freightos C-level executives make decisions and set marketing strategies.
Freightos is currently using MATLAB and BigQuery to develop a dynamic index based on the most up-to-date rates and routes in the BigQuery data. This index will enable them to accurately estimate shipping costs from major cities in China to major cities in North America.
Freightos plans to use machine learning algorithms from Statistics and Machine Learning Toolbox to project future index prices.
Results
- Analyses completed in minutes instead of hours. “With our MATLAB and BigQuery integration, even large reports, such as quarterly reports on an entire data pool, now take only 10 minutes,” says Leonid Hatskin, data analyst at Freightos. “In the past this would have taken at least a full work day. MATLAB has freed up at least one workday per month of our data team’s time.”
- BigQuery integration rapidly implemented. “When we asked bat365 Consulting for assistance with BigQuery integration, we got real action from a real person really quickly,” notes Buchman. “That support turned MATLAB from powerful into irreplaceable for us.”
- Insights acquired 20 times faster. “Compared with our previous approach, our time-to-insight improved 20-fold with MATLAB and BigQuery,” says Buchman. “We can’t put a price tag on the time-to-market improvement that enables.”
