In the past, engineers download terabyte-sized ADAS datasets to look for edge cases. This approach consumes huge amount of network bandwidth and local storage space. We created a new and more efficient way, which utilizes MATLAB to access Apache Spark resources to decode, analyze data, and search for edge cases right on the Hadoop file system. It dramatically improves throughput and reduces the amount of data downloaded to the engineer’s workstation.

This approach was successfully used to analyze ADAS feature usage from the CAN traffic on Ford’s Big-Data-Drive fleet of vehicles. It will be deployed for all future Big-Data-Drive vehicle analysis.