Delivering 5D Perception on-the-edge®: VizioR&I™ Imaging Radar with embedded NVIDIA DRIVE Orin™

Introduction

The VizioR&I™ platform from Provizio is not just another radar system; it represents a transformative leap in radar based perception technology. By integrating a robust mmWave radar sensor with an embedded GPU, VizioR&I™ delivers unparalleled performance while optimising size, weight, power, and cost in a single, scalable device. This innovation dramatically simplifies the perception challenge for OEMs, replacing complex, multi-component development stages and data processing steps with a single, streamlined solution.

The unique combination of proprietary hardware and software within VizioR&I™ provides unprecedented computational flexibility, enabling robust perception capabilities that surpass the state-of-the-art not just for radar, but also for LiDAR and camera technologies. The immense level of performance delivered by the embedded GPU, coupled with a streamlined plug & play design, make it an invaluable tool across various industries, from automotive and agriculture to mining and smart cities.

Why Perception On-The-Edge?

In the context of autonomous systems, perception refers to a machine's ability to interpret sensory data from its surroundings and make informed decisions. This involves detecting, classifying, and tracking objects, understanding spatial relationships, and anticipating potential hazards. However, developing a unified perception system that addresses all these tasks is a highly intricate and resource-intensive process.

With VizioR&I™, this heavy lifting has already been done. What would otherwise take months of painstaking work—developing, training, and fine-tuning perception models—can now be effortlessly replaced with our ready-to-use and fully optimised perception blocks. In essence, customers can simply plug-in the system and immediately benefit from advanced perception capabilities.

By eliminating the need to invest resources into understanding complex data and building customised models, VizioR&I™ provides a turnkey solution that allows customers to rapidly iterate & refine new innovations and application-specific enhancements. This not only reduces the time to market for new products, but also enables shifting developmental focus from the foundational aspects of perception to leveraging it in more strategic ways.

To contextualise the unique advantages of perception on-the-edge, let's examine the performance of VizioR&I™ across a variety of challenging perception scenarios.

How Do We Do It?

Introducing 5D Perception®

Central to the notion of 5D Perception® is the synthesis of long-range, high-resolution 3D radar point clouds with Doppler velocity information, constituting the foundational four dimensions. By embedding perception capabilities directly on-the-edge through GPU-enabled radar processing, we introduce the fifth dimension, empowering VizioR&I™ to not merely detect objects, but to discern intricate details of their spatial context and movement dynamics. Using this technology, VizioR&I™ offers a unique take on the industry’s quest for robust & scalable perception by combining the sensor and perception platform into a single device, resulting in significant size, weight, power & cost benefits. Let’s take a closer look at how we achieve this.

DSP Point Cloud Enhancement

1. Initial Signal Processing:

Strong windowing is applied to reduce side-lobes, which are unwanted secondary peaks in the signal spectrum. This improves signal clarity, while also enabling the system to more quickly identify regions of interest with minimal computational load. After windowing, the time-domain radar signals are transformed into the frequency domain using a Fast Fourier Transform (FFT). This enables the identification of different frequency components corresponding to the range and velocity of targets.

2. 3D Domain Utilisation:

In the 2D range-Doppler domain, the Signal-to-Noise Ratio (SNR) span is narrower since detection thresholds need to be set low, thereby increasing the amount of unfiltered noise that passes through azimuth filtering. However, by utilising the 3D range-Doppler-Azimuth domain, the detection peaks are more pronounced, allowing for higher thresholds and reduced noise. In the image below, you can observe the enhanced clarity of detection peaks compared to a 2D and 1D FFT approach.

3. Azimuth Response and Doppler Correction:

To obtain a valid azimuth response per range-Doppler bin, robust Doppler correction and demultiplexing are performed based on the modulation type used. The unique antenna array configuration of the VizioR&I™ sensor significantly enhances performance in this respect by ensuring optimal spatial sampling and reducing interference, leading to more accurate azimuth responses.

4. Zoom-in Processing:

After initial detection sectors are obtained, a propriatary Radar Resolution Enhancement Approach (RREA) is used to enhance the resolution within these sectors. RREA functions as a "zoom-in tool" by iteratively refining azimuth estimates within the detected sectors, increasing the resolution by 2x compared to the native resolution. The integration of RREA in the processing pipeline allows for improved azimuth estimation and the identification of hidden targets within radar captures.

Subsequently, by employing the above approach, a more intricate scene is produced with a resolution beyond the native setting and with significantly reduced noise, as exemplified in the images below:

Object Detection, Classification, and Tracking

Utilising our innate knowledge of the VizioR&I™ hardware, data processing system, and the point clouds it generates, we developed a novel approach in how we preprocess and encode our 4D radar data to maximise the performance of our downstream detection, classification & tracking systems. In addition, the Orin NX used in VizioR&I™ is equipped with NVIDIA's advanced Deep Learning Accelerators (DLAs) and GPUs, which are optimised for different types of neural network computations. Leveraging NVIDIA’s TensorRT software, our neural networks were designed to use this hardware in the most efficient way, resulting in superior performance over traditional algorithms for object detection, classification and tracking tasks, all without the need for any external compute resources.

As a result of these techniques, we observe the following benefits:

• Data preprocessing, inferencing and post-processing are all completed on-device
• End-end processing latency is less than 25 ms per frame
• More than 300 detected objects can be processed in parallel
• The total power consumption for data processing is <20W.

Additionally, VizioR&I™ offers long-range, proficient categorisation of detected objects into several distinct classes, including large vehicles, cars, pedestrians, motorbikes, cyclists, and traffic signs.

Radar Odometry

VizioR&I™ applies a proprietary odometry algorithm that utilises the relationship between radar points and their relative velocity within a frame, combined with an analysis of between-frame position shifts, to calculate the ego vehicle’s velocity and yaw rate. This enables the accumulation of radar points over several seconds through dead-reckoning, which in-turn enables advanced localisation and mapping applications using radar data only. Internal benchmarking with GNSS reference data shows very accurate performance, with radar-based odometry being a viable alternative to GNSS-based localisation in challenging locations such as mines or urban canyons, where GNSS signals may not be available.

Radar Free space

By leveraging the detailed point cloud, robust radar odometry data and embedded NVIDIA DRIVE Orin™ within VizioR&I™, we can perform real-time, on-device radar free space mapping. As a complementary functionality to camera-based free space mapping, radar free space mapping can provide additional sensing range and enhanced robustness in harsh weather or environmental conditions. Moreover, the algorithm can provide robust free space mapping of indoor environments, such as parking garages, mines, and warehouses, due to its independence from GNSS signals.

In the figure below, we see an illustration of radar-only free space mapping performance:

For more information about our free space mapping algorithm visit: Accurately constructing free space maps using radar only - Provizio.

Working With VizioR&I™

Microservices Architecture

VizioR&I™ leverages a software-defined microservices architecture to share data for downstream services. Each microservice plays a crucial role in enhancing specific aspects of radar perception, and each can be updated independently. This allows for great flexibility in the distribution of computing resources, rapid prototyping of features, and a streamlined OTA update process.

Interfaces And APIs

VizioR&I™ is designed to be easy to use and integrate with existing systems across a wide variety of applications.

Communication Protocols

For communication, there is a choice between a simple UDP protocol or DDS to get ROS compatibility out-of-the-box.

• Provizio’s UDP protocol supports up to 65535 points per point cloud, permits identifying radars by position IDs to allow for easy replacement, and enables single or multiple radars on the same or different ports.
• The DDS (Data Distribution Service) protocol is a publish-subscribe based networking middleware for sending and receiving data, events, and commands among nodes. It is used to facilitate communication between different microservices, where publishers create topics and subscribers receive the data they are interested in.

Provizio APIs

VizioR&I™ offers APIs for communication and integration with various systems.

• The C API supports C99+ and can be used in C++ and other client languages. It is compatible with Linux, macOS, and Windows, and supports single-threaded and multi-threaded use. The API provides built-in UDP interfacing and allows for integrating custom transport or replays. Additionally, it is MISRA-compatible and open source.
• The DDS API includes official C++ and Python libraries, and is also compatible with other languages using DDS or ROS2. It offers automatic discovery along with multiple high-performance transport options.
• Our APIs are designed to be simple and portable, with 100% code coverage.

NVIDIA DRIVE Integration

VizioR&I™ is approved for use with the NVIDIA DRIVE AGX Platform, enabling customers to build and test ADAS solutions with maximum compatibility and flexibility within their existing R&D platforms. As the only radar platform on the market with a built-in NVIDIA Orin™ NX GPU, VizioR&I™ offers customers never-before-seen radar perception capabilities that can be tailored to a wide variety of use cases.

Provizio GUI

The VizioR&I™ radar only needs power and a network connection before you can browse to its web interface and see the live point cloud output, or access radar configuration settings.

Foxglove

The Foxglove data visualisation platform is used by various companies to streamline debugging and accelerate development cycles in industries like autonomous vehicles and robotics research. VizioR&I™ offers native support for Foxglove using the MCAP format, enabling customers to easily visualise data captures and better understand the unique performance benefits of VizioR&I™ for their applications.

Multi-Sensor Compatibility

Each VizioR&I™ radar can be configured as a Precision Time Protocol (PTP) Master or PTP Slave on the local Ethernet network. This allows synchronisation of the radar with other sensors to within less than a microsecond accuracy, thereby enabling fusion of point clouds and/or images from multiple sensors for maximum perception performance.

Conclusion

In conclusion, VizioR&I™ is not just another radar system; it represents a quantum leap in radar technology. Until now, the seamless execution of multiple complex perception tasks & microservices required by an advanced perception platform like 5D Perception® would have required the use of high-power external processing systems and complex networking architectures. However, with the power of NVIDIA Orin™, we have been able to consolidate all of these systems into a single device for the first time, enabling massive size, weight, power & cost optimisations compared to existing state-of-the-art solutions.

The unique combination of hardware and software within VizioR&I™ allows it to deliver high-resolution 3D point clouds, robust object detection and classification up to 300 meters, radar odometry, and radar free space mapping, far surpassing the performance and capabilities of existing market offerings. Additionally, our software-defined approach ensures continuous upgrades, extending the radar's useful life and enabling easy adaptability to new challenges and applications.

The immense level of innovation and performance built into VizioR&I™ make it an invaluable tool across various industries, from automotive and agriculture to mining and even smart cities. With its plug-and-play design, ease of integration, and compatibility with other sensors, VizioR&I™ is designed to meet the needs of any customer.

To experience the future of radar technology, we invite you to visit the VizioR&I™ webpage to learn more about our technology and stay updated on the availability of our VizioR&I™ unit. As Turing Award laureate Alan Kay wisely said, "The best way to predict the future is to invent it." With VizioR&I™, the future of mainstream radar perception has arrived.