Provizio logo. Provizio text is in white. Provizio "P" logo is in red.

Provizio

Nov 20, 2024
Nov 20, 2024
14 minute read

Delivering 5D Perception on-the-edge®: VizioR&I™ Imaging Radar with embedded NVIDIA DRIVE Orin™

14 minute read

Introduction

The VizioR&I™ platform from Provizio is not just another radar system; it represents a transformative leap in radar based perception technology. By integrating a robust mmWave radar sensor with an embedded GPU, VizioR&I™ delivers unparalleled performance while optimising size, weight, power, and cost in a single, scalable device. This innovation dramatically simplifies the perception challenge for OEMs, replacing complex, multi-component development stages and data processing steps with a single, streamlined solution.

VizioR&I™ Exploded View

The unique combination of proprietary hardware and software within VizioR&I™ provides unprecedented computational flexibility, enabling robust perception capabilities that surpass the state-of-the-art not just for radar, but also for LiDAR and camera technologies. The immense level of performance delivered by the embedded GPU, coupled with a streamlined plug & play design, make it an invaluable tool across various industries, from automotive and agriculture to mining and smart cities.

On-the-edge GPU

A built-in NVIDIA Orin NX GPU delivers never before seen radar capabilities

Freespace Mapping

High performance radar-only freespace mapping for ADAS & AD applications

Odometry

Radar odometry enables GPS-free localisation in challenging environments

Robust low light & all weather performance

All Weather Reliability

VizioR&I™ provides robust performance in harsh conditions

Doppler Velocity

Intrinsic velocity measurement adds an additional layer of safety

Icon depicting VizioPrime spatial resolution as equivalent to that of LiDAR

High Spatial Resolution

On-device processing & proprietary software enhances radar point clouds

Why Perception On-The-Edge?

In the context of autonomous systems, perception refers to a machine's ability to interpret sensory data from its surroundings and make informed decisions. This involves detecting, classifying, and tracking objects, understanding spatial relationships, and anticipating potential hazards. However, developing a unified perception system that addresses all these tasks is a highly intricate and resource-intensive process.

With VizioR&I™, this heavy lifting has already been done. What would otherwise take months of painstaking work—developing, training, and fine-tuning perception models—can now be effortlessly replaced with our ready-to-use and fully optimised perception blocks. In essence, customers can simply plug-in the system and immediately benefit from advanced perception capabilities.

Radar-based 5D Perception on the left, Camera-based perception on the right

By eliminating the need to invest resources into understanding complex data and building customised models, VizioR&I™ provides a turnkey solution that allows customers to rapidly iterate & refine new innovations and application-specific enhancements. This not only reduces the time to market for new products, but also enables shifting developmental focus from the foundational aspects of perception to leveraging it in more strategic ways.

To contextualise the unique advantages of perception on-the-edge, let's examine the performance of VizioR&I™ across a variety of challenging perception scenarios.

Highway Obstacle Detection

VizioR&I™ Advantage

High resolution sensing with intrinsic Doppler velocity. Coupled with on-device perception, this provides effective detection & tracking of obstacles at range.

Embedded GPU Benefit

100 TOPS processing power from the built-in NVIDIA Orin™ enables on-device data processing with minimal latency, delivering near instantaneous reaction times.

Localisation in GPS Denied Environments

VizioR&I™ Advantage

On-device localisation using radar point cloud based SLAM provides robust performance in adverse conditions & environments.

Embedded GPU Benefit

The high-speed 16GB LPDDR5 memory of the embedded GPU enables inter-frame point cloud position shift calculations in real-time.

Environmental Understanding

VizioR&I™ Advantage

Proprietary Neural Net (NN) algorithms provide high accuracy, low latency drivable free-space estimation using radar point cloud data.

Embedded GPU Benefit

The Deep Learning Accelerators (DLAs) and GPUs of the embedded NVIDIA Orin™ enable advanced on-device NN processing.

Robustness in Adverse Conditions

VizioR&I™ Advantage

mmWave radar technology provides more robust sensor performance in poor conditions. Advanced DSP & generative point cloud enhancements reduce noise and signal interference.

Embedded GPU Benefit

The 8-core ARM CPU & 32 Tensor core GPU of the embedded NVIDIA Orin™ enable simultaneous DSP and NN optimisation of the radar point cloud in real-time.

How Do We Do It?

Introducing 5D Perception®

5D Perception® running on VizioR&I™

Central to the notion of 5D Perception® is the synthesis of long-range, high-resolution 3D radar point clouds with Doppler velocity information, constituting the foundational four dimensions. By embedding perception capabilities directly on-the-edge through GPU-enabled radar processing, we introduce the fifth dimension, empowering VizioR&I™ to not merely detect objects, but to discern intricate details of their spatial context and movement dynamics. Using this technology, VizioR&I™ offers a unique take on the industry’s quest for robust & scalable perception by combining the sensor and perception platform into a single device, resulting in significant size, weight, power & cost benefits. Let’s take a closer look at how we achieve this.

DSP Point Cloud Enhancement

1. Initial Signal Processing:

Strong windowing is applied to reduce side-lobes, which are unwanted secondary peaks in the signal spectrum. This improves signal clarity, while also enabling the system to more quickly identify regions of interest with minimal computational load. After windowing, the time-domain radar signals are transformed into the frequency domain using a Fast Fourier Transform (FFT). This enables the identification of different frequency components corresponding to the range and velocity of targets.

2. 3D Domain Utilisation:

In the 2D range-Doppler domain, the Signal-to-Noise Ratio (SNR) span is narrower since detection thresholds need to be set low, thereby increasing the amount of unfiltered noise that passes through azimuth filtering. However, by utilising the 3D range-Doppler-Azimuth domain, the detection peaks are more pronounced, allowing for higher thresholds and reduced noise. In the image below, you can observe the enhanced clarity of detection peaks compared to a 2D and 1D FFT approach.

1D vs 2D vs 3D Azimuth Thresholding

3. Azimuth Response and Doppler Correction:

To obtain a valid azimuth response per range-Doppler bin, robust Doppler correction and demultiplexing are performed based on the modulation type used. The unique antenna array configuration of the VizioR&I™ sensor significantly enhances performance in this respect by ensuring optimal spatial sampling and reducing interference, leading to more accurate azimuth responses.

4. Zoom-in Processing:

After initial detection sectors are obtained, a propriatary Radar Resolution Enhancement Approach (RREA) is used to enhance the resolution within these sectors. RREA functions as a "zoom-in tool" by iteratively refining azimuth estimates within the detected sectors, increasing the resolution by 2x compared to the native resolution. The integration of RREA in the processing pipeline allows for improved azimuth estimation and the identification of hidden targets within radar captures.

Subsequently, by employing the above approach, a more intricate scene is produced with a resolution beyond the native setting and with significantly reduced noise, as exemplified in the images below:

VizioR&I™ point cloud denoising

Object Detection, Classification, and Tracking

Utilising our innate knowledge of the VizioR&I™ hardware, data processing system, and the point clouds it generates, we developed a novel approach in how we preprocess and encode our 4D radar data to maximise the performance of our downstream detection, classification & tracking systems. In addition, the Orin NX used in VizioR&I™ is equipped with NVIDIA's advanced Deep Learning Accelerators (DLAs) and GPUs, which are optimised for different types of neural network computations. Leveraging NVIDIA’s TensorRT software, our neural networks were designed to use this hardware in the most efficient way, resulting in superior performance over traditional algorithms for object detection, classification and tracking tasks, all without the need for any external compute resources.

As a result of these techniques, we observe the following benefits:

  • Data preprocessing, inferencing and post-processing are all completed on-device
  • End-end processing latency is less than 25 ms per frame
  • More than 300 detected objects can be processed in parallel
  • The total power consumption for data processing is <20W.

Additionally, VizioR&I™ offers long-range, proficient categorisation of detected objects into several distinct classes, including large vehicles, cars, pedestrians, motorbikes, cyclists, and traffic signs.

Vehicles detected and tracked above 300m with VizioR&I™ radar.
5D Perception, detecting cars, pedestrians and freespace, running on the edge in dense city areas.

Radar Odometry

VizioR&I™ applies a proprietary odometry algorithm that utilises the relationship between radar points and their relative velocity within a frame, combined with an analysis of between-frame position shifts, to calculate the ego vehicle’s velocity and yaw rate. This enables the accumulation of radar points over several seconds through dead-reckoning, which in-turn enables advanced localisation and mapping applications using radar data only. Internal benchmarking with GNSS reference data shows very accurate performance, with radar-based odometry being a viable alternative to GNSS-based localisation in challenging locations such as mines or urban canyons, where GNSS signals may not be available.

Radar odometry vs GNSS: Estimated yaw rate & velocity of ego-vehicle for a dual carriageway / urban environment point cloud capture.
VizioR&I™ precisely mapping streets.

Radar Free space

By leveraging the detailed point cloud, robust radar odometry data and embedded NVIDIA DRIVE Orin™ within VizioR&I™, we can perform real-time, on-device radar free space mapping. As a complementary functionality to camera-based free space mapping, radar free space mapping can provide additional sensing range and enhanced robustness in harsh weather or environmental conditions. Moreover, the algorithm can provide robust free space mapping of indoor environments, such as parking garages, mines, and warehouses, due to its independence from GNSS signals.

In the figure below, we see an illustration of radar-only free space mapping performance:

Satellite view (left), freespace view (middle) and overlapping view (right)
Radar freespace running in real-time on VizioR&I™

For more information about our free space mapping algorithm visit: Accurately constructing free space maps using radar only - Provizio.

Working With VizioR&I™

Microservices Architecture

VizioR&I™ leverages a software-defined microservices architecture to share data for downstream services. Each microservice plays a crucial role in enhancing specific aspects of radar perception, and each can be updated independently. This allows for great flexibility in the distribution of computing resources, rapid prototyping of features, and a streamlined OTA update process.

Radar Microservices Architecture

Raw Radar Signal

The initial input obtained from the radar sensor, including reflections from various objects and surfaces in the environment.

Point Cloud

Processes the raw radar signal to generate a point cloud, which is a collection of data points in space representing the physical world as detected by the radar.

Radar-based Odometry

Estimates the position, velocity and orientation of the radar sensor by analysing changes in the radar point cloud over time.

SLAM Accumulation

Accumulates data over time to build a comprehensive map of the environment and provides accurate localisation within the mapped area.

Generative AI Enhanced Point Cloud

Uses generative AI techniques to enhance the raw point cloud data, improving its accuracy and resolution for downstream processes.

CNN Detection & Classification

Used to detect and identify different types of objects and their respective positions within the radar data, providing detailed insights into the environment.

Object Tracking

Uses data from CNN detection and classification, along with radar-based odometry, to maintain a continuous track of objects.

Freespace

Uses the accumulated SLAM data and object tracking information to determine areas free of obstacles, providing a clear path for navigation.

Sensor Fusion

Combines data from multiple sensors (including cameras or LiDAR) to create a more robust and comprehensive understanding of the environment.

Situational Awareness

Utilises fused sensor data to generate a complete situational awareness model. This model helps in making informed decisions by providing a holistic view of the environment, identifying potential hazards, and understanding the dynamic context of the scene.

Interfaces And APIs

VizioR&I™ is designed to be easy to use and integrate with existing systems across a wide variety of applications.

Communication Protocols

For communication, there is a choice between a simple UDP protocol or DDS to get ROS compatibility out-of-the-box.

  • Provizio’s UDP protocol supports up to 65535 points per point cloud, permits identifying radars by position IDs to allow for easy replacement, and enables single or multiple radars on the same or different ports.
  • The DDS (Data Distribution Service) protocol is a publish-subscribe based networking middleware for sending and receiving data, events, and commands among nodes. It is used to facilitate communication between different microservices, where publishers create topics and subscribers receive the data they are interested in.

Provizio APIs

VizioR&I™ offers APIs for communication and integration with various systems.

  • The C API supports C99+ and can be used in C++ and other client languages. It is compatible with Linux, macOS, and Windows, and supports single-threaded and multi-threaded use. The API provides built-in UDP interfacing and allows for integrating custom transport or replays. Additionally, it is MISRA-compatible and open source.
  • The DDS API includes official C++ and Python libraries, and is also compatible with other languages using DDS or ROS2. It offers automatic discovery along with multiple high-performance transport options.
  • Our APIs are designed to be simple and portable, with 100% code coverage.

NVIDIA DRIVE Integration

VizioR&I™ is approved for use with the NVIDIA DRIVE AGX Platform, enabling customers to build and test ADAS solutions with maximum compatibility and flexibility within their existing R&D platforms. As the only radar platform on the market with a built-in NVIDIA Orin™ NX GPU, VizioR&I™ offers customers never-before-seen radar perception capabilities that can be tailored to a wide variety of use cases.

Provizio GUI

The VizioR&I™ radar only needs power and a network connection before you can browse to its web interface and see the live point cloud output, or access radar configuration settings.

Foxglove

The Foxglove data visualisation platform is used by various companies to streamline debugging and accelerate development cycles in industries like autonomous vehicles and robotics research. VizioR&I™ offers native support for Foxglove using the MCAP format, enabling customers to easily visualise data captures and better understand the unique performance benefits of VizioR&I™ for their applications.

Multi-Sensor Compatibility

Each VizioR&I™ radar can be configured as a Precision Time Protocol (PTP) Master or PTP Slave on the local Ethernet network. This allows synchronisation of the radar with other sensors to within less than a microsecond accuracy, thereby enabling fusion of point clouds and/or images from multiple sensors for maximum perception performance.

Conclusion

In conclusion, VizioR&I™ is not just another radar system; it represents a quantum leap in radar technology. Until now, the seamless execution of multiple complex perception tasks & microservices required by an advanced perception platform like 5D Perception® would have required the use of high-power external processing systems and complex networking architectures. However, with the power of NVIDIA Orin™, we have been able to consolidate all of these systems into a single device for the first time, enabling massive size, weight, power & cost optimisations compared to existing state-of-the-art solutions.

The unique combination of hardware and software within VizioR&I™ allows it to deliver high-resolution 3D point clouds, robust object detection and classification up to 300 meters, radar odometry, and radar free space mapping, far surpassing the performance and capabilities of existing market offerings. Additionally, our software-defined approach ensures continuous upgrades, extending the radar's useful life and enabling easy adaptability to new challenges and applications.

On-the-edge GPU

A built-in NVIDIA Orin NX GPU delivers never before seen radar capabilities

Freespace Mapping

High performance radar-only freespace mapping for ADAS & AD applications

Odometry

Radar odometry enables GPS-free localisation in challenging environments

Robust low light & all weather performance

All Weather Reliability

VizioR&I™ provides robust performance in harsh conditions

Doppler Velocity

Intrinsic velocity measurement adds an additional layer of safety

Icon depicting VizioPrime spatial resolution as equivalent to that of LiDAR

High Spatial Resolution

On-device processing & proprietary software enhances radar point clouds

The immense level of innovation and performance built into VizioR&I™ make it an invaluable tool across various industries, from automotive and agriculture to mining and even smart cities. With its plug-and-play design, ease of integration, and compatibility with other sensors, VizioR&I™ is designed to meet the needs of any customer.

To experience the future of radar technology, we invite you to visit the VizioR&I™ webpage to learn more about our technology and stay updated on the availability of our VizioR&I™ unit. As Turing Award laureate Alan Kay wisely said, "The best way to predict the future is to invent it." With VizioR&I™, the future of mainstream radar perception has arrived.

Share

Sign up for our newsletter

Stay up to date with all the latest news and events at Provizio.
Email address
Subscribe
Provizio logo. Provizio text is in white. Provizio "P" logo is in red.

Testbed, Deliveries and 5D Perception® Demo Drives

Provizio, Future Mobility Campus Ireland
Shannon Free Zone
V14WV82
Ireland

Company Information

Provizio Ltd
VAT Number: IE3638928AH
Company Number: 654660 (Registered in Ireland)
Testbed, deliveries and 5D Perception® demo drives

Provizio, Future Mobility Campus Ireland
Shannon Free Zone
V14WV82
Ireland

Sales & support

Newlab Michigan Central,
Detroit,
MI 48216,
United States

Provizio Ltd
VAT Number: IE3638928AH
Company Number: 654660 (Registered in Ireland)
Reach out to us and see how we can help
Talk to us

Quality Management Systems

QAS International ISO 9001:2015 Registered Company

Information Security Management Systems

QAS International ISO/IEC 27001:2022 Registered Company
Copyright © 2024 Provizio, Ltd. All rights reserved.
crossmenuchevron-down