Introduction

The VizioR&I™ platform from Provizio is not just another radar system; it represents a transformative leap in radar based perception technology. By integrating a robust mmWave radar sensor with an embedded GPU, VizioR&I™ delivers unparalleled performance while optimising size, weight, power, and cost in a single, scalable device. This innovation dramatically simplifies the perception challenge for OEMs, replacing complex, multi-component development stages and data processing steps with a single, streamlined solution.

The unique combination of proprietary hardware and software within VizioR&I™ provides unprecedented computational flexibility, enabling robust perception capabilities that surpass the state-of-the-art not just for radar, but also for LiDAR and camera technologies. The immense level of performance delivered by the embedded GPU, coupled with a streamlined plug & play design, make it an invaluable tool across various industries, from automotive and agriculture to mining and smart cities.

Why Perception On-The-Edge?

In the context of autonomous systems, perception refers to a machine's ability to interpret sensory data from its surroundings and make informed decisions. This involves detecting, classifying, and tracking objects, understanding spatial relationships, and anticipating potential hazards. However, developing a unified perception system that addresses all these tasks is a highly intricate and resource-intensive process.

With VizioR&I™, this heavy lifting has already been done. What would otherwise take months of painstaking work—developing, training, and fine-tuning perception models—can now be effortlessly replaced with our ready-to-use and fully optimised perception blocks. In essence, customers can simply plug-in the system and immediately benefit from advanced perception capabilities.

By eliminating the need to invest resources into understanding complex data and building customised models, VizioR&I™ provides a turnkey solution that allows customers to rapidly iterate & refine new innovations and application-specific enhancements. This not only reduces the time to market for new products, but also enables shifting developmental focus from the foundational aspects of perception to leveraging it in more strategic ways.

To contextualise the unique advantages of perception on-the-edge, let's examine the performance of VizioR&I™ across a variety of challenging perception scenarios.

How Do We Do It?

Introducing 5D Perception®

Central to the notion of 5D Perception® is the synthesis of long-range, high-resolution 3D radar point clouds with Doppler velocity information, constituting the foundational four dimensions. By embedding perception capabilities directly on-the-edge through GPU-enabled radar processing, we introduce the fifth dimension, empowering VizioR&I™ to not merely detect objects, but to discern intricate details of their spatial context and movement dynamics. Using this technology, VizioR&I™ offers a unique take on the industry’s quest for robust & scalable perception by combining the sensor and perception platform into a single device, resulting in significant size, weight, power & cost benefits. Let’s take a closer look at how we achieve this.

DSP Point Cloud Enhancement

1. Initial Signal Processing:

Strong windowing is applied to reduce side-lobes, which are unwanted secondary peaks in the signal spectrum. This improves signal clarity, while also enabling the system to more quickly identify regions of interest with minimal computational load. After windowing, the time-domain radar signals are transformed into the frequency domain using a Fast Fourier Transform (FFT). This enables the identification of different frequency components corresponding to the range and velocity of targets.

2. 3D Domain Utilisation:

In the 2D range-Doppler domain, the Signal-to-Noise Ratio (SNR) span is narrower since detection thresholds need to be set low, thereby increasing the amount of unfiltered noise that passes through azimuth filtering. However, by utilising the 3D range-Doppler-Azimuth domain, the detection peaks are more pronounced, allowing for higher thresholds and reduced noise. In the image below, you can observe the enhanced clarity of detection peaks compared to a 2D and 1D FFT approach.

3. Azimuth Response and Doppler Correction:

To obtain a valid azimuth response per range-Doppler bin, robust Doppler correction and demultiplexing are performed based on the modulation type used. The unique antenna array configuration of the VizioR&I™ sensor significantly enhances performance in this respect by ensuring optimal spatial sampling and reducing interference, leading to more accurate azimuth responses.

4. Zoom-in Processing:

After initial detection sectors are obtained, a propriatary Radar Resolution Enhancement Approach (RREA) is used to enhance the resolution within these sectors. RREA functions as a "zoom-in tool" by iteratively refining azimuth estimates within the detected sectors, increasing the resolution by 2x compared to the native resolution. The integration of RREA in the processing pipeline allows for improved azimuth estimation and the identification of hidden targets within radar captures.

Subsequently, by employing the above approach, a more intricate scene is produced with a resolution beyond the native setting and with significantly reduced noise, as exemplified in the images below:

Object Detection, Classification, and Tracking

Utilising our innate knowledge of the VizioR&I™ hardware, data processing system, and the point clouds it generates, we developed a novel approach in how we preprocess and encode our 4D radar data to maximise the performance of our downstream detection, classification & tracking systems. In addition, the NVIDIA Jetson Orin™ NX used in VizioR&I™ is equipped with NVIDIA's advanced Deep Learning Accelerators (DLAs) and GPUs, which are optimised for different types of neural network computations. Leveraging NVIDIA’s TensorRT software, our neural networks were designed to use this hardware in the most efficient way, resulting in superior performance over traditional algorithms for object detection, classification and tracking tasks, all without the need for any external compute resources.

As a result of these techniques, we observe the following benefits:

• Data preprocessing, inferencing and post-processing are all completed on-device
• End-end processing latency is less than 25 ms per frame
• More than 300 detected objects can be processed in parallel
• The total power consumption for data processing is <20W.

Additionally, VizioR&I™ offers long-range, proficient categorisation of detected objects into several distinct classes, including large vehicles, cars, pedestrians, motorbikes, cyclists, and traffic signs.

Radar Odometry

VizioR&I™ applies a proprietary odometry algorithm that utilises the relationship between radar points and their relative velocity within a frame, combined with an analysis of between-frame position shifts, to calculate the ego vehicle’s velocity and yaw rate. This enables the accumulation of radar points over several seconds through dead-reckoning, which in-turn enables advanced localisation and mapping applications using radar data only. Internal benchmarking with GNSS reference data shows very accurate performance, with radar-based odometry being a viable alternative to GNSS-based localisation in challenging locations such as mines or urban canyons, where GNSS signals may not be available.

Radar Free space

By leveraging the detailed point cloud, robust radar odometry data and embedded NVIDIA Jetson Orin™ NX within VizioR&I™, we can perform real-time, on-device radar free space mapping. As a complementary functionality to camera-based free space mapping, radar free space mapping can provide additional sensing range and enhanced robustness in harsh weather or environmental conditions. Moreover, the algorithm can provide robust free space mapping of indoor environments, such as parking garages, mines, and warehouses, due to its independence from GNSS signals.

In the figure below, we see an illustration of radar-only free space mapping performance:

For more information about our free space mapping algorithm visit: Accurately constructing free space maps using radar only - Provizio.

Working With VizioR&I™

Microservices Architecture

VizioR&I™ leverages a software-defined microservices architecture to share data for downstream services. Each microservice plays a crucial role in enhancing specific aspects of radar perception, and each can be updated independently. This allows for great flexibility in the distribution of computing resources, rapid prototyping of features, and a streamlined OTA update process.

Interfaces And APIs

VizioR&I™ is designed to be easy to use and integrate with existing systems across a wide variety of applications.

Communication Protocols

For communication, there is a choice between a simple UDP protocol or DDS to get ROS compatibility out-of-the-box.

• Provizio’s UDP protocol supports up to 65535 points per point cloud, permits identifying radars by position IDs to allow for easy replacement, and enables single or multiple radars on the same or different ports.
• The DDS (Data Distribution Service) protocol is a publish-subscribe based networking middleware for sending and receiving data, events, and commands among nodes. It is used to facilitate communication between different microservices, where publishers create topics and subscribers receive the data they are interested in.

Provizio APIs

VizioR&I™ offers APIs for communication and integration with various systems.

• The C API supports C99+ and can be used in C++ and other client languages. It is compatible with Linux, macOS, and Windows, and supports single-threaded and multi-threaded use. The API provides built-in UDP interfacing and allows for integrating custom transport or replays. Additionally, it is MISRA-compatible and open source.
• The DDS API includes official C++ and Python libraries, and is also compatible with other languages using DDS or ROS2. It offers automatic discovery along with multiple high-performance transport options.
• Our APIs are designed to be simple and portable, with 100% code coverage.

NVIDIA DRIVE Integration

VizioR&I™ is approved for use with the NVIDIA DRIVE AGX™ Platform, enabling customers to build and test ADAS solutions with maximum compatibility and flexibility within their existing R&D platforms. As the only radar platform on the market with a built-in NVIDIA Jetson Orin™ NX, VizioR&I™ offers customers never-before-seen radar perception capabilities that can be tailored to a wide variety of use cases.

Provizio GUI

The VizioR&I™ radar only needs power and a network connection before you can browse to its web interface and see the live point cloud output, or access radar configuration settings.

Foxglove

The Foxglove data visualisation platform is used by various companies to streamline debugging and accelerate development cycles in industries like autonomous vehicles and robotics research. VizioR&I™ offers native support for Foxglove using the MCAP format, enabling customers to easily visualise data captures and better understand the unique performance benefits of VizioR&I™ for their applications.

Multi-Sensor Compatibility

Each VizioR&I™ radar can be configured as a Precision Time Protocol (PTP) Master or PTP Slave on the local Ethernet network. This allows synchronisation of the radar with other sensors to within less than a microsecond accuracy, thereby enabling fusion of point clouds and/or images from multiple sensors for maximum perception performance.

Conclusion

In conclusion, VizioR&I™ is not just another radar system; it represents a quantum leap in radar technology. Until now, the seamless execution of multiple complex perception tasks & microservices required by an advanced perception platform like 5D Perception® would have required the use of high-power external processing systems and complex networking architectures. However, with the power of NVIDIA Jetson Orin™ NX, we have been able to consolidate all of these systems into a single device for the first time, enabling massive size, weight, power & cost optimisations compared to existing state-of-the-art solutions.

The unique combination of hardware and software within VizioR&I™ allows it to deliver high-resolution 3D point clouds, robust object detection and classification up to 300 meters, radar odometry, and radar free space mapping, far surpassing the performance and capabilities of existing market offerings. Additionally, our software-defined approach ensures continuous upgrades, extending the radar's useful life and enabling easy adaptability to new challenges and applications.

The immense level of innovation and performance built into VizioR&I™ make it an invaluable tool across various industries, from automotive and agriculture to mining and even smart cities. With its plug-and-play design, ease of integration, and compatibility with other sensors, VizioR&I™ is designed to meet the needs of any customer.

To experience the future of radar technology, we invite you to visit the VizioR&I™ webpage to learn more about our technology and stay updated on the availability of our VizioR&I™ unit. As Turing Award laureate Alan Kay wisely said, "The best way to predict the future is to invent it." With VizioR&I™, the future of mainstream radar perception has arrived.

Introduction

Preprocessed High Definition (HD) maps cannot capture changes in road structure, roadworks or debris. Therefore, onboard sensors must be able to estimate a real-time freespace map in order to avoid unexpected obstacles and navigate safely through an environment.

While radar has traditionally been used for object tracking because of its long-distance sensing and accurate doppler velocity measurement capabilities, the addition of our MIMSO® technology makes it a powerful sensor for gathering dense 3D pointclouds, and at a much lower cost than comparable sensors. The increased angular resolution capability of MIMSO® radars also enables complex perception tasks, like radar-only freespace estimation.

This whitepaper will explore our approach to radar free space mapping & estimation and how this compares to other freespace estimation techniques.

Definition of Freespace

Inverse Sensor Modelling (ISM) defines occupancy as either free, occupied, unobserved or partially observed [1], as seen in Figure 1. These labels are created by casting rays out from the sensor. Occupied represents any region with a sensor detection, free represents the area up to the first detection, partially observed is the region between the first and last detections, and unobserved is the region beyond the last detection.

We also include statically free and dynamically free labels to cater for the influence of static and dynamic object detections. This designation is made possible by our robust ego velocity estimation algorithm, which can estimate ground velocity and thereby remove detections that are not static i.e., dynamic targets. This allows us to create static and dynamic freespace maps, which highlight regions that will always be off limits and regions that might become traversable freespace in the near future.

Technology Overview

Our 5D Perception® system is built upon an extensive microservice architecture. These microservices work together to produce perception outputs like freespace estimation. Figure 2 outlines how the data is processed in each microservice. The Odometry, Dynamic Target Filter and SLAM accumulation microservices prepare the radar pointcloud for freespace estimation as follows:

• Odometry calculates the position, velocity, orientation angle and rate of orientation angle change from the radar pointcloud only.
• The Dynamic Target Filter uses the ego velocity estimate from Odometry to calculate the ground velocity of each detection. Dynamic detections (a ground velocity greater than 0m/s) are removed so that only the static scene remains.
• SLAM accumulation then uses the Odometry output to move old saved pointclouds to their current position with respect to the radar. This increases pointcloud density by overlapping old pointclouds with the newly captured pointcloud data.

Figure 3 illustrates the effectiveness of radar odometry for pointcloud accumulation. In the leftmost illustration, the road boundaries are very difficult to identify from a single pointcloud frame. In the middle figure, we see how accumulation significantly increases pointcloud density, thereby resulting in the structure of the road becoming clearly visible. Dynamic objects usually cause smearing with this type of accumulation because they are moving from frame to frame. However, this does not occur in our case because dynamic detections have been removed using our Dynamic Target Filter.

Once an accurate accumulated pointcloud has been produced by the microservice system, it is processed by the Freespace Estimation microservice, as described below and as illustrated in Figure 4.

• Height filtering removes elevated targets (bridges, overpasses or ceilings) that the vehicle can safely pass underneath without obstruction.
• An occupancy gridmap is created to simplify computation and to create a structured pointcloud with known neighbours (shown in the rightmost illustration of Figure 3 above).
• Clustering is performed to detect static features such as curbs, fences, walls, buildings and other static obstacles.
• Finally, a freespace polygon is stitched together from the previous processing stages.

The goal of our freespace estimation algorithm is to identify freespace beyond what can be seen by ray tracing. This makes it possible to estimate freespace that might be out of direct line-of-sight but that is still visible to the radar.

Comparing Freespace Performance

Radar compared to camera

Freespace estimation with camera has been extensively researched and many off-the-shelf road segmentation algorithms with decent performance already exist. Figure 5 below illustrates a comparison between our radar-only freespace algorithm and a camera-based road segmentation approach at a complex intersection. Road segmentation operates on the camera image and is then projected onto a plane in front of the camera to create a Bird’s-Eye-View (BEV). From the satellite image, we can clearly see that the freespace should be represented as the road diverging to the left to exit the roundabout, and the road that follows to the right around the roundabout.

The camera freespace estimate follows the road very accurately and also provides road markings, which radar and some LiDAR cannot do. It is important to have an accurate model of the ground plane and camera intrinsics to project the camera estimate to a BEV. Small changes in pitch angle can cause large errors at long ranges. Radar does not have this issue because range is measured directly. Furthermore, our radar-only freespace estimate can ignore dynamic targets and produce an estimate of static freespace. This is challenging to do using camera because velocity is not directly measured.

Figure 6 shows an example of rain reducing the accuracy of the freespace estimate. Water on the camera lens distorts and blocks the image, leading to an incorrect freespace estimate. Radar is not affected as severely by these weather conditions, which makes it a good complementary sensor to camera.

Radar compared to LiDAR

Inverse Sensor Modelling (ISM) is a common method for calculating freespace using LiDAR data. With this approach, a 3D pointcloud is converted to a 2D BEV representation and then passed to the algorithm. Freespace is marked between the sensor and the first detection along a ray. However, since a 2D BEV representation is used, there might be detail that is visible in 3D, but blocked/lost in the 2D view. As a result, curved and complex roads can have reduced freespace estimation range with this method.

The same roundabout scene from Figure 5 is shown in Figure 7. Our freespace algorithm follows the shape of the road in both directions to a range of 40m. However, the ISM freespace estimate using LiDAR data fails to follow the curvature of the road, which limits the range and accuracy of the estimate. Even though the side of the road is visible in the LiDAR data, ISM is being blocked by points closer to the sensor.

Use Cases

The freespace algorithm showcased in this whitepaper, is a digital signal processing (DSP) algorithm and therefore can be run using a low power CPU. High performing GPUs are not needed in this case, unlike some of our deep learning approaches which can compliment this approach, as they are not always availible.

Unlike LiDAR pointclouds and camera images, radar pointclouds do not require ground segmentation. Reflections from the road are usually weak enough that they do not appear as a detection. LiDARs without velocity estimation and camera images also require dynamic target segmentation. As a result, pointcloud filtering with radar is simplified significantly because ground segmentation and dynamic target segmentation are not necessary. The low cost of this radar processing solution makes it a viable candidate for many industries such as automotive, mining and agriculture.

Curved roads and complex junctions

Figure 8 shows the same scenario as the comparison section above. A scene including curved roads and intersections with islands has been selected because many algorithms tend to fail in these situations. Satellite view, freespace view and the two views overlapping are shown in the figure. The overlapped view illustrates how the freespace estimate follows the shape of the roundabout and the shape of the diverging road.

Figure 8 contains a single example of freespace estimation in this environment. The video in Figure 9 contains more examples of freespace estimation around a roundabout. As can be seen in the video, freespace is quickly and consistently estimated as the vehicle approaches each junction.

Long range scenarios

Radar has excellent range capabilities, which allows obstacles to be detected early. Detection range is important for fast moving vehicles like cars and drones because large distances can be covered in a small amount of time. In Figure 10, we see an example of a truck pulled into the hard shoulder of a curved road. In the corresponding freespace map, we can see how the road visibly narrows next to the obstacle at 125m.

This scenario is difficult because the curve of the road obscures the obstacle. Despite this, the radar first detects the obstacle at greater than 300m and it is clearly visible in the freespace map at 125m as seen in Figure 11. The speed limit of the vehicle is 100km/h or 27.8m/s. Therefore the vehicle has 10.8 seconds from first detecting the object to process the obstacle and 4.5 seconds to avoid the vehicle using the freespace map.

Indoor scenarios

Indoor environments can be challenging to map because they are often dimly lit, cannot avail of GPS and have targets at both close and long range. These environments are particularly common in industrial use cases such as mining and warehouses. Since our approach uses radar-only to estimate the pose of the vehicle, it is not reliant on GPS or other external odometry methods.

Figure 12 shows an indoor car park environment which has harsh lighting, a variety of close and long range targets, and clutter above the radar. The clutter from the ceiling is removed correctly so that it does not block the freespace estimate. Using our technique, spaces between vehicles on the left and a junction to turn right are both detected at greater than 20m.

Figure 12 is a snapshot from a longer video (shown in Figure 13), which illustrates freespace mapping performance when driving through an indoor car park. Freespace is estimated past a range of 30m, including free parking spaces, junctions, large open areas and dead-ends.

Conclusion

Our radar free space mapping algorithm can be used in a variety of environments, including short range indoor and long range outdoor applications. It has also been shown that radar performs well in challenging lighting and weather conditions, which makes it a good complementary sensor to camera. The low cost of the radar and processing stack makes radar-only freespace a viable option for industries such as automotive, mining and agriculture.

Our radar free space mapping algorithm can function using just one radar or by combining multiple radar pointclouds together. Odometry is calculated using the radar pointcloud and used to accumulate the pointcloud to provide greater density. This is possible because of the accurate doppler velocity estimate that radar provides and our robust ego velocity estimation algorithm.

Estimate accuracy and range is improved by looking beyond line-of-sight. This allows us to predict where the road will become free in the future, well before the vehicle arrives at that location. Many freespace algorithms fail around curved and complex road structures but our estimate does not degrade in these conditions. Furthermore, dynamic objects are easily removed, which allows us to provide a separate static freespace estimate and dynamic freespace estimate.

References

[1] Probably Unknown: Deep Inverse Sensor Modelling In Radar: https://arxiv.org/pdf/1810.08151.pdf