Lidar sensor technology is a new non-intrusive way to capture precise, real-time multi-modal data about all road users.
Unlike other systems, with lidar sensors you get reliable data regardless of the weather or lighting conditions, and unlike cameras there are no privacy issues to worry about. The data collected is used to inform cities about areas with traffic problems, help improve safety and aid in urban planning.
In order to entice planners and traffic operations teams to switch to a new cost-effective option such as lidar, the accuracy of the analytics and data being provided must be confirmed.
Bluecity has set out to confirm the accuracy of IndiGO 3D’s lidar sensor solution and AI algorithms.
The study took place in Kelowna, British Columbia, one of the early adopters of Bluecity’s IndiGO 3D solution.
An intersection where an IndiGO 3D sensor had been installed would be monitored in 15-minute aggregation intervals for a period of three hours. An analyst would watch the lidar sensor screen and count the images of the cars as they moved in various directions and this would be referred to as the ground truth. This collected data would then be compared with the data that the AI algorithm processed.
The intersection was identified by four quadrants: North, South, East and West. Cars would be counted as they moved from one quadrant into another. For example, a car could be classified as moving from the north quadrant to the south quadrant (NtoS), meaning that it had crossed the intersection to the other side. Cars could also be classified as moving from the north quadrant to the east quadrant (NtoE) or west quadrant (NtoW) if they turned in either of those directions.
Total Count Accuracy
The result was a series of total cars that were counted during each 15-minute interval. A comparison was then made to see whether the algorithm was as accurate as a human. Over the three hours, the accuracy of the algorithm ranged from 90.57% to 99.21%, with an average accuracy of 95.84%.
While 95.84% accuracy is excellent, the data allowed the Bluecity team to dig deeper into the car counts to see where the differences lay and to find ways to improve the accuracy even more. With twelve different trajectories of cars possible, the team quickly realized that accuracy was abnormally lower than expected in two of the trajectories. With the north to west trajectories, the algorithm counted 4 out of 36 cars and with the south to east trajectories, it counted 130 cars instead of 87.
Knowing that the system was nearly perfect in other trajectories, the analysts needed to explore these unusual results.
The Bluecity team determined that the height of the lidar sensor was causing undercounting of cars moving in a north to west trajectory. At its current height, the sensor was not reading cars that were farther out in the sensor’s field of vision.
They ascertained that the algorithm needed more data in order to learn to count these outlying cars. The team would retrain the algorithm by providing this additional data. The good news was that a physical reconfiguration was not necessary.
In the case of the south to east trajectory, the team determined that the overcounting of cars resulted from noisy data, or data that the system was interpreting incorrectly.
The Case of the Southeast Trajectory
This issue could be corrected quickly. Feeding more data to the algorithms and changing the size and position of the virtual loops would address the areas where errors were occurring. With these tasks identified, the next step was to repeat the study and see how these actions impacted the accuracy. Could an average of 96% accuracy be improved and to what level?
The team reconfigured the lidar sensor remotely and did a quick 15-minute data collection to see how this would affect results. They found a remarkable improvement.
The accuracy increased to 97.50% and in each of the two problemed trajectories the new process yielded a delta of only 1-car.
Bluecity also plans to continue with a series of studies that will investigate variables such as speed, weather and lighting conditions, and safety analytics. The objective of these studies is to confirm IndiGO lidar sensor accuracy, reliability, and the broader insights available.
Bluecity’s goal is to make streets safer for pedestrians, cyclists and motorists. We envision a world where traffic accidents are rare, driving to work is a pleasure and carbon emissions are reduced. Make sure to read our follow-up articles and see how IndiGO’s accuracy continues to improve – coming soon.