Hybrid method for obstacle and road edge segmentation in multimodal onboard computer vision system

Segmentation vehicle obstacles and roadway edges is important for driver assistance systems as well as for unmanned vehicles and intelligent robots. Existing solutions are predominantly focused on driving on paved roads in urban environments and may perform unstably in adverse weather conditions and off-road environments. In this paper, we propose a hybrid method that combines neural network algorithms for roadway segmentation and positive (rocks, cars) and negative (pits, puddles) obstacles and a heuristic algorithm for recognizing the right and left edges of the road using two modalities: RGB image and depth map. To qualitatively and quantitatively evaluate the performance of the algorithms, a target domain dataset including three different locations with a variety of weather and lighting conditions is collected and labeled. It is shown that the complexization of depth maps and images in the proposed method can provide distance estimation to obstacles and road edges. It is demonstrated that the developed algorithms can operate above 12 frames per second, indicating their applicability in on-board vision systems.