For many applications, we need to know what is where underwater. Side-scan sonar is one of the core sensors for imaging the bed of water bodies and it is increasingly being used for inland applications such as finding missing persons, dumped items and unexploded explosives. Traditionally, human operators analyze the images. However, sidescan surveys often take many hours, with repeated passes at different angles to obtain good images. As a result, operator fatigue can become a real constraint, even more so in poor weather or with less experienced operators. Object recognition software can now help make underwater surveys more effective by adding intelligence to sidescan and video data.
EvoLogics is a high-tech German company specializing in underwater communications, positioning, sensor systems and smart robotics. Most of its products are inspired by marine animals. For example, the S2C spread-spectrum communication technology is based on long-distance dolphin communication. The company has also developed an autonomous underwater vehicle inspired by manta rays. Many marine scientists use these products and they are also increasingly being used for professional and commercial applications.
Sonobot
Over the years, EvoLogics has designed and built a number of innovative surface and underwater vehicles. The company has now launched a series of commercially available systems, initially of uncrewed surface vehicles (USV).
Sonobot 5 is the latest version of their survey USV. The unit folds up, so a single operator can transport it in a car and deploy it. It can be fitted with a range of sensors (depth sounder, sidescan sonar, camera, etc.), WLAN and cellular communications, and a choice of GNSS options.
The modular design of the Sonobot 5 makes it a versatile platform for hydrographic surveying, monitoring and search and rescue in inland waters. Its basalt fibre reinforced floats are light, strong and corrosion resistant so the unit can even be used in industrial wastewater reservoirs. The two propellers are driven by 500W brushless motors and are mounted in recesses for protection against impact and to prevent aquatic plants fouling the propellers. Sonobot is powered by two lithium-ion batteries, for an endurance of around 10 hours at typical survey speeds. The special controllers drive the motors with sine waves to minimize noise emissions to the payload electronics. EvoLogics builds the Sonobot in-house and even assembles the batteries itself.
For seafloor imaging, the Sonobot is equipped with a sidescan sonar at 300, 700, or 1200kHz depending on range and resolution requirements, and with an integrated echosounder for depth measurements. These are mounted in the centre section of the USV. All the data is processed on board and then transmitted to the operator over a WLAN or cellular link.
At the end of 2020, EvoLogics launched an exciting new feature for the Sonobot platform – real-time object recognition. The AI-based system runs directly on board the vehicle and analyses raw sidescan sonar output to detect and visually highlight various objects in the operator’s control software on shore. A cloud-based ecosystem around the new feature provides users with regular updates and new detectable object classes. It also enables the upload of user datasets for the system to be trained for new object types – available for all users to download and use in their missions.
Object recognition was first offered to select Sonobot customers and over the following months went through extensive testing and optimization.
Object Recognition Software
Philipp Bannasch, team leader sensor integration at EvoLogics, explains, ‘Our Sonobot 5 USV is a truly high-tech platform for optimal sidescan surveys. To make it even smarter, we have now equipped it with AI-based automatic object recognition. It can autonomously find a wide range of objects underwater: our launch customers have been primarily interested in detecting missing persons, discarded objects, unexploded ordnance (UXO) and abandoned fishing nets.
‘It takes training and experience to recognize object shapes in a sonar picture, and less experienced operators will often miss cues such as contours and sonar shadows. You have to look from multiple angles and normally make several runs at different headings. Thus a survey can be very time-consuming and tiring for the operator who is often working under pressure, outdoors, in poor weather. So now the surveying robot needs to become smart and adjust its mission to what it sees.
‘We developed a high-end object recognition system that runs on the USV itself and directly analyses the raw data, as cloud-based computing is not available in open water. We had to build neural network algorithms that work on sonar data, can handle huge data volumes in real-time, and run on the dedicated embedded hardware. Then, any neural network must be trained to be effective and we have invested a lot of time in developing pre-trained models for missing persons and other objects, relevant to our key customers. We put much time and effort into selecting the right training data, augmenting, and fitting it. That has paid off, the system now detects objects in less than a second. We made sure the system comes with pre-trained object classes and is ready to go “out of the box”.
‘Many of our customers also train the software themselves for a particular type of object they are interested in: they perform scans of the objects of interest and upload their data into the cloud. We train the system for these objects and make the new object class available for download to all users, if we get permission.
‘System operation is uncomplicated. After arrival at the site, the operator programs the Sonobot 5 with the survey grid and the type of object to look for, takes the unit out of the car, unfolds it and launches it into the water. The Sonobot will then follow the grid, process the side-scan data, and send the results to shore. The operator can watch the incoming imagery in real-time, and the object recognition system highlights any relevant findings.
‘In calm waters, you get the best results at a speed over ground of around 0.8m/s. If there are waves, you can increase the speed to, say, 1.2m/s to make it easier to stay on course. The Sonobot 5 can actually reach a speed over water of 5m/s, so it can follow the survey grid even in strong currents and fast-flowing rivers.’
The object recognition has now been successfully tested by several customers such as the Dutch national police unit. 'EvoLogics has recently added object recognition to our Sonobot system. As an operator, it’s great to have an extra set of eyes, looking over your shoulder. It’s work in progress but we will continue to train the software and hopefully it will lead to more recoveries and contribute to our mission to bring all missing people back to their loved ones.'
Gerd Heller of DLRG (German Lifeguard Association) explains 'We have already used the new AI-based sensor capabilities in some missions to retrieve lost equipment from rivers and lakes.'
Lukas Goldmann, an operator from the Underwater Archaeology Association Berlin-Brandenburg, adds that 'If you search for sunken underwater structures such as medieval housing and bridges, where large areas need to be scanned for subtle geometric hints on the ground, a well-trained neural network is a great help to distinguish between natural features and archaeological remains, saving a lot of time on both later analyses and follow-up dives.
‘Besides side-scan underwater imagery, the software also works with video from an onboard camera. This is especially useful for our autonomous underwater vehicles. You can patrol a predefined area on the seafloor, navigate the water by sonar systems, identify critical objects directly in the video feed onboard the submerged AUV, and then automatically surface it to establish a data link and alert the operator.’
Philipp Bannasch continues, ‘We have completed testing and our object recognition with state-of-the-art hardware and software is now available, embedded into our commercial surface and underwater robots. For professional applications, especially in underwater searches where the stakes are often high, it can make a real difference to the success of a mission.’
More Info About Object Recognition software→
Conclusion
A robotic system on or under water with object recognition capabilities can assist a wide range of users to make more effective side-scan sonar surveys of inland water bodies. Key applications include searching for missing persons, objects and UXOs. The same hard- and software also processes underwater video images obtained with UUVs for even more detailed surveys. As EvoLogics produces both the software and hardware in-house and has a large team of specialized scientists, the company can effectively support customers implementing the technology, and provide custom solutions where necessary.
For more information, please see EvoLogics.de.
Locating victims of drowning accidents
Searches for missing persons after an accident on water can be a race against time, often with tragic results. Locating missing persons can be difficult as their bodies can be carried away by strong currents, or become entangled in plants, fishing nets or debris. Because a body absorbs the sonar beam, it can appear as an odd shape in a sonar image. Often the operator recognizes the sonar shadow rather than the shape of the body. The appearance of a body in sonar images depends on many factors, such as the clothes the person is wearing and how long they have been submerged. Incidents often occur at night, in poor weather or in complex settings which make work difficult. This is where automatic, real-time object recognition can bring real benefits. If a potential target is found, its position has to be marked for further inspection by ROVs or divers, and for recovery.
