AI/ML boosts hydrographic data processing in SONARMUS

Artificial intelligence (AI) is currently one of the hottest trends in many fields, including GeoAI for the processing of spatial data. This study briefly describes research in one project working to achieve HydroAI – the use of AI/ML methods for the processing of hydrographic data to ease the work of hydrographers and to automate some of the routine tasks. Results show promising automation potential, but also underline where classic algorithms still outperform AI.

The information presented here is closely related to the SONARMUS project. This was a research project, lasting for 12 months, entitled ‘Technology for intelligent processing of hydrographical data acquired with the use of imaging sonar and single-beam echosounder, mounted on autonomous surface vehicle’. It was funded by the Foundation for Polish Science under grant number FENG.02.07-IP.05-0489/23. The project was carried out at InnoPM Ltd., which is part of the Maritime University of Szczecin (MUS), Poland, and all the researchers involved were employees of the MUS Department of Hydrography and Spatial Analysis and the Department of Geoinformatics and Teledetection. They are listed on the project website (www.sonarmus.eu).

The project in a nutshell

The goal of the project was to develop AI-based methods for the processing of sonar and single-beam echosounder data. As part of this effort, the SONARMUS_app – a visual and testing environment – was developed to enable the qualitative evaluation of the implemented methods and system functionalities.

The technological goal was to propose machine learning (ML) methods and algorithms suitable for various hydrographic data to automate and facilitate the workflow and reduce the hydrographer’s workload during the data processing phase. It was assumed that these methods can effectively handle many typical tasks in a hydrographer’s work.

The project included three main phases: workflow analysis and method development with initial training of the algorithms; new data acquisition and retraining and implementation of the algorithms; and testing and evaluation. One of the main challenges was to gather datasets suitable for training, especially for imaging sonar.

The data was acquired using two MUS research platforms. These are the inland survey research vessel HYDROGRAF XXI and the autonomous surface vehicle MINIMUS (see Figure 1). Survey equipment included the Edgetech 4125 SSS, the StarFish 992H Hull-Mount System for ASV, the EA400 single-beam echosounder and the Echologger EU400 for ASV with additional auxiliary sensors. The survey areas included inland waters of the Odra River around Szczecin with depths typical for harbour areas, and various objects in the riverbed.

AI/ML approach for hydrographic data processing

GeoAI, understood as the application of AI to handle geospatial data, has been proven as an interesting solution for various marine-related problems. AI, or more widely speaking, ML methods are however computationally demanding, and as such are not suitable for all applications.

Therefore, the first task in the project was to analyse the process of various forms of hydrographic data handling and to extract the steps in which AI could be useful. Then, suitable algorithms were proposed. The project’s scope covered the handling of sidescan sonar and single-beam echosounder data (SBES). The use of AI made it possible to create a SMART interface in which most processing steps are automated and final products are created for the user.

In the case of sidescan sonar, typical final products are target reports, with information about contacts, and sonar mosaics, presenting georeferenced sonograms fused within one image. In the case of SBES, typical outcomes are digital bottom models and depth contour maps. Additionally, surveys often include the processing of data from auxiliary sensors, namely positioning and heading, to filter out outliers. These products are available and automated in SONARMUS. Generally, AI was proposed for sonar image handling, target detection and classification, bathymetric data reduction and interpolation and navigational data processing. For automation of other steps, numerical algorithms were implemented. All the algorithms were implemented with open source libraries for data science in Python.

Sidescan sonar data processing

Sidescan sonar data processing can be effectively handled with AI in the form of images, as some of these methods, such as convolutional neural networks, are designed for images. In SONARMUS, AI was proposed for sonograms’ denoising and for target detection and classification.

For denoising, the Neighbour2Neighbour method, based on the U-net architecture, was proposed and validated. This is a deep learning approach used for denoising images. In SONARMUS, the approach without ground truth was proposed. The results are very good, although the implementation requires some modifications and recalculations related to the sonograms’ resolution and other parameters.

A large part of the work was concerned with target detection on the sonograms and classification – a phase that required both the elaboration of algorithms and the preparation of the training samples database. Selected algorithms, used in data science for detection and classification, were used.

Training data for the detection and classification was gathered in several areas using both sonars. The targets were identified and labelled manually on the sonograms as a whole, and also clipped as snippets. This allowed additional augmentation to increase the volume of the training dataset. Six classes were proposed: tire, boulder, corpse, trash, longitudinal target and wreck, and were artificially created for the testing cube (1x1x1 metre). In the first step of the research, the focus was on the classification.

During the tests, the following algorithms, all based on convolution neural networks, were analysed for the classification: AlexNet, ResNet50, ResNet34, GoogLeNet and Vgg19, showing that the classification potential is very high for these known approaches, even though the training set was not very large. Figure 2 shows the accuracy achieved for the training set in each test.

In the second part of the research, the detection component was added to the algorithm, which led directly to the use of the YOLO (You Only Look Once) algorithm. As with the classification, this is based on the assumptions of deep convolution neural networks as a backbone, but with the detection component added. YOLO therefore handles both tasks – detection and classification – during one training process.

The custom implementation of YOLOv8x in SONARMUS was prepared as an additional detection/classification component for the user. The main window of this component is presented in Figure 3.

The implementation enables satisfactory results of detection and classification to be achieved, although the accuracy depends very much on the number of objects of a particular class in the training datasets. Better results can therefore be achieved following training iterations in which additional training datasets are added. The training of the model took a few hours and the detection/classification process in the software is dependant on the implementation and processing parameters of the workstation. However, the timeframe is reasonable and significantly shorter than the manual processing of the sonograms. As a result, the user can produce a report with target position, class, estimated dimensions and IHO classification.

Single-beam echosounder data processing

In SONARMUS, SBES data is understood as filtered xyz points, as the filtering tasks are moved to the preprocessing stage. Thus, the input data to SBES processing is in the form of a point cloud with a density highly related to the survey pattern. ASVs can easily run profiles with the ASVs very close to each other if needed, thus providing dense data even though only single beam. The data can therefore be treated as a point cloud. In this situation, the methods appropriate for point cloud processing were proposed. The goal was to provide a 3D surface from which two typical products – the DBM and depth contour map – can be developed. The only setting that the user has to make in the SMART variant is the resolution. SBES processing includes three major steps – data reduction, interpolation and the generation of isobaths.

Reduction aims at deleting part of the data, while retaining the most important data so as not to lose the characteristics of the seabed. Several ML algorithms were analysed, mainly focusing on the clustering techniques and including, but not limited to, self-organizing map (SOM), density-based spatial clustering of applications with noise (DBSCAN), K-means, agglomerative clustering and autoencoders. Some of these were rejected due to large distortions, and finally the best results were achieved with SOM and hierarchical clustering. For various parameters of the methods and the area, results were achieved that made it possible to reduce 60–80% of the data while maintaining the quality of the seabed model.

Interpolation was the next step. The main purpose of this was to achieve a regular grid from randomly distributed reduced data. The challenge is to achieve a smooth yet accurate surface. For this purpose, self-teaching algorithms were again analysed, including random forest, gradient boosting machine, multilayer perception, general regression neural networks and generative adversarial networks. The methods were analysed based on qualitative and quantitative criteria and also in terms of processing speed and computational burden. Based on this, random forest and gradient boosting methods were chosen for implementation. These provide good accuracy, while maintaining quantitative characteristics of the seabed model. An example of the 3D surface for reduced data is presented in Figure 4a.

The final step was to determine depth contours. This stage was done without AI, using the classical approach based on Gaussian filters for isobath smoothing. It was assumed that the AI implemented in the interpolation stage sufficiently affects the process, making it unnecessary to employ additional AI algorithms for depth contours. The result were highly correlated with interpolation method. An example of a depth contours map is presented in Figure 4b.

Data filtering and preprocessing

While auxiliary information such as heading and position is not directly derived from sonar or SBES, it is still crucial for effective data processing. The GNSS and heading signal tends to be jammed and susceptible to local disturbances, shadowing and spoofing, so that it is important to filter such data prior to the processing of sonar or SBES.

In SONARMUS, it was decided to analyse this data gathered in profiles, using time series analysis methods known from data science. Both numerical and neural approaches were proposed. Numerical filters included the Gaussian filter, moving average, Holt-Winters filter and the Whittaker smoother. From neural filters, mainly recurrent networks were tested, namely RNN (recurrent neural network), LSTM (long short-term memory) and GRU (gated reccurent unit). The classical General Regression Neural Network (GRNN) was also proposed.

The results were very interesting, showing the high potential of numerical and neural filters to smooth functions and detect outliers. RNN, LSTM and GRU required iterative teaching for each profile, which resulted in longer computational times. Therefore, the Whittaker smoother and GRNN were chosen for final implementation.

A similar approach was proposed for filtering SBES profiles to reject outliers, including outlier detection using global and local interquartile range detection before smoothing. An example of an automatically smoothed SBES profile is presented in Figure 5.

Conclusion

The SONARMUS project led to the development of the demonstrator of a technology for sonar and SBES data processing with the use of AI/ML methods for several processing steps. Particularly valuable results were achieved for target detection and classification on sonograms, as well as for depth data reduction and interpolation. Other steps, such as sonogram denoising, mosaicing, depth contour generation, outlier detection and smoothing were also automated to reduce the hydrographer’s workload.

The project shows that some tasks can be effectively automated, but that AI is not a solution for everything. In some stages (e.g. preprocessing), numerical algorithms are faster and sufficiently accurate. In other stages, the use of AI requires long training and considerable computational facilities. AI algorithms are also very much dependent on the volume and quality of training data. This seems to be a crucial part of AI development for hydrographic purposes. The data is specific, so that existing solutions typically need to be additionally trained for this purpose.

Nevertheless, the project has shown that this direction is promising, and very positive feedback for the technology was received during tests performed by external experienced hydrographers. Automated functionalities and the entire flow were highly ranked. Future works are therefore planned, the main direction of which will be the processing of MBES data, the acquisition of further datasets for future development and the implementation and training of new AI/ML models.

References

An example of using AI for hydrographic purposes: https://www.hydro-international.com/content/article/mbes-feature-detection-using-machine-learning

More about GeoAI: GeoAI in the marine domain | Hydro International

Previous paper published on this subject:
https://ieeexplore.ieee.org/document/11132345

DL methods description:

https://openaccess.thecvf.com/content/CVPR2021/papers/Huang_Neighbor2Neighbor_Self-Supervised_Denoising_From_Single_Noisy_Images_CVPR_2021_paper.pdf

https://www.ultralytics.com/blog/introducing-ultralytics-yolov8

Project website (currently only in Polish): www.sonarmus.eu