Research-driven Tools for Ocean Mappers

Current technologies can help ocean mappers and cartographers to fight repetitive, manual and error-prone routines the smart way.

In a world where artificial intelligence is playing a key role in everyday life, it is disappointing when the everyday routines of an ocean mapper or cartographer involve solutions that are crudely manual, repetitive and error prone. Although the root causes of this situation might be debatable (e.g., constraints from abstruse formats, obscure algorithms, obsolete specifications, or a combination thereof), the advantages to the ocean mapping community of catching up with current-era technologies and moving towards smarter and more automated solutions are not.

At first sight, many of the current issues are well-known generic problems, often with robust and widely adopted solutions (e.g., cartographic generalisation). However, when the solution is put into the context of ocean mapping, in a large number of cases it cannot be directly applied. Furthermore, it is sometimes the case that, with the addition of specific requirements, an originally strong solution no longer represents the optimal answer. Although the need for more automated tools is clear, it is also important not to be trapped by one-click solutions. Next-generation surveyors will still need to be qualified to understand when the outcomes of a workflow-specific step are not correct, and be able to troubleshoot the causes. This judgement becomes harder (if not impossible) when the processing details are obfuscated by hidden software logic.

To ease the creation of better and well-understood tools for ocean mapping, since 2016 the Center for Coastal and Ocean Mapping (University of New Hampshire) has conducted a collaborative effort with the NOAA Coast Survey Development Laboratory to develop an open research software environment with applications that cover all phases of the ping-to-public process. This effort, called HydrOffice, aims to facilitate data acquisition, to automate and enhance data processing, and to improve hydrographic products. These themes are driving the creation of a growing collection of hydro-packages, each dealing with specific aspects of the ocean mapping workflow.

The HydrOffice environment, by minimising the effort to initiate and test new ideas, facilitates the creation of new tools for researchers, students and those in the field; and, potentially, eases the industrialisation. The overall goal is to speed up the testing of new ideas and the Research-to-Operations (R2O) transition.

Framework

HydrOffice was designed with a modular structure to facilitate its integration into existing infrastructures and workflows. Additionally, appropriate open licenses are preferred to encourage community contributions. Given that a key goal is to stimulate the creation of new tools and to explore innovative approaches, several base packages performing commonly required operations have been developed and made available.

To provide access to commonly-used sonar raw data formats, the framework has adopted the Hydrographic Universal Data Description Language (Huddl). This avoids having to create and maintain format-specific libraries, or creating (another) generic format which tends to either provide only a subset or a complex superset of all the information stored in the native formats. Avoiding these limitations, Huddl enables the creation of flexible XML descriptions of existing hydrographic data formats. Once a format is described, both code libraries and human-readable documentation are automatically generated. From an archiving perspective, the Huddl-based description provides a label that can be used anytime in the future to access and interpret the data.

Together with Huddl-based packages, a library to access the environmental data collected by various CTD instruments and sound speed profilers is also available. Moving up the ocean mapping workflow, HydrOffice provides packages to access single- and variable-resolution raster data in the Open Navigation Surface’s BAG format as well as vector data in the IHO S-57 format.

Available Tools

In addition to basic building blocks, several applications have been implemented. These applications range from survey management to data distribution and quality control, and illustrate the range of tools enabled by the HydrOffice framework.

Sound Speed Manager

Sound Speed Manager is an application that integrates several best-practices and functionalities to process sound speed profiles and other environmental data for use with sonar systems. The application has been designed to easily integrate with existing systems and streamline data acquisition workflows. In addition to the usual open-source opportunity of learning from (and validating) the algorithms, the liberal licensed used preserves the ability to adapt the application for specific organisational needs.

The main functionalities include: wide support of commonly-used sound speed profile formats, integration with common data acquisition/integration applications, profile enhancement based on real-time and climatological models (Figure 1), and database management of the collected data with built-in functionalities for analysis and visualisation. The most recent developments include the ability to monitor the data acquisition process and to provide time estimation for the next cast (Figure 2).

With a long-term support and development plan, the app represents a turnkey application that has already been adopted by many professionals and institutions in the hydrographic community. The application is freely available for download from: https://www.hydroffice.org/soundspeed/main.

SmartMap

Although existing oceanographic atlases and models provide an enormous amount of four-dimensional information for surveys, such information is delivered in a way that is not easy to translate to the expected survey data quality. The Sea Mapper's Acoustic Ray Tracing Monitor and Planner (SmartMap) project aims to facilitate the evaluation of how oceanographic temporal and spatial variability impacts hydrographic surveys. The task is reached through the calculation of a quality factor that expresses depth uncertainty as a function of local depth. This can help identify problematic spatial and temporal areas to the end-user who is lacking the specialised knowledge.

Currently, the quality factor is calculated on a global scale using oceanographic data derived from the NOAA Global Real-Time Ocean Forecast System. The resulting data is made publicly available through OGC WMS and WCS services as well as a task-specific WebGIS (Figure 3) that can be accessed from: https://www.hydroffice.org/smartmap/.

BAG Explorer

BAG Explorer is a lightweight application specifically developed to inspect the content of an Open Navigation Surface BAG data file.

It provides an intuitive mechanism to explore the tree-like multi-layered structure of the file. The application supports visualisation and validation of the XML metadata content, and inspection of the changes applied to the original data and stored in the tracking list. It can also be used to read the optional layers for single- and variable-resolution grids, and to plot the elevation and the uncertainty layers (Figure 4). The application can be downloaded from: https://www.hydroffice.org/bag/main.

QC Tools

QC Tools is a set of software solutions aiming to alleviate the tedious and monotonous tasks, born from applying hydrographic office specifications to vasts amount of data, which continue to persist in hydrographic data processing and review. Many such specifications are objective and quantitative, and are ripe for automation, which allows more time for humans to perform more challenging tasks, at which they excel. These tasks are generally more subjective and unsuitable for automation, such as those requiring the “judgment of an expert hydrographer”.

By way of example, consider validation of a bathymetric model and S-57 combination. Gridded bathymetry, which quite often represents the final product of a hydrographic survey, routinely consists of tens of millions of nodes - quite unsuitable for human, manual review, although this has been the industry expectation for years. S-57 files often consist of hundreds of features, each with strictly-required attribution, all of which must be consistent with the corresponding bathymetry. Traditionally, this consistency has been evaluated through a manual process which is prone to human error.

QC Tools has numerous functions to rapidly scan through such datasets: gridded bathymetry consisting of millions of data points, bulky feature files, or high-density sounding selections. The application of automated scans, customised to user-specified requirements, can rapidly identify anomalous data points (Figure 5), ensuring product consistency and that all agency-specific requirements are met. The application of QC Tools to NOAA Office of Coast Survey workflow has resulted in improved data quality and timeliness. Yearly updates and edits to the Hydrographic Survey Specifications and Deliverables are now made with an eye toward facilitating automation and anticipating implementation. QC Tools can be downloaded from its official webpage (https://www.hydroffice.org/qctools/main) which also contains manuals and video tutorials.

Conclusions

Over the years, Hydrographic offices and other ocean mapping agencies have generally accumulated large, sometimes wordy, specifications that express thousands of experience-based rules. In order to transition this priceless information into automated data acquisition and processing, it is required to convert them to code that can then be executed and scaled using parallel computing.

Although this operation may appear straightforward, it presents complexities related to human interpretation and possible vagueness of specification. Conflicting interpretations of a rule presents a great opportunity to rethink it so as to reach a unique interpretation. This does not mean that all the rules should be algorithmically rewritten, given that some of them may require the “judgement of an expert hydrographer”. However, such a subset of rules should be carefully identified and labelled as left to human evaluation. The degree of achievable automation is directly linked to the volume of rules that cannot be machine interpreted.

HydrOffice has developed a number of applications that translate to code both existing specifications and long-term best practices, enabled and extended by recent discoveries and research-driven techniques. Increased workflow efficiency reported by many users confirms the benefits of this approach.

Acknowledgements

HydrOffice has received relevant inputs from a number of passionate contributors that would be too long to list. However, we want to explicitly thank Dr. Jonathan Beaudoin (now with QPS), whose original research has been foundational for Sound Speed Manager and SmartMap; Sam Greenaway, Jack Riley, Janice Eisenberg, and John Doroba (NOAA), for the endless efforts in improving the tools; and Paul Johnson (UNH), who greatly helped in their adoption.

The HydrOffice research is funded under NOAA Grant NA15NOS4000200 and NSF Grant 1524585.

More Information

• Masetti, G.; Calder, B.R. (2015), Huddl: the Hydrographic Universal Data Description Language, International Hydrographic Review, 13, pp. 17-32.
• Masetti, G.; Gallagher, B.; Calder, B.R.; Zhang, C.; Wilson, MJ (2017), Sound Speed Manager: An Open-Source Application to Manage Sound Speed Profiles, International Hydrographic Review, 17, pp. 31-40.
• Wilson, M.J.; Masetti, G.; Calder, B.R. (2017), Automated Tools to Improve the Ping-to-Chart Workflow, International Hydrographic Review, 17, pp. 21-30.