GeoAI in the marine domain

GeoAI is the application of artificial intelligence (AI) fused with geospatial data, science and technology to solve geographic-based problem sets. GeoAI, therefore, is not a product to be bought and sold, but an integrated method, or pattern, for conducting spatial analysis using the power of computers. The use of GeoAI on land has been widely publicized for use cases such as object detection (buildings, roads, trees), land use classification and change detection. The application of GeoAI to the marine environment may be less apparent, but the basic uses of GeoAI – object detection, classification and change detection – can certainly be applied to the marine domain, both above and below the water. These uses can be applied across a wide spectrum of marine-related activities, including chart production, marine security and environmental protection and monitoring.

Hydrographic offices have a mandate to protect life at sea. To do this, they need to produce accurate navigational charts with timely updates. However, accurate and timely have long been at odds. Using an object detection model with a GIS, such as ArcGIS Pro, the path from data to chart can be automated. The Esri GeoAI team set out to prove this use case in preparation for the 2020 User Conference.

Natural disasters or other regional phenomena can mean drastic changes are required to navigational charts. In 2012, Hurricane Sandy devastated the area around Jamaica Bay, NY. A post-disaster bathymetric survey identified numerous wrecks that were unaccounted for in the S-57 chart over this 100km² area. The Esri GeoAI team decided that a supervised classification method would be required, so they would need training samples to feed into the algorithm. They enhanced the bathymetric surface with a shaded relief function to better highlight the elevation changes. Using the built-in deep learning toolset in ArcGIS Pro, they used the collected training samples to train then execute the model, resulting in hundreds of new detections. They then reviewed the detections to determine which required charting and which could be excluded. Note that, while shipwrecks were the target in this experiment, other objects with a unique pattern could easily take their place.

Monitoring the natural environment

In addition to surveying for safe navigation, many hydrographic offices have an array of sensors collecting vast amounts of data to monitor the biology, chemistry and physical oceanography of their coastal waters. For example, NOAA estimates that it collects about 20TB of data every day. Models are required to sift through this information and alert humans when action is needed.

Canada’s Department of Fisheries and Oceans (DFO) regularly surveys Canadian Arctic waters to assess whale populations for stock assessment. Aerial imagery is regularly collected and used in conjunction with satellite imagery for this purpose. DFO and Esri Canada set out to see if there could be a use for GeoAI in the form of deep learning to detect Beluga whales from imagery. Beluga whales were chosen to train the model because of their lighter colour and size, which makes them easier to detect than other Arctic marine mammals such as narwhals, walrus or seals. They found that using an object identification model led to many false positives as the whales, floating sea ice and whitecaps all look similar. Instead, they used pixel-based classification, which applies a similar process to object detection in that labelling and training are required, but these models segment out the pixels by value and location, grouping similar pixels into classes. The models effectively do the tedious ‘panning and scanning’ for the human analyst and target areas of the imagery that need review and verification. By employing these models, DFO hopes to be able to review imagery in the season it was collected, providing more timely information. The model accuracy was estimated to be 80–85%. Like all models, it can be tuned further with more training samples. So, while not perfect, the models give analysts a major boost in efficiency, especially when they can be run inside a software suite already in use.

To give another example, identifying and monitoring the coastline is a monumental task, but one that allows scientists to gain an understanding of the effects of climate change. When coupled with the estimate that nearly 15% or about one billion people live within ten kilometres of a coastline, this data collection is of great importance to our society. The United Kingdom Hydrographic Office (UKHO) set out to establish a baseline extent of the global coastline with a greater accuracy than ever before. Its combined use of data science techniques, machine learning (ML) algorithms and human expertise to solve a geographic problem is a notable use of GeoAI. Because of the many variations of coastlines around the world, it used a pixel classification method called Otsu Thresholding, essentially segmenting the image into two categories: coastline and not coastline. Using this method, it was able to produce a much more detailed coastline map than that previously available. This more detailed map will allow for the finer-grain study of the effects of changing climate and the environmental impacts of natural phenomena such as shoreline erosion.

GeoAI is not just useful to ‘detect things’ in imagery. In fact, machine learning and deep learning models are built into a variety of tools at the disposal of a GIS analyst. Esri and Hypack partnered to complete a proof of concept where Esri forecasting tools would run on Hypack collected data to forecast where sedimentation would occur around the Port of Tuxpan, MX. The idea was to use the 20+ years of bathymetric data to train the model to predict when and where sedimentation is occurring around the port area that could affect the size of vessels using the port. This analysis used a deep learning technique called time series forecasting. Time series forecasting searches for patterns and trends to use in making a prediction for a future value. In this case, that was a value for the depth of the channel at that location. Using this forecasted depth measurement to subtract from the safe navigational depth, Esri was able to predict where, when and how much sediment would need to be removed to keep the channel safe for navigation.  

Protecting the blue economy

Continuing the theme of massive scale, it is estimated that 70% of global trade is carried by marine transportation. One study by the United Nations concluded that the value of the blue economy is US$3–6 trillion every year. There is obviously a lot at stake if the resources are not used equitably and sustainably. One of the scourges of the blue economy is illegal, unreported and unregulated (IUU) fishing, the impacts of which have been widely studied. Global Fishing Watch, an international non-profit organization, uses a GeoAI pattern by combining satellite imagery, big data feeds and machine learning to determine where and when IUU is taking place, and the participants. It does this in near real time by analysing AIS and VMS feeds and satellite imagery object detections. It also filters out non-fishing activities through analysis of the speed and direction of a given vessel and makes the data available for download and access via API. This use of the GeoAI pattern gives anyone with an internet connection access to a massive global dataset. Authorities can also use this data to monitor IUU in their economic zones.

Another example: about 30% of the world’s oil comes from undersea reserves. While big spills dominate the news cycle, NOAA estimates that thousands of spills occur in US waters every year. Fortunately, oil spills leave a pattern on the water that can be recognized by computer vision and used in a GeoAI pattern to indicate where an oil spill has occurred and its extent. Models can then be used to predict where the oil will travel based on currents and other environmental factors. Chen and Small (2022) used image/pixel classification techniques to isolate pixels containing oil from the non-tainted water. The unique aspect of their study was to go beyond the visible spectrum and use the imagery collected from infrared sensors. The result shows that oil can be detected from the surrounding seawater due to the contrast in thermal properties, indicating that GeoAI can go beyond the visible spectrum to detect anomalies that the human eye cannot.

Risks and challenges for GeoAI in the marine environment

Using GeoAI patterns is not without risk or challenges. The largest risk is misidentification or misclassification. Even if a model reaches a confidence level average of 90%, that confidence level changes from feature to feature and pixel to pixel. Human intervention is therefore required to verify the results of any output from the models. In fact, subject matter expertise is required not only to interpret the results, but also to provide the training samples for the model. In a similar vein, there is a real risk of misinterpreting or overestimating what a model is outputting, for example if the model was misconfigured or the analyst does not fully understand what the model is doing. Take the case of the sedimentation research in Tuxpan, in which it was easy to conclude that the model reported dangerous levels of sedimentation in certain precise locations. However, the nuance is important. What the model is actually saying is that, based on past measurements, dangerous levels of sedimentation are statistically likely to happen in this location rather than somewhere else. There are of course many additional factors that can cause this to change; the model can only use the data it is supplied with, which leads to the next major challenge: the lack of appropriate data.

Data needs to be appropriate for the application and the objects that need to be detected. In the example of the oil spill detection, three-band RGB imagery would not have been appropriate as the near infrared (NIR) band was needed to detect the heat differences between the oil slicks and surrounding water. Similarly, in the whales example, researchers needed a resolution of 30cm or better to detect whales. A good rule is the pixel size needs to be smaller than the subject; that is, a 10m resolution will not be sufficient to detect Baluga whales, which average 4.6m in length. On the topic of resolution, analysts need to be aware that finer resolution images require more computer resources for processing. It may be determined during the project planning phase that, to balance resources and accuracy, the data needs to be resampled into a coarser resolution. If using a pre-trained model, the analysis needs to understand which data was used to train the model and use the same data format and resolution, or else the model will produce unreliable and incorrect results or fail entirely.

Running AI/ML models in the GeoAI arena is a compute-intensive exercise. To run GeoAI and deep learning workflows within the ArcGIS Platform, Esri recommends, at a minimum, a CPU with four cores, 8GB of RAM and a dedicated GPU with 4GB of memory. The optimal requirement to run the processes on larger datasets is nearly double. The machine will also need ample storage space to hold the base data, temporary output and final results.

Using GeoAI methodologies requires a significant investment in time. The time spent labelling objects for deep learning processes or resampling datasets to fit the requirements of the model is not trivial. In fact, most time will be spent preparing the data for the model. Depending on the size, resolution and available computing power, models can take hours or days to run. Keep in mind though: a single person may be able to process one image a day, but the machine can process hundreds, usually making the time invested in preparing the data worthwhile.

Reducing risks and challenges with GIS

The potential for GeoAI to help map and understand our oceans is compelling, but only if the risks and challenges can be mitigated. GeoAI is focused on solving geographic problems, so it makes sense to select a GIS platform in which the analysis can be performed from start to finish. The GIS can ingest images from satellites, aerial and drone-borne cameras, terrestrial scanners and sonar devices. This flexibility allows the analyst to first explore the dataset to understand its format, resolution and pixel type, then focus on the application that would be best supported by this dataset.

Using the correct resolution for the object to be detected is the first step in preventing misclassification. To begin exploiting the image, there are well-documented built-in tools and Python libraries so the analyst can select the best tool for the task at hand, whether that is labelling the objects for deep learning or running an object detection workflow. These models usually require a few minor adjustments to account for the new data. Additionally, GIS writes the results of the model directly to an interactive map, facilitating the QA/QC process. Likewise, the GeoAI tools provide result metadata which helps assess the accuracy of the model. Misclassification can also be avoided by including a human in the loop prior to publishing the results.

Desktop GIS, such as ArcGIS Pro, can access data directly from cloud storage, which eliminates having to move large amounts of data locally. Furthermore, the use of spatial extent parameters available in the GIS-based tools let the analyst prepare smaller areas to test the models prior to running them on the entire dataset. This can help mitigate some of the risk pertaining to available computing resources. To help save time in the data preparation process, pre-trained models are available to establish baseline outputs and identify whether additional training samples are needed.

GeoAI has many fascinating use cases for the marine domain. Hydrographic offices and chart producers can leverage the ability to detect objects and create an automated pipeline from data collection to chart production, and organizations monitoring the effects of climate change can automate the analysis of decades worth of data to find change. The ongoing efforts of non-profit organizations can be enhanced by leveraging this new technology to protect Earth’s oceans from overfishing and contamination. This is of course not without risk, but leveraging a modern GIS platform such as ArcGIS can greatly enhance the efficiency of the analyst to go from data to results. GeoAI is an accessible method for almost anyone in the marine domain.

More information

What Is GeoAI? | Accelerated Data Generation & Spatial Problem-Solving. (n.d.). https://www.esri.com/en-us/capabilities/geoai/overview (accessed 1 February 2024).

Hains, D., Schiller, L., Ponce, R., Bergmann, M., Cawthra, H. C., Cove, K., Echeverry, P., Gaunavou, L., Kim, S., Lavagnino, A. C., Maschke, J., Mihailov, M. E., Obura, V., Oei, P., Pang, P. Y., Plackal, G. P. N., & Sharma, S. L. (2022). Hydrospatial - Update and progress in the definition of this term. The International Hydrographic Review, 28, 221–225.https://doi.org/10.58440/ihr-28-n14

Singh, R., & Singh, R. (2020, July 14). How we did it: Detecting Shipwrecks using Deep Learning at UC 2020. ArcGIS Blog. https://www.esri.com/arcgis-blog/products/arcgis-pro/analytics/detecting-shipwrecks-using-deep-learning/ (accessed 1 February 2024).

NOAA: https://oceanservice.noaa.gov/ocean/observations/data-standards.html

Esri Canada and DFO: https://storymaps.arcgis.com/stories/cd9b97dd21a84440823de71243e8b3ae

UKHO Blog Post: https://ukhodigital.blog.gov.uk/2020/02/12/creating-coastlines-using-data-science/

Global Fishing Watch: https://globalfishingwatch.org/research/

Chen Z. and Small G. W., Pattern recognition analysis of marine oil spills in airborne passive infrared multispectral remote sensing images, Analyst. (2022) 147, 5018–5027, https://doi.org/10.1039/D2AN01065H, 33341595.

Esri recommendations: https://www.esri.com/arcgis-blog/products/arcgis-pro/imagery/deep-learning-with-arcgis-pro-tips-tricks/