Subsea Autonomy is Moving Beyond Waypoints - 09/06/2020

Underwater an AUV Must Rely Upon its Own Sensor Network

The AUV has been the tool of choice for shallow-water MCM and EOD operations for a number of years. The coverage by popular news channels of unfortunate events, such as the MH370 tragedy, has also shown that the AUV has undisputedly become the tool of choice to carry out salvage operations at extreme depths.

The AUV has no physical link to the surface and the majority of missions are still pre-programmed by a human, which can be time consuming and error prone. Missions are normally a sequence of defined waypoints where the AUV may be required to carry out certain specific manoeuvres to achieve each waypoint.

Recovered and Analysed

The AUV operator runs through specific checks to ensure that the AUV is ready before downloading the mission to the AUVs guidance computer. The AUV is then launched and proceeds to run through each of the assigned waypoints. Once the mission is over, the AUV returns to a predetermined location and is recovered by the AUV operator.  When the AUV is back on the vessel, the gathered data is recovered and analysed. This is also the time to swap or recharge the batteries. The next mission can be loaded and the process repeated.

The development of new AUV technology means that many operators are now seeking to improve the range of tasks that AUVs can tackle and the length of their operation.  For example, Oil and Gas is driving a requirement for infield AUV deployment, often as a hybrid AUV/ROV concept. In this scenario, an AUV can reside and operate an oilfield with varying levels of human interaction, with the long-term goal being that the AUV is able to carry out inspection and maintenance tasks. The industry has made significant strides towards this vision but much more work remains to be done. Another active example is the development of large diameter AUVs capable of being deployed for extended periods of time and operating without any human intervention. In this case, it is mainly oceanographic and military requirements that are driving this technology.

Figure 1: A typical scene in planning missions, this time for multiple UUVs.
Figure 1: A typical scene in planning missions, this time for multiple UUVs.

No Easy Use of Waypoints

However, these new concepts do not fit easily into the concept of defining a mission as a series of pre-planned waypoints because the operator cannot know, in advance, everything that might occur during the mission. Picture 2 and Picture 3 provide examples where waypoints simply cannot be used. These mission profiles require true adaptive autonomy, where the AUV monitors its surroundings with a range of sensors (e.g. video, sonar, navigation, laser), interpreting the incoming information, matching this against mission goals, and making real-time decisions about the best course.

This rest of this article will look at some of the advances in perception, such as deep learning sensor processing, which are starting to allow truly adaptive autonomy to be deployed onto AUVs.  The article will also consider where the technology needs to continue developing, and how this might fit alongside competing concerns such as safety, and security of subsea operations.

Figure 2: Subsea 7’s AIV performing a mid-water riser inspection using real-time sonar processing. (Image courtesy: Subsea 7)
Figure 2: Subsea 7’s AIV performing a mid-water riser inspection using real-time sonar processing. (Image courtesy: Subsea 7)

There are two major challenges that differentiate subsea autonomy from any other domain: navigation and communication. The physical limitations of working underwater are well known, but the lack of global navigation information and the limits of underwater communication provide a unique challenge to subsea autonomy. These limitations also extend to the sensing technology.  As an example, it is possible to cheaply equip a hobbyist drone with a GPS and HD video camera, and have this data stream back to a ground control station. These drones are able to efficiently execute a waypoint-based mission with full human oversight. In contrast, when underwater, an AUV has no global navigation system to leverage and must rely upon its own sensor network to understand the environment around it, posing a far more challenging robotics problem.

‘Eyes and Ears’

The AUVs perception and navigation algorithms on board the vehicle become the ’eyes and ears’ of the AUV, helping it understand its local environment locate itself within that scene, and make decisions.  Regardless of how often a human intervenes, or the sophistication of the vehicle autonomy engine, poor quality perception will almost inevitably lead to incorrect analysis, and mission failure.

The recent advances in artificial intelligence (AI) technology, largely driven by Silicon Valley technology companies, has revolutionised the performance of modern machine learning algorithms.  Deep Learning is a good, and well publicized, example of these systems.  This technique relies on using extensive data sets, tagged with features, that are fed into multi-layer (deep) neural networks.  When the technology is applied correctly, there can be an order of magnitude improvement in the performance of the perception algorithm over historical approaches. 

Figure 3: A small ROV performing precise manoeuvres relative to an object in shallow water. (Image courtesy: NIWC)
Figure 3: A small ROV performing precise manoeuvres relative to an object in shallow water. (Image courtesy: NIWC)

SeeBytes ongoing work developing sonar-based target detection and recognition algorithms has seen this level of improvement, with the most significant improvement seen in the reduction of false positives reported by the algorithm. There are undoubtedly numerous other advances in automated perception, from applications such as autonomous cars, that are also adaptable to the subsea domain and should eventually be leveraged.

However, developing the deep learning target detection and recognition algorithms flagged a potential issue in subsea applications – the volume of required data.  Deep Learning approaches, like most modern AI algorithms, rely on terabytes of data for training.  Ideally, the training images should contain possible variations in appearance of the objects being identified, including pose, lighting and environmental effects. The good news is that alongside Deep Learning, Silicon Valley has also advanced Deep Fake technology.  This is the ability of multi-layer neural networks to learn how to fake” data with remarkable degrees of accuracy. 

SeeByte has pioneered the ability to take a small data set of real subsea images, and use this to train a Deep Fake system.  The Deep Fake technology is then used to produce a much larger data set, allowing the Deep Learning target system to be trained on a mix of both the real and fake data.  Picture 4 provides an example of fully synthetic side-scan.  The improvements in performance were remarkable and the quality of perception, coupled with ever improving sensing technology, starts to allow more adaptive AUV missions to be routinely used.

Deep Learning Perception

This example does raise one obvious concern because the deep learning perception is trained by the deep learning simulator. While the initial results offer a remarkable step improvement in performance, the difficulty is in understanding where and why the system will fail. This is normally termed predictability and explainability. There is significant research into these aspects of artificial intelligence, and it is hoped that the situation will improve as operator trust in autonomous systems is directly linked to predictability and explainability. This issue represents the broader problem of safety cases involved in deploying artificial intelligence. While the issue of safety cases is much more complex than a short article can address, it is clear that a gradual introduction of adaptive autonomy is going to be crucial to success. The best parallel is the autonomous car, which has been gradually introducing technology, starting with warning technologies (proximity detection), before moving to assist technology (adaptive cruise control).  These assist technologies are slowly increasing in complexity (automated lane assist), as trust and understanding of the systems grow.

Looking back through the history of the ROV introduces another aspect where technology needs to continue developing. The standardisation of many subsea interfaces, mainly mechanical and electrical, dramatically increased the number of tasks that an ROV could accomplish.  This standardisation is now needed at the software component level, to allow better interoperability of AUV technology.  This applies to software interfaces both within the vehicle and between other pieces of subsea equipment.

Figure 4: Fully synthetic side-scan image.
Figure 4: Fully synthetic side-scan image.

Another way of looking at this is to take a practical example where a commercial company has carried out extensive investment that has led to the development of a riser inspection autonomy behaviour, and this has been demonstrated on a single AUV.  Having invested heavily in this autonomy behaviour, it is preferable that this autonomy behaviour can be used wherever required. In the same manner that a different INS can be fitted to an ROV should the job require it, an AUV should also be capable of being fitted with the appropriate autonomy behaviours to perform the task.

This standardisation is going to be a slow process, particularly given that autonomy technology is still maturing at a rapid rate. However, it is important that the broader industry acknowledges that flexibility is vital to the continued success of the AUV, and that building flexible toolkits of software technology is a crucial part of this success.

In conclusion, it is clear that the real value of AUV technology is finally starting to be realised in a broader set of operational circumstances.  However, in a similar vein to the introduction of ROV technology in the 1970s, there needs to be a gradual maturing of the technology, with incremental improvements tackling additional scope step-by-step.


Last updated: 09/06/2020