Computer Visioned Offshore Oil & Gas Operation, Maintenance and Decommissioning: NDT [VT] Visual…

Original article was published on Artificial Intelligence on Medium

Computer Visioned Offshore Oil & Gas Operation, Maintenance and Decommissioning: NDT [VT] Visual Testing Deep Learning with Drone and Welding Technology.

Real-Time Damage and Defect Detection with Drone Streaming Technology

1. Introduction

Visual inspection by an unmanned drone is a quick and cost-effective inspection method of critical areas on onshore and offshore constructions. Drone inspection enables access to otherwise inaccessible structures and details, offering immediate action as an alternative to more traditional, time-consuming and costly inspection methods like rope access, sky-lifts, cherry pickers and scaffolding.

Aerial inspections can be applied anywhere outdoor, indoor and offshore. Any limitations from national or local regulation must be taken into account. Furthermore, any safety procedures of the facility owner must be complied with.

The use of drones for visual inspection provides benefit such as:

  • quick overview and evaluation of hard to reach areas
  • preventive maintenance planning and optimised production
  • access to areas that pose health, safety and environmental risk to humans
  • high level of safety for personnel
  • high quality images and video — sharp and detailed photographs of defects
  • quick on-site deployment of the drone inspection system
  • authorized and qualified UAV-inspection personnel
  • reduced downtime.
Fig. 1. Example of a oil and gas platform Flare Tower Maintenance with Helicopter with old Method

There are numerous defect types that can appear on a structural mechanical from cracks and pitting to full delamination in sections. Most of these defects are either caused by an acute event such as operational and environmental load are caused by small physical impacts over time. Corrosion caused by water droplet impingement reduces structural and mechanical strength output. If left uncorrected, the structural integrity of the platform can be compromised.

2. Background

UAV of offshore oil and gas towers and flares

The offshore environment is often regarded as tough and demanding, where it is vital to keep faults as well as loss of production at both the installation and the inspection equipment to a minimum.

Drone inspection enables both pilot and NDT personnel to distance themselves from the danger zones, making a safe inspection of high-risk areas possible. Furthermore, detailed images of critical components can be obtained, without shutdown or disturbance of the production process.

One example of an assignment containing these elements is inspection of the flare and the flare tower on an oil/gas platform with an active flare. Here the drone collects images or thermographic data of critical components on the flare and the flare tower, while production continues unaffected.

The data collection takes place in a turbulent and flame-filled environment, whilst the drone pilot, inspector and employees are at a safe distance from the inspection area.

Fig 2. Drone inspection of active flare on offshore oil and gas platforms.
Fig. 2.1 Flare Tower at 170m above sea level

Air photo of an oil rig and its flare tower.

Bridge inspection by drone

The inspection drone is capable of viewing an item at any angle between +45° up and 90° down. The drone’s ability to look up is particularly useful when inspecting bridges offshore. Flying underneath the bridge and inspecting for corrosion etc. without the cost of e.g. a rope access team makes aerial inspection an attractive alternative.

Refineries — storage tanks and confined spaces

Drone inspection of boilers, fuel storage tanks and confined spaces at power plants, refineries and other industrial plants is becoming commonplace. Here visual inspections are advantageous as it often is too expensive or time-consuming to set up scaffolding, or too hazardous for physical inspection.

There are though considerable challenges when using drones in confined spaces, these include lack of proper lighting, color nuances, radio signal reflection and propeller turbulence.

We meet these challenges by using powerful LED lamps, thus ensuring good picture quality with true, natural colors, as well as using specialised equipment and experienced pilots who can predict areas with a high risk of turbulence.

The quick deployment time and ability to either screen a whole area or quickly get details of specific sections make drone inspection ideal for time critical inspection.

The data provided by the drone inspection can significantly improve maintenance planning, as it can be used to reduce maintenance-related downtime and costs.

3. Literature Review

Structural and Mechanical inspection present many challenging aspects, especially offshore: a very harsh environment, high costs and losses due to the significant duration of the inspections and turbines downtimes, the involvement of highly specialized technicians risking while climbing up and down each blade. Visual Working developed a remote inspection technique that revolutionizes this highly specialized task: using custom made UAVs (drones), Visual Working can inspect in the shortest time a Platform(onshore or offshore) acquiring super high definition photographs of all the structural and mechanical welding surface. The collected images and videos are stored in a database and processed through an imaging software that allows quick detection and reporting of early-stage damages. Trained and specialized teams of Visual Working technicians are capable of inspecting entire platform in a fraction of the time required by standard inspection methods, in total safeness, cutting costs and platform downtimes dramatically.

Fig. 3. Drone Images Classified into Weld Crack Damage Detection

4. Solution Approach(deep learning)

A new approach to platform inspection and damage detection. Frame damage detection as separated bounding boxes and associated class probabilities on the program output report generator. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Our base YOLO model processes images in real-time at 45 frames per second. Compared to state-of-the-art detection systems, YOLO makes more localization errors but is less likely to predict false positives on the background. Finally, YOLO learns very general representations of objects. It outperforms other detection methods, including DPM and R-CNN, when generalizing from natural images in real-time streams.

How to use OpenCV’s ‘dnn’ module with NVIDIA GPUs, CUDA, and cuDNN

In this tutorial, you will learn how to use OpenCV’s “Deep Neural Network” (DNN) module with NVIDIA GPUs, CUDA, and cuDNN for.

module to (1) load a pre-trained network from disk, (2) make predictions on an input image, and then (3) display the results, allowing you to build platform computer vision/deep learning pipeline for your particular project.

OpenCV 4.2 now supports NVIDIA GPUs for inference using OpenCV’s

In today’s tutorial, I show you how to compile and install OpenCV to take advantage of your NVIDIA GPU for deep neural network inference.

Assumptions when compiling OpenCV for NVIDIA GPU support

In order to compile and install OpenCV’s “deep neural network” module with NVIDIA GPU support, I will be making the following assumptions:

  1. You have an NVIDIA GPU. If you do not have an NVIDIA GPU, you cannot compile OpenCV’s “dnn” module with NVIDIA GPU support.
  2. You are using Ubuntu 18.04 (or another Debian-based distribution). When it comes to deep learning, I strongly recommend Unix-based machines over Windows systems.

Step #1: Install NVIDIA CUDA drivers, CUDA Toolkit, and cuDNN

This tutorial makes the assumption that you already have:

  • The CUDA drivers for that particular GPU installed
  • CUDA Toolkit and cuDNN configured and installed

If you have an NVIDIA GPU on your system but have yet to install the CUDA drivers, CUDA Toolkit, and cuDNN. This can be accomplished in your terminal How to install TensorFlow 2.0 on Ubuntu:

$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt-get update

Go ahead and install your NVIDIA graphics driver:

$ sudo apt-get install nvidia-driver-418

And then issue the reboot command and wait for your system to restart:

$ sudo reboot now
$ nvidia-smi

Let’s go ahead and download CUDA 10.0

$ cd ~
$ mkdir installers
$ cd installers/
$ wget
$ mv cuda_10.0.130_410.48_linux
$ chmod +x
$ sudo ./ — override

You may safely ignore the error message after the above command. Now let’s update our bash profile

$ sudo nano ~/.bashrc
# NVIDIA CUDA Toolkit(set below path on bashrc file)
export PATH=/usr/local/cuda-10.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64

see Figure 4 below for illustration purposes.

Wget and download cuDNN v7.6.4 for CUDA 10.0 from the following link:

You then may need to copy) it from your home machine to your remote deep learning box name folder called installers:

$ cd ~/installers
$ tar -zxf cudnn-10.0-linux-x64-v7.6.4.38.tgz#check with download filename
$ cd cuda
$ sudo cp -P lib64/* /usr/local/cuda/lib64/
$ sudo cp -P include/* /usr/local/cuda/include/
$ cd ~

Step #2: Install OpenCV and “dnn” GPU dependencies

How to use OpenCV’s ‘dnn’ module with NVIDIA GPUs, CUDA, and cuDNN

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install build-essential cmake unzip pkg-config
$ sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
$ sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
$ sudo apt-get install libv4l-dev libxvidcore-dev libx264-dev
$ sudo apt-get install libgtk-3-dev
$ sudo apt-get install libatlas-base-dev gfortran
$ sudo apt-get install python3-dev

Step #3: Download OpenCV source code

There is no “pip-installable” version of OpenCV that comes with NVIDIA GPU support — instead, we’ll need to compile OpenCV from scratch with the proper NVIDIA GPU configurations set.

The first step in doing so is to download the source code for OpenCV v4.2:

$ cd ~
$ wget -O
$ wget -O
$ unzip
$ unzip
$ mv opencv-4.2.0 opencv
$ mv opencv_contrib-4.2.0 opencv_contrib

Step #4: Configure Python virtual environment

$ wget
$ sudo python3
$ sudo pip install virtualenv virtualenvwrapper
$ sudo rm -rf ~/ ~/.cache/pip

You then need to open up your ~/.bashrc file and update it to automatically load virtualenv/virtualenvwrapper whenever you open up a terminal.

$ mkvirtualenv opencv_cuda -p python3
$ $ sudo nano ~/.bashrc
# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/

Figure 4: How to install TensorFlow 2.0 on Ubuntu with an NVIDIA CUDA GPU.

If you ever close your terminal or deactivate your Python virtual environment, you can access it again via the workon command:

$ workon opencv_cuda
$ pip install numpy

Step #5: YOLO: Detection Using A Pre-Trained Model

Download Darknet from the link given below:-

$ git clone
$ cd darknet
mkdir build-release
cd build-release
cmake ..
make install

You already have the config file for YOLO in the cfg/ subdirectory. You will have to download the pre-trained weight file here (237 MB). Or just run this:


Step #6:Download Pretrained Convolutional Weights

For training we use convolutional weights that are pre-trained on Imagenet. We use weights from the darknet53 model. You can just download the weights for the convolutional layers here (76 MB).


save these pre-trained files on darknet the cfg/ subdirectory.

Step #7: LabelImg

LabelImg is a graphical image annotation tool. It is written in Python and uses Qt for its graphical interface. Annotations are saved as XML files in PASCAL VOC format, the format used by ImageNet. Besides, it also supports YOLO format. collect image from source and start labeling them as shown below:-

  1. #for windows
  2. #for Linux, ubuntu

Figure 5: How to label image on YOLO format with predefined classes.

Labelimg softwaredata/predefined_classes subdirectory its .txt file where you define your damage classes. Make sure you select YOLO format and save them after you have mark them as shown above in .txt format.

#Predefine your Classes

Step #8: How to train (to detect structural mechanical damages):

Training Yolo v3:

  1. For training cfg/yolov3-custom.cfg download the pre-trained weights-file: darknet53.conv.74 (see step 5)
  2. Create file yolo-obj.cfg with the same content as in yolov3-custom.cfg and:
  • change line batch to batch=64
  • change line subdivisions to subdivisions=32
  • change line max_batches to (classes*2000 but not less than the number of training images, and not less than 8000), f.e. max_batches=8000 if you train for 4 classes
  • change line steps to 80% and 90% of max_batches, f.e. steps=6400,7200
  • set network size width=416 height=416 or any value multiple of 32:
  • change line classes=80 to your number of objects in each of 4[yolo]-layers:
  • change [filters=255] to filters=(classes + 5)x3 in the 4[convolutional] before each [yolo] layer, keep in mind that it only has to be the last [convolutional] before each of the [yolo] layers.

So if classes=1 then should be filters=18. If classes=4 then write filters=27.


Create file in the directory \darknet\data\, containing (where classes = number of objects):

classes= 4
train = data/train.txt
valid = data/test.txt
names = data/obj.names
backup = backup/

Create file obj.names in the directory \darknet\data\, with objects names – each in new line

  1. Put image-files (.jpg) of your objects in the directory \darknet\data\obj\
  2. You should label each object on images from your dataset. Use this visual GUI-software(see step Labelimg above) for marking bounded boxes of objects and generating annotation files for Yolo v3:

It will create .txt-file for each .jpg-image-file – in the same directory and with the same name, but with .txt-extension, and put to file: object number and object coordinates on this image, for each object in new line:

<object-class> <x_center> <y_center> <width> <height>


  • <object-class> – integer object number from 0 to (classes-1)
  • <x_center> <y_center> <width> <height> – float values relative to width and height of image, it can be equal from (0.0 to 1.0]
  • for example: <x> = <absolute_x> / <image_width> or <height> = <absolute_height> / <image_height>
  • atention: <x_center> <y_center> – are center of rectangle (are not top-left corner)

For example for img1.jpg you will be created img1.txt containing:

1 0.716797 0.395833 0.216406 0.147222
0 0.687109 0.379167 0.255469 0.158333
1 0.420312 0.395833 0.140625 0.166667
  1. Create file train.txt in directory darknet\data\, with filenames of your images, each filename in new line, with path relative to darknet, for example containing:
  1. Download pre-trained weights for the convolutional layers and put to the directory darknet\cfg
  • foryolov4-custom.cfg : darknet53.conv.74)
  1. Create darknet\data\obj\train.txtby using the command line: python
  2. To start train on Linux use command: ./darknet detector train data/ yolov3-custom.cfg darknet53.conv.74
  • (file yolo-obj_last.weights will be saved to the build\darknet\backup\ for each 100 iterations)
  • (file yolo-obj_xxxx.weights will be saved to the build\darknet\backup\ for each 1000 iterations)

For training with mAP (mean average precisions) calculation for each 4 Epochs (set valid=valid.txt or train.txt in file) and run: ./darknet detector train data/ yolov3-custom.cfg darknet53.conv.74 -map

Step #9: How to test (to detect structural mechanical damages on real-time streaming by drone):

Training Yolo v3:

  1. For training cfg/yolov3-custom.cfg download the pre-trained weights-file: darknet53.conv.74 (see step 5)
  2. Create file yolo-obj.cfg with the same content as in yolov3-custom.cfg and:
  • After each 100 iterations you can stop and later start training from this point. For example, after 2000 iterations you can stop training, and later just start training using: ./darknet detector train data/ yolo-obj.cfg backup\yolo-obj_2000.weights Erosion.jpg
  • (in the original repository the weights-file is saved only once every 10 000 iterations if(iterations > 1000))
  • uring training you see nan values for avg (loss) field – then training goes wrong, but if nan is in some other lines – then training goes well.

Note: If you changed width= or height= in your cfg-file, then new width and height must be divisible by 32.

Note: After training use such command for detection: darknet.exe detector test data/ yolo-obj.cfg yolo-obj_8000.weights

Note: if error Out of memory occurs then in .cfg-file you should increase subdivisions=16, 32 or 64: link

When should I stop training:

Usually sufficient 2000 iterations for each class(object), but not less than number of training images and not less than 6000 iterations in total. But for a more precise definition when you should stop training, use the following manual:

  1. During training, you will see varying indicators of error, and you should stop when no longer decreases 0.XXXXXXX avg:

Region Avg IOU: 0.798363, Class: 0.893232, Obj: 0.700808, No Obj: 0.004567, Avg Recall: 1.000000, count: 8 Region Avg IOU: 0.800677, Class: 0.892181, Obj: 0.701590, No Obj: 0.004574, Avg Recall: 1.000000, count: 8

9002: 0.211667, 0.60730 avg, 0.001000 rate, 3.868000 seconds, 576128 images Loaded: 0.000000 seconds

  • 9002 — iteration number (number of batch)
  • 0.60730 avg — average loss (error) — the lower, the better

When you see that average loss 0.xxxxxx avg no longer decreases at many iterations then you should stop training. The final avgerage loss can be from 0.05 (for a small model and easy dataset) to 3.0 (for a big model and a difficult dataset).

Or if you train with flag -map then you will see mAP indicator Last accuracy mAP@0.5 = 18.50% in the console – this indicator is better than Loss, so train while mAP increases.

  1. Once training is stopped, you should take some of last .weights-files from \darknet\backup and choose the best of them:

For example, you stopped training after 9000 iterations, but the best result can give one of previous weights (7000, 8000, 9000). It can happen due to overfitting. Overfitting — is case when you can detect objects on images from training-dataset, but can’t detect objects on any others images. You should get weights from Early Stopping Point:

To get weights from Early Stopping Point:

2.1. At first, in your file you must specify the path to the validation dataset valid = valid.txt (format of valid.txt as in train.txt), and if you haven’t validation images, just copy data\train.txt to data\valid.txt.

2.2 If training is stopped after 9000 iterations, to validate some of previous weights and comapre last output lines for each weights (7000, 8000, 9000):

Choose weights-file with the highest mAP (mean average precision) or IoU (intersect over union)

For example, bigger mAP gives weights yolo-obj_8000.weights – then use this weights for detection.

Or just train with -map flag:

./darknet detector train data/ yolov3-custom.cfg darknet53.conv.74 -map

So you will see mAP-chart (red-line) in the Loss-chart Window. mAP will be calculated for each 4 Epochs using valid=valid.txt file that is specified in file (1 Epoch = images_in_train_txt / batch iterations)

(to change the max x-axis value — change max_batches= parameter to 2000*classes, f.e. max_batches=6000 for 3 classes)

Example of custom object detection: ./darknet detector test data/ yolov3-custom.cfg yolo-obj_8000.weights

  • IoU (intersect over union) — average instersect over union of objects and detections for a certain threshold = 0.24
  • mAP (mean average precision) — mean value of average precisions for each class, where average precision is average value of 11 points on PR-curve for each possible threshold (each probability of detection) for the same class (Precision-Recall in terms of PascalVOC, where Precision=TP/(TP+FP) and Recall=TP/(TP+FN) ), page-11:

mAP is default metric of precision in the PascalVOC competition, this is the same as AP50 metric in the MS COCO competition. In terms of Wiki, indicators Precision and Recall have a slightly different meaning than in the PascalVOC competition, but IoU always has the same meaning.