Data Augmentation techniques in Python

Source: Deep Learning on Medium

Data Augmentation techniques in Python

A sneak peak into data augmentation techniques in python using imgaug

In this article we will explore different data augmentation techniques in Python using imgaug library

What is Image Augmentation?

Image augmentation is a very powerful technique to artificially create variations in existing images to expand the existing image dataset. This creates new and different images from the existing image dataset that represents comprehensive set of possible images.

This is done by applying different transformation techniques like zooming the existing image or rotating the existing image by a few degrees or , shearing or cropping the existing set of images etc.

Why do we need Image Augmentation?

Deep learning Convolutional Neural Networks(CNN) need huge number of images for the model to be trained effectively. This helps to increase the performance of the model by generalizing better and there by reducing over fitting. Most popular datasets for classification and object detection datasets have a few thousands to millions of images.

Generalizing refers to a model’s performnace when evaluated on previously seen data during training against the test or validation data it has never seen before.

CNN’s due to it’s invariance property can classify objects even when visible in different size, different orientations or different illumination. Hence, if we take the small dataset of images and transform the objects to different sizes by zooming in or zooming out, flipping them vertically or horizontally or shaping the brightness whatever makes sense for the object. This way we create a rich, diverse dataset with variations.

Image Augmentation creates a rich, diverse set of images from a limited small set of images for image classification or object detection or image segmentation.

With image augmentation we increase the size of the existing small data set of images by adding different variations. This increases the diversity in the images.

Augmentation strategies to increase the size of the training dataset needs to be applied after carefully understanding the problem domain.

for example if we are classifying different types of cars then applying image augmentation by flipping a car vertically is not useful for the problem domain or when we are doing a classification of hand written digits flipping a 6 vertically does not make sense

When to apply Image Augmentation?

Image augmentation can be applied as a pre-processing step before we train the model or can be applied in real time.

Offline or preprocessing Augmentation

Augmentation is applied as a preprocessing step to increase the size of the dataset. This is usually done when we have a small training dataset that we want to expand.

Generating augmentation on smaller dataset is helpful but we need to consider the disk space when applying on larger datasets

Online or real time Augmentation

As the name suggests augmentation is applied in real time. This is usually applied for larger datasets as we do not need to save the augmented images on the disk.

In this case we apply transformation in mini-batches and then feed it to the model.

Online augmentation model will see a different images at each epoch.In Offline augmentation, augmented image is part of the training set, it see sthe augmented image multiple times depending on the number of epochs.

Model generalizes better with online augmentation as it see more samples during training with online data augmentation.

We will be using imgaug class for demonstrating Image Augmentation.

imgaug supports wide range of data augmentation techniques

Basic data augmentation techniques

  • Flipping : flipping the image vertically or horizontally
  • Rotation : rotates the image by a specified degree.
  • Shearing : shifts one part of the image like a parallelogram
  • Cropping : object appear in different positions in different proportions in the image
  • Zoom in , Zoom out
  • Changing brightness or contrast

We will now explore these data augmentation techniques using imgaug library

imgaug

imgaug is a library for image augmentation along with keypoints/landmarks, bounding boxes, heatmaps and segmentation maps.

Installing the library

pip install imgaug

In some cases we get error with Shapely in that case we can try the below command

pip install imgaug — upgrade — no-deps

We will take an image and transform it using basic data augmentation techniques.

Importing required libraries

import imageio
import imgaug as ia
import imgaug.augmenters as iaa
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as patches
import matplotlib
%matplotlib inline

Displaying the original image

We display the original image using imageio

image = imageio.imread(“.\\car2.jpeg”)
ia.imshow(image)
Original Image

Rotating the image

We can rotate the image by specifies degree. We are rotating the image by -50 degree to 30 degrees

rotate=iaa.Affine(rotate=(-50, 30))
rotated_image=rotate.augment_image(image)

ia.imshow(rotated_image)
Image rotated by -50 to 30 degrees

Adding noise to the image

We add different noise values sampled from gaussian distributions element wise to images.

gaussian_noise=iaa.AdditiveGaussianNoise(10,20)
noise_image=gaussian_noise.augment_image(image)
ia.imshow(noise_image)
Adding Gaussian noise

Cropping the image

cropping remove columns/rows of pixels at the sides of images. crops one side by 30%

crop = iaa.Crop(percent=(0, 0.3)) # crop image
corp_image=crop.augment_image(image)

ia.imshow(corp_image)
cropped one side of the image by 30%

Shearing the image

Shearing the image by 0 to 40 degrees

shear = iaa.Affine(shear=(0,40))
shear_image=shear.augment_image(image)
ia.imshow(shear_image)
Sheared image by 0 to 45 degrees

Flipping the image

We can flip the image vertically or horizontally. Fliplr flips images horizontally

#flipping image horizontally
flip_hr=iaa.Fliplr(p=1.0)
flip_hr_image= flip_hr.augment_image(image)
ia.imshow(flip_hr_image)
Flipping image horizontally

Flipup flips image vertically

flip_vr=iaa.Flipud(p=1.0)
flip_vr_image= flip_vr.augment_image(image)
ia.imshow(flip_vr_image)
Flipping image vertically

Changing brightness of the image

We adjust the image brightness using GammaContrast, by scaling pixel values. Values in the range gamma=(0.5, 2.0) seem to be sensible. We can use SigmoidContrast or LinearContrast also for changing the brightness of the image

image = imageio.imread(“.\\img Aug\\car2.jpeg”)
contrast=iaa.GammaContrast(gamma=2.0)
contrast_image =contrast.augment_image(image)
ia.imshow(contrast_image)
adding contrast to the image

Scaling the image

We can zoom in or zoom out images using scale. We have scaled the image below to 150% to 80% of the image height/width. we can scale each axis independently

image = imageio.imread("C:\\Data\\img Aug\\car2.jpeg")
scale_im=iaa.Affine(scale={"x": (1.5, 1.0), "y": (1.5, 1.0)})
scale_image =scale_im.augment_image(image)
ia.imshow(scale_image)
scaling image 150% to 80% of its height/width.

Augmentation for Object detection

We draw bounding boxes for Object detection. When we augment the image we want the bounding boxes also to be updated accordingly.

imgaug offers support for bounding boxes. When we rotate, shear or crop the image, bounding box around the object is also updated accordingly.

importing bounding boxes from imgaug

from imgaug.augmentables.bbs import BoundingBox, BoundingBoxesOnImage

Initializing the bounding box around the original image

bbs = BoundingBoxesOnImage([
BoundingBox(x1=10, x2=520, y1=10, y2=300)
], shape=image.shape)

Displaying the bounding box on top of the original image

ia.imshow(bbs.draw_on_image(image, size=2))

We move the image using translate_percentage, augment the bounding box and apply it over the image

move=iaa.Affine(translate_percent={"x": 0.1}, scale=0.8)
image_aug, bbs_aug = move(image=image, bounding_boxes=bbs)
ia.imshow(bbs_aug.draw_on_image(image_aug, size=2))
bounding box on the augmented image

Handling Bounding boxes outside the image after applying image augmentation

Bounding box may sometime go outside the image and we need code to handle such a scenario

We rotate the image and try to draw the bounding box around the object

rotate_bb=iaa.Affine(rotate=(-50, 30))
image_aug, bbs_aug = rotate_bb(image=image, bounding_boxes=bbs)
ia.imshow(bbs_aug.draw_on_image(image_aug, size=2))

Portions of bounding box are outside the image. In the code below we will

  • Remove the bounding box fully or partially outside the image
  • Clip the bounding boxes that are partially outside so that they are fully inside the image

We create a padding function to pad image with a 1px white and 1px black border

def pad(image, by):
image_border1 = ia.pad(image, top=1, right=1, bottom=1, left=1,
mode="constant", cval=255)
image_border2 = ia.pad(image_border1, top=by-1, right=by-1,
bottom=by-1, left=by-1,
mode="constant", cval=0)
return image_border2

We then draw the bounding boxes on the image. We first extend the image plane by BORDER pixels and then mark the bounding boxes inside the image plane

def draw_bbs(image, bbs, border):
GREEN = [0, 255, 0]
ORANGE = [255, 140, 0]
RED = [255, 0, 0]
image_border = pad(image, border)
for bb in bbs.bounding_boxes:
if bb.is_fully_within_image(image.shape):
color = GREEN
elif bb.is_partly_within_image(image.shape):
color = ORANGE
else:
color = RED
image_border = bb.shift(left=border, top=border)\
.draw_on_image(image_border, size=2, color=color)
return image_border

We now apply the same rotation to the image and draw he bounding box

rotate=iaa.Affine(rotate=(-50, 30))
image_aug, bbs_aug = rotate(image=image, bounding_boxes=bbs)
image_after = draw_bbs(image_aug, bbs_aug.remove_out_of_image().clip_out_of_image(), 100)
ia.imshow(image_after)

In the next article we will explore Keras ImageDataGenerator for image augmentation

References:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

https://nanonets.com/blog/data-augmentation-how-to-use-deep-learning-when-you-have-limited-data-part-2/

https://github.com/Lexie88rus/augmentation-packages-overview/blob/master/data-augmentation-packages-overview.ipynb

https://link.springer.com/content/pdf/10.1186%2Fs40537-019-0197-0.pdf