2. Introduction to Computer Vision with Deep Learning: Hands-on Kernels With openCV

Source: Deep Learning on Medium


Go to the profile of Deep Learner

Prerequisite- In case you missed the first session, please go ahead to the previous session on kernels and come back here.

A long long time ago (2012), when the computer community was plagued by the great war between computer vision scientists and deep learning scientists, there was one library that gave the winning edge to computer vision scientists, the legendary openCV. Today as the computer vision is defeated and deep learning took over, we barely use openCV in deep learning but still, it does have some brilliant image processing power.

Now we know that kernels (try recollecting the magnificent juicy apple) are all about matrices and the values inside these matrices decide what feature we want to extract, now let’s try to manipulate these matrices to check how they really apply to a real world image. In this article, we’ll try to visualize kernels and their behavior using OpenCV library of Python. Keep in mind that we do not use it in our deep learning algorithms but nonetheless, it’s a great place to gain some insight into the world of kernels.

So we’ll first look at the very easy process of initial environment setup.

NOTE: If you are aware of Colab or already have OpenCV installed then move to the next section.

Environment Setup (Recommended):

Google Colaboratory:

Colaboratory (Colab) is a cloud-based platform provided by Google which allows you to write interactive python code (imagine Jupiter notebook but on a cloud). You can share it with your friends, have multiple people edit and run the same notebook and so on. It has most of the libraries needed for ML and DL preinstalled as well.

Colab provides 48.97 GB of disk size, 12.72 GB of RAM and extremely powerful Tesla T4 GPUs. And all of that for absolutely no cost. You only need to sign in from your Google account.

After we’ve covered more ground, we’ll write a separate blog on Colab, but for now, this information is more than enough for us to proceed.

All right then, time for some hands-on-

1. Make sure that you are logged in to your google account.

2. Now Head to colab.research.google.com/.

3. Click on ‘new python 3 notebooks’ at right bottom.

Lo and behold, we have our very own cloud-based virtual notebook ready. We are now ready to write and execute code on colab.

Local Machine:

NOTE: We highly recommend you to follow with us on colab. It not only has added benefits of computational power and free resource management, but we will also require high GPU memory when we move onto Object detection, Image segmentation, Image captioning, etc later on in this publication. It’s best to get used to colab now so that later on, it becomes very easy for you.

If you do not want to use google Colab for some crazy reason (or maybe you are rich and want to show off your GPU) then you can always use your local machine. Follow the following steps:

1. Make sure python is installed on your system (I mean obviously)[if not, download python and follow these simple instructions].

2. Head to command prompt (CMD) and run following command: ‘pip install OpenCV-python’. Some cool looking hacker type stuff will happen in CMD window and you might be prompted to enter (y/n). Keep entering ‘y’ for such prompts and OpenCV should get installed.

To verify, fire up whichever python shell you use, and execute “import cv2”. If it executes without any errors then OpenCV is installed successfully.

Note: we highly recommend you to use google Colab as it is easy, hassle free and has a lot of computing power.

It is also strongly recommended that you have a GitHub account as well. If you do not know what GitHub is a or how to do make/clone GitHub repository then please follow our next sessions and just for this article, just follow along with us without GitHub.

Now it’s time for us to head to the core section of our article and get our hands dirty.

Practice time:

In case you’ve decided to skip the above section and jumped directly here, then don’t worry, you should be good to go as long as you have OpenCV installed and an interpreter ready.

From this point onwards, we would assume that you are either on Colab ‘python 3 notebook’ or python interpreter on your local machine.

For this session, you do not need to worry about the codes you do not understand (contrary to what people believe, coding is usually the least important part). Our main aim is to focus on the matrices and the type of kernel they represent.

Lets import few libraries which we will require to work:

import numpy as np
import cv2
import matplotlib.pyplot as plt
frm urllib.request import urlopen

In case any of those libraries are not successfully imported, follow along with these links, which will guide to successfully install and then, later on, you can import them successfully.

For urllib : [Instruction Link]

For Matplotlib : [Instruction Link]

For Numpy : [Instruction Link]

For cv2 : [Instruction Link]

Now that all the essentials are imported, let’s write some code. Firstly we need to write a function which can read an image URL and convert into an array. Why array? Simply because the machine is dumb and it only understands numbers.

def url_to_image(url):
# download the image, convert it to a NumPy array, and then read
 # it into OpenCV format
 resp = urlopen(url)
 image = np.asarray(bytearray(resp.read()), dtype="uint8")
 image = cv2.imdecode(image, cv2.IMREAD_COLOR)
 # return the image
return image

now let’s call this function by passing some URL as argument. We’ve taken URL of a boring image, feel free to try it on your friend’s face (it’s hilarious).

image = url_to_image("https://fontsarena-cd5e.kxcdn.com/wp-content/uploads/2019/04/helvetica-now-font-400x364.png")

Now to see what image we just downloaded:

## TYPE THIS ONLY ON COLAB !!!
from
google.colab.patches import cv2_imshow
cv2_imshow(image)
## TYPE THIS ONLY ON LOCAL SYSTEM !!!
plt.imshow(image)

Let’s try out some kernels on this beautiful “Helvetica” font image.

For starters, let’s design a kernel which will extract all the vertical lines from the image or simply saying, let’s build a vertical edge detector.

kernel = np.float32([[-1,2,-1],[-1,2,-1],[-1,2,-1]])
dst = cv2.filter2D(image,-1,kernel)

cv2_imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING COLAB.
plt.imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING LOCAL MACHINE.

Wow, looks cool, isn’t it? And you just made your first kernel. Congratulations!!! This is something we call a feature map, because we have the features we wanted from the image, and their original location on an image as well. We will discuss more on feature map when we cover channels in next sessions.

Now let’s build a horizontal edge detector?

Ahhaann… I see you are as excited as we are, so let’s jump right into?

kernel = np.float32([[-1,-1,-1],[2,2,2],[-1,-1,-1]])
dst = cv2.filter2D(image,-1,kernel)
cv2_imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING COLAB.
plt.imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING LOCAL MACHINE.

Well, well! Aren’t you good? In fact, you are a special one.

CHALLENGE TIME: Build a diagonal (either) edge detector. We will specially mention credits to the best one in our next session.

So now you have realized that just a small 3×3 matrix can do a lot of things in an instant magic touch.

Let me ask you a question. Have you ever wondered how blur tool works in Photoshop or any photo tool that you use in your mobile or pc? What about sharpening tool?

And why does it seriously work? Any answers? Yes!!!! You are right. We are back to kernels again. Not wasting any more time, let’s build a blur and sharping kernel.

kernel = np.float32([[1/11,1/11,1/11],[1/11,1/11,1/11],[1/11,1/11,1/11]])
dst = cv2.filter2D(image,-1,kernel)
cv2_imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING COLAB.
plt.imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING LOCAL MACHINE.

Wow, this really was simple enough. The blurring we did here is called an average blurring, there are many others such as Gaussian blur, Median blur, Bilateral blur (try googling the last one and using it, it’s really cool)

Let’s move on and build a sharpening kernel.

kernel = np.float32([[0,-1,0],[-1,5,-1],[0,-1,0]])
dst = cv2.filter2D(image,-1,kernel)
cv2_imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING COLAB.
plt.imshow(dst) ## TYPE THIS LINE ONLY IF YOU ARE USING LOCAL MACHINE.

Wow! This was good!

CHALLENGE TIME: Build a special kernel that does something unique (extracts weird features maybe?) and does not exist on the web (obviously). Comment below along with the kernel output. We will post the most special unique kernel credits in the next sessions and publish by your name. [Hint: Try all weirdest combinations in the matrix to see what if you can crack a new discovery].

You are encouraged to play around with the numbers and do some wizardry yourself.

There is a reason these number in matrices are returning what they are returning, but to cover it all in one article is impossible. Also as we are not going to use OpenCV kernels anyway, so there’s no point in delving deeper into them.

To summarize, we experienced kernels in real life image and are now fully aware of the capabilities of a small 3X3 matrix. These 9 numbers were enough to manipulate the image in so many different ways. We’ll soon see that the powers of 3X3 matrices are far more superior.

For those who read the session on the phone or because of some reasons could not have the experience to try by themselves, here is a GitHub link you can check out all the kernel and its transformations [Link].

Hope you enjoyed this session as much as we did. See you soon…