Line Follower Robot using CNN

Source: Deep Learning on Medium


Go to the profile of nawaz ahmad

Hi all, in this tutorial we’ll be learning how to make a line follower robot. Although, there are a plenty of line follower tutorials out there as this concept in itself is quite old. However, here we’ll learn how to detect the line using Convolutional Neural Networks (CNN). Basically, we will be capturing a series of images at a predetermined interval using our Raspberry Pi camera, then we’ll used a pre-trained CNN to predict the direction in which our robot should move i.e. forward, right, or left. You will require following things for this tutorial:-

  1. Raspberry Pi board,
  2. Pi camera,
  3. Jumper wires
  4. Chassis, motors, tyres
  5. Motor control IC (L293d)

I am assuming that you are already aware of how to control the motors using GPIO pins of your Raspberry Pi. In case you want to brush up your skills a little please go through my earlier tutorial. Also, make sure you have tensorflow 1.1 and open CV installed on your Pi before proceeding further.

We will divide this tutorial in to three sections:-

  1. Capture the images for CNN.
  2. Train the CNN
  3. Deploy the CNN on the raspberry PI

Capture the images for CNN

We require three sets of images for each of the three conditions i.e. forward, left, and right to train our CNN. Once we train our CNN, it will be able to predict in which direction the robot is moving and then we can take corrective measure accordingly. For e.g. if there is a left turn of line, the CNN will predict that robot is going in right direction relative to line and therefore, we should move it in left direction.

Fig. depicting robot moving in various directions and the corresponding images captured by the camera

There may be various ways in which you can create a data-set for training CNN. In my case, I first built a track. Then, I used the code mentioned below to capture a series of images at an interval of .5 seconds. Then you place the robot at different positions on the track and capture a series of images. For e.g. in the first image, I have placed the robot such that I want the CNN to detect that robot should move in left direction. Similarly you repeat this process at different positions on the track. Once done you can repeat this process for forward and right direction. Once you capture the images for the three directions, place them in separate folders with names forward, left, and right respectively.

# import the necessary packages
from picamera.array import PiRGBArray
from picamera import PiCamera
import time
import cv2

# initialize the camera and grab a reference to the raw camera capture
camera = PiCamera()
camera.resolution = (640, 480) # set the resolution
camera.framerate = 32 # set the frame rate
rawCapture = PiRGBArray(camera, size=(640, 480))

# allow the camera to warm up
time.sleep(0.1)

# capture frames from the camera
start = 1
for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
# grab the raw NumPy array representing the image, then initialize the timestamp and occupied/unoccupied text
image = frame.array
# show the frame
cv2.imshow("Frame", image)
key = cv2.waitKey(1) & 0xFF
cv2.imwrite(str(start) + ".jpg", image)
start = start + 1

# clear the stream in preparation for the next frame
rawCapture.truncate(0)

# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
time.sleep(.5)

Train the CNN

You can train the CNN either on raspberry PI itself or on a different more powerful system and then save the trained model which can be read by the Pi. Following is the code for training your CNN on the images captured.

# import the necessary packages
from keras.models import Sequential
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.layers.core import Activation, Flatten, Dense
from keras import backend as K
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing.image import img_to_array
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split
from keras.utils import to_categorical
from imutils import paths
import numpy as np
import argparse
import random
import cv2
import os
import matplotlib

class LeNet:
@staticmethod
def build(width, height, depth, classes):
# initialize the model
model = Sequential()
inputShape = (height, width, depth)
# if we are using "channels first", update the input shape
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
# first set of CONV => RELU => POOL layers
model.add(Conv2D(20, (5, 5), padding="same",
input_shape=inputShape))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
# second set of CONV => RELU => POOL layers
model.add(Conv2D(50, (5, 5), padding="same"))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
# first (and only) set of FC => RELU layers
model.add(Flatten())
model.add(Dense(500))
model.add(Activation("relu"))
# softmax classifier
model.add(Dense(classes))
model.add(Activation("softmax"))
# return the constructed network architecture
return model
dataset = '/home/pi/Desktop/tutorials/raspberry/trainImages/' # provide the path where your training images are present
# initialize the number of epochs to train for, initial learning rate,
# and batch size
EPOCHS = 15
INIT_LR = 1e-3
BS = 32

# initialize the data and labels
print("[INFO] loading images...")
data = []
labels = []

# grab the image paths and randomly shuffle them
imagePaths = sorted(list(paths.list_images(dataset)))
random.seed(42)
random.shuffle(imagePaths)
# loop over the input images
for imagePath in imagePaths:
# load the image, pre-process it, and store it in the data list
image = cv2.imread(imagePath)
image = cv2.resize(image, (28, 28))
image = img_to_array(image)
data.append(image)
# extract the class label from the image path and update the
# labels list
label = imagePath.split(os.path.sep)[-2]
print(label)
if label == 'forward':
label = 0
elif label == 'right':
label = 1
else:
label =2
labels.append(label)
# scale the raw pixel intensities to the range [0, 1]
data = np.array(data, dtype="float") / 255.0
labels = np.array(labels)

# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train_test_split(data,
labels, test_size=0.25, random_state=42)
# convert the labels from integers to vectors
trainY = to_categorical(trainY, num_classes=3)
testY = to_categorical(testY, num_classes=3)
# initialize the model
print("[INFO] compiling model...")
model = LeNet.build(width=28, height=28, depth=3, classes=3)
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
metrics=["accuracy"])

# train the network
print("[INFO] training network...")
H = model.fit(trainX, trainY, batch_size=BS,
validation_data=(testX, testY),# steps_per_epoch=len(trainX) // BS,
epochs=EPOCHS, verbose=1)

# save the model to disk
print("[INFO] serializing network...")
model.save("model")

Deploy the CNN

Once we have trained our model, we can deploy it on our Pi. You can use the code mentioned below for controlling the direction of the robot based on the prediction from CNN.

# import the necessary packages
from keras.preprocessing.image import img_to_array
from keras.models import load_model
import numpy as np
import cv2, time, sys, imutils, argparse
import RPi.GPIO as GPIO
from picamera.array import PiRGBArray
from picamera import PiCamera
import motor_control as mc
GPIO.setmode(GPIO.BCM)
#choose the GPIO pins for mptor control
fwd1 = 23 # pin 16
bwd1 = 24 #pin 18
fwd2 = 16 # pin 36
bwd2 = 20 # pin 38
# declare selected pin as output pin
GPIO.setup(fwd1, GPIO.OUT)
GPIO.setup(bwd1, GPIO.OUT)
GPIO.setup(fwd2, GPIO.OUT)
GPIO.setup(bwd2, GPIO.OUT)
model = load_model("model")
#function to control direction of robot based on prediction from CNN
def control_robot(image):
prediction = np.argmax(model.predict(image))
if prediction == 0:
print("forward")
mc.forward()
elif prediction == 2:
print("left")
mc.left()
else:
print("right")
mc.right()
if __name__ == "__main__":
try:
mc.stop()
# initialize the camera and grab a reference to the raw camera capture
camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(640, 480))
# allow the camera to warmup
time.sleep(0.1)
# capture frames from the camera
start = 1
for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
# grab the raw NumPy array representing the image, then initialize the timestamp
# and occupied/unoccupied text
image = frame.array
# show the frame
key = cv2.waitKey(1) & 0xFF
#cv2.imwrite(str(start) + ".jpg", image)
#start = start + 1
 image = cv2.resize(image, (28, 28))
image = img_to_array(image)
image = np.array(image, dtype="float") / 255.0
image = image.reshape(-1, 28, 28, 3)
#cv2.imshow("Frame", image[0])
 control_robot(image)
 # clear the stream in preparation for the next frame
rawCapture.truncate(0)
 except KeyboardInterrupt:
mc.stop()
GPIO.cleanup()
sys.exit()

If everything goes fine, you will be able to see a similar result when you run this code on your Raspberry Pi.

I hope you enjoyed the tutorial. Please feel free to comment and ask queries or make some suggestions. A big thanks to Adrian Rosebrock for his amazing tutorials on RaspberryPi which I have used as a base here. Do checkout his website.