 # Line Follower Robot using CNN

Source: Deep Learning on Medium

Hi all, in this tutorial we’ll be learning how to make a line follower robot. Although, there are a plenty of line follower tutorials out there as this concept in itself is quite old. However, here we’ll learn how to detect the line using Convolutional Neural Networks (CNN). Basically, we will be capturing a series of images at a predetermined interval using our Raspberry Pi camera, then we’ll used a pre-trained CNN to predict the direction in which our robot should move i.e. forward, right, or left. You will require following things for this tutorial:-

1. Raspberry Pi board,
2. Pi camera,
3. Jumper wires
4. Chassis, motors, tyres
5. Motor control IC (L293d)

I am assuming that you are already aware of how to control the motors using GPIO pins of your Raspberry Pi. In case you want to brush up your skills a little please go through my earlier tutorial. Also, make sure you have tensorflow 1.1 and open CV installed on your Pi before proceeding further.

We will divide this tutorial in to three sections:-

1. Capture the images for CNN.
2. Train the CNN
3. Deploy the CNN on the raspberry PI

### Capture the images for CNN

We require three sets of images for each of the three conditions i.e. forward, left, and right to train our CNN. Once we train our CNN, it will be able to predict in which direction the robot is moving and then we can take corrective measure accordingly. For e.g. if there is a left turn of line, the CNN will predict that robot is going in right direction relative to line and therefore, we should move it in left direction. Fig. depicting robot moving in various directions and the corresponding images captured by the camera

There may be various ways in which you can create a data-set for training CNN. In my case, I first built a track. Then, I used the code mentioned below to capture a series of images at an interval of .5 seconds. Then you place the robot at different positions on the track and capture a series of images. For e.g. in the first image, I have placed the robot such that I want the CNN to detect that robot should move in left direction. Similarly you repeat this process at different positions on the track. Once done you can repeat this process for forward and right direction. Once you capture the images for the three directions, place them in separate folders with names forward, left, and right respectively.

`# import the necessary packagesfrom picamera.array import PiRGBArrayfrom picamera import PiCameraimport timeimport cv2# initialize the camera and grab a reference to the raw camera capturecamera = PiCamera()camera.resolution = (640, 480) # set the resolutioncamera.framerate = 32 # set the frame raterawCapture = PiRGBArray(camera, size=(640, 480))# allow the camera to warm uptime.sleep(0.1)# capture frames from the camerastart = 1`
`for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True): # grab the raw NumPy array representing the image, then initialize the timestamp and occupied/unoccupied text image = frame.array # show the frame cv2.imshow("Frame", image) key = cv2.waitKey(1) & 0xFF cv2.imwrite(str(start) + ".jpg", image) start = start + 1 # clear the stream in preparation for the next frame rawCapture.truncate(0) # if the `q` key was pressed, break from the loop if key == ord("q"): break time.sleep(.5)`

### Train the CNN

You can train the CNN either on raspberry PI itself or on a different more powerful system and then save the trained model which can be read by the Pi. Following is the code for training your CNN on the images captured.

`# import the necessary packagesfrom keras.models import Sequentialfrom keras.layers.convolutional import Conv2D, MaxPooling2D from keras.layers.core import Activation, Flatten, Densefrom keras import backend as Kfrom keras.preprocessing.image import ImageDataGeneratorfrom keras.preprocessing.image import img_to_arrayfrom keras.optimizers import Adamfrom sklearn.model_selection import train_test_splitfrom keras.utils import to_categoricalfrom imutils import pathsimport numpy as npimport argparseimport randomimport cv2import osimport matplotlibclass LeNet:@staticmethod def build(width, height, depth, classes): # initialize the model model = Sequential() inputShape = (height, width, depth)`
`# if we are using "channels first", update the input shape if K.image_data_format() == "channels_first": inputShape = (depth, height, width)`
`# first set of CONV => RELU => POOL layers model.add(Conv2D(20, (5, 5), padding="same", input_shape=inputShape)) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))`
`# second set of CONV => RELU => POOL layers model.add(Conv2D(50, (5, 5), padding="same")) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))`
`# first (and only) set of FC => RELU layers model.add(Flatten()) model.add(Dense(500)) model.add(Activation("relu"))`
`# softmax classifier model.add(Dense(classes)) model.add(Activation("softmax"))`
`# return the constructed network architecture return model`
`dataset = '/home/pi/Desktop/tutorials/raspberry/trainImages/' # provide the path where your training images are present# initialize the number of epochs to train for, initial learning rate,# and batch sizeEPOCHS = 15INIT_LR = 1e-3BS = 32# initialize the data and labelsprint("[INFO] loading images...")data = []labels = []# grab the image paths and randomly shuffle themimagePaths = sorted(list(paths.list_images(dataset)))random.seed(42)random.shuffle(imagePaths)`
`# loop over the input imagesfor imagePath in imagePaths: # load the image, pre-process it, and store it in the data list image = cv2.imread(imagePath) image = cv2.resize(image, (28, 28)) image = img_to_array(image) data.append(image)`
`# extract the class label from the image path and update the # labels list label = imagePath.split(os.path.sep)[-2] print(label) if label == 'forward': label = 0 elif label == 'right': label = 1 else: label =2 labels.append(label)`
`# scale the raw pixel intensities to the range [0, 1]data = np.array(data, dtype="float") / 255.0labels = np.array(labels)# partition the data into training and testing splits using 75% of# the data for training and the remaining 25% for testing(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.25, random_state=42)`
`# convert the labels from integers to vectorstrainY = to_categorical(trainY, num_classes=3)testY = to_categorical(testY, num_classes=3)`
`# initialize the modelprint("[INFO] compiling model...")model = LeNet.build(width=28, height=28, depth=3, classes=3)opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)model.compile(loss="binary_crossentropy", optimizer=opt, metrics=["accuracy"])# train the networkprint("[INFO] training network...")H = model.fit(trainX, trainY, batch_size=BS, validation_data=(testX, testY),# steps_per_epoch=len(trainX) // BS, epochs=EPOCHS, verbose=1)# save the model to diskprint("[INFO] serializing network...")model.save("model")`

### Deploy the CNN

Once we have trained our model, we can deploy it on our Pi. You can use the code mentioned below for controlling the direction of the robot based on the prediction from CNN.

`# import the necessary packagesfrom keras.preprocessing.image import img_to_arrayfrom keras.models import load_modelimport numpy as npimport cv2, time, sys, imutils, argparseimport RPi.GPIO as GPIOfrom picamera.array import PiRGBArrayfrom picamera import PiCameraimport motor_control as mc`
`GPIO.setmode(GPIO.BCM)`
`#choose the GPIO pins for mptor controlfwd1 = 23 # pin 16bwd1 = 24 #pin 18fwd2 = 16 # pin 36bwd2 = 20 # pin 38`
`# declare selected pin as output pinGPIO.setup(fwd1, GPIO.OUT)GPIO.setup(bwd1, GPIO.OUT)GPIO.setup(fwd2, GPIO.OUT)GPIO.setup(bwd2, GPIO.OUT)`
`model = load_model("model")`
`#function to control direction of robot based on prediction from CNNdef control_robot(image): prediction = np.argmax(model.predict(image)) if prediction == 0: print("forward") mc.forward() elif prediction == 2: print("left") mc.left() else: print("right") mc.right()`
`if __name__ == "__main__": try: mc.stop() # initialize the camera and grab a reference to the raw camera capture camera = PiCamera() camera.resolution = (640, 480) camera.framerate = 32 rawCapture = PiRGBArray(camera, size=(640, 480)) # allow the camera to warmup time.sleep(0.1) # capture frames from the camera start = 1`
`for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True): # grab the raw NumPy array representing the image, then initialize the timestamp # and occupied/unoccupied text image = frame.array # show the frame key = cv2.waitKey(1) & 0xFF #cv2.imwrite(str(start) + ".jpg", image) #start = start + 1`
` image = cv2.resize(image, (28, 28)) image = img_to_array(image) image = np.array(image, dtype="float") / 255.0 image = image.reshape(-1, 28, 28, 3) #cv2.imshow("Frame", image)`
` control_robot(image)`
` # clear the stream in preparation for the next frame rawCapture.truncate(0)`
` except KeyboardInterrupt: mc.stop() GPIO.cleanup() sys.exit()`

If everything goes fine, you will be able to see a similar result when you run this code on your Raspberry Pi.

I hope you enjoyed the tutorial. Please feel free to comment and ask queries or make some suggestions. A big thanks to Adrian Rosebrock for his amazing tutorials on RaspberryPi which I have used as a base here. Do checkout his website.