1. Overview

Deep learning has been proven as a powerful tool to recognize patterns in images. In this project, we aim to build a convolutional neural network (CNN) to map image pixel values to steering angles for an autonomous car in a simulator. We collect data by driving the car manually in the simulator and recording both the images and angles. We train the CNN on an AWS EC2 instance with GPU enabled. To test the effectiveness of our CNN model, we test it in the simulator in an autonomous mode meaning the model takes image as an input and outputs the steering angles to control the car.

Here are two clips of the car under the autonomous mode.

2. Network Architecture

General strategy to find a solution architecture

Generally we start from the LeNet-5 since it has been shown as an effective neural network for image recognition task. We then use bias-variance trade-off to firstly identify a satisfactory low-bias model and secondly leverage regularization techinique to further identify a satisfactory low bias and low variance model.

Architecture for this project

Fortunately for this specific task, we don’t need to start from LeNet-5 since Nvidia has published a very effective neural network architecture for self-driving cars in its paper End to End Learning for Self-Driving Cars. We start from this neural network and it turns out to be an effective architecture for our project.

3. Model Building

We split the entire dataset into training set (80%) and validation set (20%). We use the Adam optimizer to adaptively set the step length in the optimization algorithm. Additionly we divide the training set into batches, each of which consists of 32 samples. We train the model for 13 epoches as the validation loss starts to increase after 13 epoches.

From the figure above, we can see that the model suffers from high variance. To remedy this problem, we seek to add l2 penalization to all weights in the neural network. We observe that regularization decreases the variance, the regularized model is worse than the un-regularized one under autonomous mode in the simulator. So we keep the un-regularized network as our final model.

4. Testing

We deploy the trained neural network in the simulator to drive the car autonomously. The result demonstrate that the trained model can effectively drive the car without leaving away the road.

You can find related technical details at

model.py — Python code for building the CNN model

model.h5 — the trained CNN model

video.mp4 — recording of one lap of autonomous drive

written summary — Technical summary

Source: Deep Learning on Medium