Filming “Deep Reinforcement Learning and Generative Adversarial Network LiveLessons”


Following on the back of my Deep Learning with TensorFlow and Deep Learning for Natural Language Processing tutorials, I recently recorded a new set of videos covering Deep Reinforcement Learning and Generative Adversarial Networks. These two subfields of deep learning are arguably the most rapidly-evolving so it was a thrill both to learn about these topics and to put the videos together.

Generative Adversarial Networks (GANs) pit two deep learning networks against each other as adversaries. One of these networks is tasked with generating novel images while the adversary critiques these images, attempting to discriminate the real (e.g., a photograph, drawing, painting) from synthetic, machine-generated analogues. GANs produce stunning photorealistic images with flexible, user-specifiable features, and offer many of the most surprising advances in Deep Learning in recent years.

Stunning examples of the photorealistic images that can be produced by Generative Adversarial Networks. (Image credit: Zhu et al., 2017, arXiv: 1703.10593)

Deep Reinforcement Learning has produced equally exciting results, including the bulk of the most widely-publicised recent “Artificial Intelligence” breakthroughs. Achievements on this list include AlphaGo defeating world-leading board-game players, a single algorithm passing human-level performance on a broad range of Atari games, and robots becoming capable of subtle manipulation tasks previously reserved for human hands alone. The underlying theme across Deep RL is the training of an “agent” to become adept in given “environments” by rewarding them when they exhibit particular, desired behaviours.

Once edited, the videos will be about six hours long. They should be available within Safari Books Online by March. Here’s a breakdown of the tutorial’s five lessons:

Lesson 1: The Foundations of Artificial Intelligence

This lesson starts off by examining what the term “AI” means and how it relates to Deep Learning. It continues by discussing cutting-edge applications of Generative Adversarial Networks and Deep Reinforcement Learning algorithms that have recently revolutionized the field of machine learning. We then quickly review how to run the code in these LiveLessons on your own machine as well as the foundational Deep Learning theory that is essential for building these advanced-topics specializations upon.

Lesson 2: Generative Adversarial Networks (GANs)

Lesson Two begins by covering the high-level theory of what GANs are and how they are able to generate realistic-looking images. Next up is the “Quick, Draw!” game, which is used as the source of hundreds of thousands of hand-drawn images from a single class — say apples, rhinoceroses, or rainbows — for a GAN to learn to imitate. The bulk of the lesson is spent developing the intricate code for the three primary components of a GAN: the discriminator network, the generator network, and the adversarial network that pits them against each other. The completed Jupyter notebook is available here.

Lesson 3: Deep Q-Learning Networks (DQNs)

Lesson Three is the first of three lessons that explores Deep Reinforcement Learning algorithms. It commences by introducing a simple game called the Cartpole Game that is used throughout the rest of the lessons to train our Deep Reinforcement Learning algorithms. Next, the lesson delves a bit into the essential theory of Deep Reinforcement Learning as well as Deep Q-Learning Networks (DQNs), a popular type of Deep Reinforcement Learning agent. With that theory under your belt, you’ll be able to understand at an intuitive level the code that you subsequently develop when you define your own DQN Agent and have it interact with The Cartpole Game within a handy library called OpenAI Gym. The completed Jupyter notebook is available here.

The Deep Reinforcement Learning lessons begin where the foundational content (“Deep Learning with TensorFlow LiveLessons”) left off on the left-hand side of the board. On these foundations, we build the Deep RL theory shown on the right-hand side.

Lesson 4: OpenAI Lab

In the previous lesson, a Deep Q-Learning Network was used to master the Cartpole Game. This lesson builds upon those Deep Reinforcement Learning foundations by using Wah Loon Keng and Laura Graesser’s OpenAI Lab both to visualize our DQN agent’s performance in real-time and to straightforwardly modify its hyperparameters. You learn how to gauge your agent’s overall fitness and automate the search through hyperparameters to optimize your agent’s performance.

Lesson 5: Advanced Deep Reinforcement Learning Agents

The previous two lessons covered Deep Reinforcement Learning largely through the lens of the Deep Q-Learning Network. This lesson introduces the Coach library to easily expand your arsenal of Deep Reinforcement Learning agents. We focus on the REINFORCE Policy Gradient algorithm and the Actor-Critic algorithm in particular. The lesson closes out these LiveLessons by covering why Deep Learning is reshaping software in general as well as by returning to the discussion of AI, specifically addressing the limitations of contemporary Deep Learning approaches in attaining Artificial General Intelligence.


Jon Krohn is Chief Data Scientist at the machine-learning startup untapt. He presents an acclaimed series of tutorials on artificial neural networks, including Deep Learning with TensorFlow LiveLessons. He also teaches his deep learning curriculum in-classroom at the NYC Data Science Academy. Jon holds a doctorate in neuroscience from Oxford University and has been publishing on machine learning in leading academic journals since 2010. His forthcoming book, Deep Learning Illustrated, is being published on Pearson’s Addison-Wesley imprint and will be distributed in 2018.

Source: Deep Learning on Medium