Introducing PyRobot: How Facebook is Democratizing Robotics Programming Using Deep Learning

Source: Deep Learning on Medium


Robotics are one of the clear applications of artificial intelligence(AI) and deep learning research. In certain way, robots can be considered an ideal environment for AI as they don’t only combine the processing of sensory and motor information but also operate in incomplete environments in which information needs to process real time. However, robotics research and experimentation remains largely behind compared to AI frameworks. Recently, Facebook AI Research(FAIR) teamed up with robotics-powerhouse Carnegie Mellon University to build PyRobot, an open source framework to accelerate robotics research and experimentation.

Among the factors that make robotics research difficult, the dependencies on hardware and the lack of research environments are the two most influential ones. Until a few years ago, the cost of hardware for robots resulted prohibited for most research institutions. This situation has been improving in the last few years and now the cost of robots is comparable to the cost of GPU for a deep learning model. However, you still need to factor in the challenges of operating a robotic environment. Additionally, robotics research has been lacking open source environments that don’t require deep domain expertise and that can be used in a hardware agnostic manner.

Enter PyRobot

PyRobot is a Python-based environment for robotics research. Built on top of the popular robot operating system(ROS), PyRobot provides a lightweight, high-level interface that abstracts the fundamental mechanics of a robotic environment. Leveraging PyRobot, researchers can focus on authoring machine learning models without being concerned about the underlying robot mechanics. Conceptually, PyRobot was created following a series of basic principles:

· Beginner-Friendly: PyRobot allows users to start interacting with a robot using a few lines of code.

· Hardware-Agnostic: PyRobot is designed to easily accommodate common robotic manipulators and mobile bases. Currently, it supports LoCoBot, a low-cost mobile robot with a 5-DOF manipulator and a Sawyer robot.

· Open Source: PyRobot looks to break away from the traditional distribution models of robotic research environments which have been constrained to either academic research of industry -experts. To achieve that, PyRobot has been distributed under an open source license that allows any machine learning expert to start applying models to robotic research.

The PyRobot architecture was built on top of the ROS platform which allows researchers to have access to a rich robotics environment without requiring a deep level of expertise. Specifically, PyRobot provides a python wrapper around the mid-level features provided by ROS and the low-level C++/C controllers and driver backends. PyRobot has common utility functions for all robots, such as joint position control, joint velocity control, joint torque control, cartesian path planning, forward kinematics and inverse kinematics(based on the robot URDF file), path planning, visual SLAM, among other features. The following diagram illustrates the relationship between ROS and PyRobot.

Even though PyRobot abstracts the complexities of the underlying robot mechanics, developers still have the flexibility to use components at varying levels of the hierarchy, such as commanding low-level velocities and torques by-passing a planner. PyRobot will make it easier for the research community to use robotics data sets, algorithm implementations, and models, and also help them set benchmarks, compare their work, and build on each others’ results. When paired with low-cost robot platforms such as LoCoBot, PyRobot will reduce the entry barrier into robotics and help democratize robotics research.

To illustrate the simplicities of PyRobot, let’s use a scenario in which we are programming a robot to wave its hand. Just this basic operation has been the subject of PHD dissertations. With PyRobot, we can control the hard with a few lines of code. Notice that the code is not directly setting the target position using some specific coordinate but, instead, relies on a machine learning model abstracted behind the set_joint_positions operation.

Looking back at the PyRobot architecture, we can see several areas in which robot mechanics are abstracted via machine learning models. Here are some examples of the most relevant ones:

· Visual SLAM: The PyRobot framework includes Visual SLAM algorithms that provide more accurate odometry as compared to odometry that is derived purely from inertial sensors on the base. Specifically, PyRobot uses the ORB-SLAM2 model to process visual information from its cameras.

· Learned Visual Navigation: PyRobot includes learned policies for visual navigation in the in the Lo-CoBot environment. Specifically, PyRobot uses cognitive planning and visual policies models optimized for navigation environments.

· Grasping: PyRobot includes a learned-based grasping algorithm to grasp objects placed on the ground from RGB images. The model is trained on data from people’s homes and is robust to a wide variety of objects and backgrounds.

· Pushing: PyRobot includes a heuristic-based pushing algorithm which leverages depth and texture sensors to move an object effectively.

PyRobot is currently integrated with a number of real robots and simulator environments which facilitates the tasks for researchers. The current version of PyRobot supports the following integrations:

· LoCoBot: LoCoBot is a low-cost mobile manipulator platform built for easy setup and benchmarking robot learning research.

· Sawyer: The Sawyer is a 7-DOF collaborative robot arm from Rethink Robotics. PyRobot interfaces with the Intera SDK provided with the Sawyer.

· Gazebo: Gazebo is a a 3D rigid body simulator popular in the robotics community. For LoCoBot and LoCoBot-Lite, PyRobot supports tight integration with Gazebo i.e., the same code can be run on both Gazebo and the real robot.

PyRobot is a very solid attempt to streamline robotic research by enabling a bridge to apply deep learning models to robotic environment without requiring deep domain expertise. Combining the high level APIs of PyRobot with the availability of inexpensive robotic environments such as LoCoBot drastically lowers the barriers of entry for robotics research. Abstracting away the hardware characteristics will also help to benchmark how different models perform on different environments. As PyRobot evolves, it will support more deep learning frameworks and robotic environments which should streamline its adoption for robotic research.