PyTorch3D : RoboLove

Source: Deep Learning on Medium

Using Facebook’s 3D Deep Learning Library to shape a robot’s heart!

FAIR (Facebook AI Research) recently introduced a Python library which works with PyTorch to facilitate the manipulation and processing of 3D data, such as 3D models. Read more about it on their release blog here or watch the video below:

Video published by Irfan Ahmad giving a preview of PyTorch3D

Given that it’s Valentines day and my experience working with 3D data, apart from some half hearted attempts at using Blender some years ago, is close to none, I thought it’d be interesting to take one of the tutorials and ‘Valentinise’ it. So I downloaded the following models from the Free3D website.

The projects guinea pigs. Robot and Heart

The objective of the project is to use PyTorch3D’s sphere to dolphin tutorial as a base and adapt it to convert a robot into a heart.

The first step is to load these up into the Jupyter Notebook. I used Google Colab for the free access to GPUs and easiness of configuration. Also, I could directly clone the tutorial. Loading and structuring the files is really straightforward, however, the robot loaded in a lying down position by default.

Lazy robot

To fix this, after some fiddling around, I managed to use the library’s RotateAxisAngle class to get him into an upright position. I used the following code to rotate the object, which, in hindsight, is really obvious:

Having learnt to use it, I thought I’d show off! Below we’ve got our source mesh, which is the robot object.

360 degree turn

The images used to build this GIF were generated using this function:

On the other hand, we have the heart object (our target mesh), which looks like this:

A rotating heart wasn’t very interesting

Using the tutorial’s ‘optimisation loop’ code plus a very basic adjustment of the loss and learning rate hyper-parameters, I obtained the following mutation:

And of course, the final mesh rotating!

360 degree turn again!

I managed to turn the robot into something that resembles a heart with antennas, so I can say this introduction to PyTorch3D was probably successful. However, it was interesting to work with FAIR’s library and I’m looking forward to using it for higher impact projects!