How to understand Pytorch Source Code?

Original article can be found here (source): Deep Learning on Medium

How to understand Pytorch Source Code?

Based on [1], the fundamental unit in PyTorch is the Tensor. [1] provides an overview of how they implement Tensors in PyTorch, such that the user can interact with it from the Python shell. Four main questions are explained in [1]:

  • How does PyTorch extend the Python interpreter to define a Tensor type that can be manipulated from Python code?
  • How does PyTorch wrap the C libraries that actually define the Tensor’s properties and methods?
  • How does PyTorch cwrap work to generate code for Tensor methods?
  • How does PyTorch’s build system take all of these components to compile and generate a workable application?

However, the above doesn’t touch the autograd system though.
The autograd system is moved into C now and is multi-threaded, so stepping through the python debugger is probably a bit pointless. [3]

Here’s a pointer to very old source code, where all the logic was in Python, in the early days of PyTorch. The logic is largely the same still but moved into C.

https://github.com/pytorch/pytorch/blob/v0.1.1/torch/autograd/function.py

Code it up by yourself

The above two links still cannot give us a comprehensive python code to explore what Pytorch implements the autograd. No worries. Watch out these videos here. Joel Grus is live coding an Autograd Library.

Reference

[1] https://pytorch.org/blog/a-tour-of-pytorch-internals-1/

[2] https://pytorch.org/blog/a-tour-of-pytorch-internals-2/

[3] https://discuss.pytorch.org/t/how-to-understand-pytorchs-source-code/7600/2