Define-and-Run vs. Define-by-Run

Source: Deep Learning on Medium

Define-and-Run vs. Define-by-Run

Define-and-Run and Define-by-Run are frequently mentioned when introducing deep learning frameworks. These are the strategy to define and run neural network or computational graph. To understand and remember these concepts, I write down this post. Since this post is kind of summary, I hope you to read references to learn concepts deeply.

Define-and-Run

Define-and-Run:

  • Network(or computational graph) is defined and fixed.
  • And then mini-batch data are fed into the predefined network.
  • TensorFlow, Caffe, torch.nn, and Theano-based frameworks
Define-by-Run

Define-by-Run

  • Network is defined dynamically via the actual forward computation.
  • This dynamic definition allows conditionals and loops into the network definitions easily.
  • Chainer, PyTorch

References

[1] https://docs.chainer.org/en/stable/guides/define_by_run.html

[2] https://ko.wikipedia.org/wiki/PyTorch