Understanding Optical Flow & RAFT

Original article was published by Shuchen Du on Artificial Intelligence on Medium


What is optical flow ?

Optical flow field of two images

Optical flow is a vector field between two images, showing how the pixels of an object in the first image can be moved to form the same object in the second image. It is a kind of correspondence learning, because if the corresponding pixels of an object are known, the optical flow field can be calculated.

Optical flow equation & traditional methods

One point flow between two images H and I

Let ‘s take the most succinct form: one point flow between two images. Imagine a pixel at (x, y) in H flows to (x+u, y+v) in I, the optical flow vector is therefore (u, v).

How to solve for (u, v) ? Is there any constraints for us to build some equations ?

Firstly, as H(x, y) = I(x+u, y+v), let ‘s break I(x+u, y+v) using Taylor series:

Taylor series approximation of I(x+u, y+v)

Then, abandon higher order terms and combine with H(x, y) = I(x+u, y+v):

The derivation process of optical flow equation

Finally, in the limit as u and v go to zero, we got the optical flow equation as:

Optical flow equation

However, in real applications, u and v might be large or small, spanning several to tens of pixels, other than being zero-limit. Thus we can only get an approximation of the real optical flow. However, the flow field would be more accurate if u and v are closer to zero.

In the above equation, the unknowns are u and v, because other variables can be calculated from differences from x, y and time dimensions. Thus, there are two unknowns in one equation, which cannot be solved. Therefore, in the past 40 years, many researchers tried to provide other set of equations of u, v to make it solvable. Among them, the most famous method is Lucas-Kanade method.

In deep learning era, can we solve optical flow by deep neural networks ? If we can, what is the point of network designing ?

The answer is yes, and there are works on this area these years, the result is getting better and better. I will introduce a representing work called RAFT, which got the Best paper award of ECCV 2020.