Matrix Phylogeny for Machine Learning

Source: Deep Learning on Medium

README

The diagram is to be read top to bottom. I built it for Real, mainly square matrices.

One of the first and key separations I make is I break down matrices into two types based on a)their eigenvalues/vectors (“eig”) and b)their determinant behavior (“det”). That gives us two separate branches to explore (left and right, respectively).

Black (vertical) arrows between boxes indicate hereditary relationships. Any box is necessarily a subtype of the box right above it, and necessarily inherits any properties of its parent(s). For example, every Normal matrix is necessarily Diagonalizable and every Symmetric matrix is necessarily Normal. This also means that every Symmetric matrix has both n linearly independent (L.I.) eigenvectors (inherited from Diagonalizable) and commutes with its conjugate transpose (inherited from Normal).

Colored (non-vertical) arrows fill in the relationship gaps for matrix-types that don’t otherwise sit under another. For example, because Orthogonal matrices sit in a different branch to say Normal and Symmetric, following just the black vertical arrows alone is insufficient to understand how they relate to each other. Are all Normal matrices Orthogonal? Are all Orthogonal matrices Normal?

I didn’t bother to fill in every possible relationship, just the ones that felt important. By the way, if you’re more of a table person, this was the table I filled out prior to drawing the arrows:

(matrices on top defined through matrices on the left)

Next, colors. Blue text inside the boxes is the best / most succinct definition I could find for each matrix type. Sometimes I also included useful to know properties in blue, eg “λ are Real” (eigenvalues are Real) or “∃ SVD” (there exists a diagonal form that can be reached through SVD).

Box colors:

  • Gray boxes = terminal and not very interesting ones.
  • Blue boxes = connected through spectral theorem (see at the bottom of the diagram).
  • Thin orange boxes = Complex equivalents (the diagram is about Real matrices, but I couldn’t help to include a few popular Complex versions).
  • White boxes = normal, don’t belong to the above.

Arrow colors:

  • Red = two-way “not necessarily” relationship
  • Orange = one-way “not necessarily” relationship
  • Green = special case of rotational matrices being both defective and not (over Reals and over Complex numbers, respectively)

And a few final things:

  • Numbers in bottom right corners represent example matrices for each type.
  • I’ve included the spectral theorem at the bottom and listed how it changes depending on whether A = just Diagonalizable, Normal, or Symmetric. It should be pretty self-explanatory.
  • I’ve also included the 5 rules for Positive-Definite matrices (stolen from Gilbert Strang’s awesome YT lecture here), because why not.

Living document

The diagram is as complete as I could make it, but I’m sure I’ve missed things (and maybe even gotten some stuff wrong 🤭🤭🤭).

Treat this as a living document — if you comment below with any ideas/fixes, I will add the changes and update the post.

Happy learning!