Deep Learned Lessons

  1. Always start with a very simple model and see how the results are so that we can decide if at all it is productive in terms of business to start building more complicated models. Very importantly such simple models work as efficient baselines to see how well more complicated models perform. Without a baseline one cannot appreciate the effectiveness of a complicated model.
  2. If at all you and very importantly your boss both are convinced that we need a complicated model, then do not immediately start with a Deep Neural network of 3–4 layers.
  3. Instead start with a very simple Neural network of one layer and slowly build upon it to see the effect of deepening the network.
  4. Monitor the validation accuracy to appreciate how well the model performs. Plot graphs with all the hyper-parameters such as depth of the network, number of hidden neurons, number of epochs, learning rate,training loss etc, as X-axis and Validation accuracy as Y-axis.
  5. Once you have decided the depth of the network start using some Hyper-parameter search tool to improve the validation accuracy.

Source: Deep Learning on Medium