Source: Deep Learning on Medium
In addition to, or in combination with cross-validation, I recommend model blending. The hidden decision trees methodology that I developed a while back, do just that:
- decision trees on observations that are better handled by decision trees
- regression on observations that are better handled by regression
Each model produces a score, and both scores (regression based or decision tree based) are re-scaled so that the two scores are interchangeable.
See details here: