Recipes for models

Recommended model to start with depends on type of data

Image-like

Start with:

  • ResNet (CNN)

Then consider:

  • Transformer

Sequence-like

Start with:

  • Transformer (or RNN-like)

Then consider:

  • Dilated attention models

Tabular

Start with:

  • LightGBM

Then consider:

  • Make feature embeddings + 2 layer FC network

Combination of above

  • Build each unit separately, then ensemble with 2 layer FC network

Other hyperparameters

  • Adam Optimizer with variable LR (see fastai lrfind)
  • ReLU and Kaiming init for most models
  • Disable regularization (including dropout)