Deep Learning Generalization

My project to recreate the results in the paper "Understanding Deep Learning Requires Rethinking Generalization" and do a similar analysis on LSTMs in NLP.

Contents

• Code

  • Mini Alexnet

    • cifar10_alexnet.py: Code to train mini Alexnet on CIFAR10 without regularization.
    • cifar10_alexnet_rand_crop.py: Code to train mini Alexnet on CIFAR10 without regularization and with random crop.
    • cifar10_alexnet_rand_labels.py: Code to train mini Alexnet on CIFAR10 without regularization and with random training labels.
    • cifar10_alexnet_wd.py: Code to train mini Alexnet on CIFAR10 with weight decay.
    • cifar10_alexnet_wd_rand_crop.py: Code to train mini Alexnet on CIFAR10 with weight decay and with random crop.

  • Mini Inception

    • cifar10_tf_inception.py: Code to train mini Inception on CIFAR10 without regularization.
    • cifar10_tf_inception_rand_crop.py: Code to train mini Inception on CIFAR10 without regularization and with random crop.
    • cifar10_tf_inception_wd.py: Code to train mini Inception on CIFAR10 with weight decay.
    • cifar10_tf_inception_wd_rand_crop.py: Code to train mini Inception on CIFAR10 with weight decay and with random crop.

  • MLP1 - MLP with 1 hidden layer

    • cifar10_keras_mlp.py: Code to train MLP with 1 hidden layer on CIFAR10 without regularization.
    • cifar10_keras_mlp_rand.py: Code to train MLP with 1 hidden layer on CIFAR10 without regularization and with random training labels.
    • cifar10_keras_mlp_wd.py: Code to train MLP with 1 hidden layer on CIFAR10 with weight decay.

  • MLP3 - MLP with 3 hidden layers

    • cifar10_keras_mlp3.py: Code to train MLP with 3 hidden layers on CIFAR10 without regularization.
    • cifar10_keras_mlp3_rand.py: Code to train MLP with 3 hidden layers on CIFAR10 without regularization and with random training labels.
    • cifar10_keras_mlp3_wd.py: Code to train MLP with 3 hidden layers on CIFAR10 with weight decay.

• Saved Models

  • Mini Alexnet

    • cifar10_alexnet.h5: A saved Keras Sequential model for mini Alexnet trained on CIFAR10.
    • cifar10_alexnet_rand_crop.h5: A saved Keras Sequential model for mini Alexnet trained on CIFAR10 with random crop.
    • cifar10_alexnet_rand_labels.h5: A saved Keras Sequential model for mini Alexnet trained on CIFAR10 with random labels.
    • cifar10_alexnet_wd.h5: A saved Keras Sequential model for mini Alexnet with L2 regularization, trained on CIFAR10.
    • cifar10_alexnet_wd_rand_crop.h5: A saved Keras Sequential model for mini Alexnet with L2 regularization, trained on CIFAR10 with random crop.

  • MLP1 - MLP with 1 hidden layer

    • cifar10_keras_mlp.h5: A saved Keras Sequential model for MLP with 1 hidden layer, trained on CIFAR10.
    • cifar10_keras_mlp_rand.h5: A saved Keras Sequential model for MLP with 1 hidden layer, trained on CIFAR10 with random labels.
    • cifar10_keras_mlp_wd.h5: A saved Keras Sequential model for MLP with 1 hidden layer and L2 regularization, trained on CIFAR10.

  • MLP3 - MLP with 3 hidden layers

    • cifar10_keras_mlp3.h5: A saved Keras Sequential model for MLP with 3 hidden layers, trained on CIFAR10.
    • cifar10_keras_mlp3_rand.h5: A saved Keras Sequential model for MLP with 3 hidden layers, trained on CIFAR10 with random labels.
    • cifar10_keras_mlp3_wd.h5: A saved Keras Sequential model for MLP with 3 hidden layers and L2 regularization, trained on CIFAR10.

• Results

  • Mini Alexnet

    • Alexnet_Cifar10.png: Mini Alexnet with no regularization.
    • Alexnet_Cifar10_rand_crop.png: Mini Alexnet with no regularization and with random crop.
    • Alexnet_Cifar10_rand_labels.png: Mini Alexnet with no regularization and with random labels.
    • Alexnet_Cifar10_wd.png: Mini Alexnet with weight decay.
    • Alexnet_Cifar10_wd_rand_crop.png: Mini Alexnet with weight decay and with random crop.

  • Mini Inception

    • Inception_Cifar10.png: Mini Inception with no regularization.
    • Inception_Cifar10_rand_crop.png: Mini Inception with no regularization and with random crop.
    • Inception_Cifar10_wd.png: Mini Inception with weight decay.
    • Inception_Cifar10_wd_rand_crop.png: Mini Inception with weight decay and with random crop.

  • MLP1 - MLP with 1 hidden layer

    • MLP1_Cifar10.png: MLP with 1 hidden layer with no regularization.
    • MLP1_Cifar10_rand_labels.png: MLP with 1 hidden layer with no regularization and with random labels.
    • MLP1_Cifar10_wd.png: MLP with 1 hidden layer with weight decay.

  • MLP3 - MLP with 3 hidden layers

    • MLP3_Cifar10.png: MLP with 3 hidden layers with no regularization.
    • MLP3_Cifar10_rand_labels.png: MLP with 3 hidden layers with no regularization and with random labels.
    • MLP3_Cifar10_wd.png: MLP with 3 hidden layers with weight decay.