A deep learning library for training end-to-end Artificial Neural Networks (ANNs), primarily based on numpy
and autograd
. (This is a small 2-day project for learning purposes.)
Dense_ANN
,Dense_Layer
zero_init
,uniform_init
,xavier_init
,he_init
,variance_scaling_init
,constant_init
sigmoid
,tanh
,relu
,leaky_relu
,elu
,swish
,fixed_point
mse
(Mean Squared Error),mae
(Mean Absolute Error),binary_cross_entropy
,hinge_loss
SGD_Optimizer
(Stochastic Gradient Descent) capable of training fully connected ANNs.
L1 (lasso)
,L2 (ridge)
,Elastic Net
stop_loss_min
,lr_scheduler
,checkpoints
loss_plot
,mean_validation_plot
- Uses mostly
numpy
withautograd
, which does not take advantage of GPU and parallelism as modern libraries would. - Only implements end-to-end training, not model-based deep learning.
git clone https://github.com/AFLProjects/deepl.git
cd deepl
python3 setup.py
python3 example.py
from deepl import core
from deepl import training
from deepl import visualization
import autograd.numpy as np
# Example
# Define structure
structure = (3, 3, 3)
nn = core.Dense_ANN(structure, [core.relu, core.fixed_point])
# Training data
data_size = 16000
train_x = [np.random.rand(3) for _ in range(data_size)]
train_y = 100 * train_x
# Validation data
validate_size = 64
validate_x = [np.random.rand(3) for _ in range(validate_size)]
validate_y = 100 * validate_x
# Optimizer and parameters
trainer = training.SGD_Optimizer(nn,
loss=training.mse,
init=core.uniform_init,
init_args=(0, 1),
start_lr=0.1,
callbacks=[training.stop_loss_min,
training.checkpoints],
callback_args=[(10e-3,),
(250,)],
validate_x=validate_x,
validate_y=validate_y,
reg=training.lasso,
reg_params=(0.001,))
# Train
weights_tensor, loss_values = trainer.train(train_x, train_y, data_size)
# Plots
visualization.loss_plot(trainer, 0.5, 'MSE')
visualization.mean_validation_plot(trainer, 0.5, 'MSE')
Performance for this specific case( (3,3,3) ) : init took 8.833e-06 seconds uniform_init took 2.813e-05 seconds init took 6.354e-05 seconds train took 9.781e-01 seconds