Tutorial: Neural Network

I will cover the neural network module in this tutorial. My original purpose of introducing neural network module into Owl is two-fold:

• Test the expressiveness of Owl. Neural network is a useful and complex tool for building modern analytical applications so I chose it.

• To validate my research argument on how to structure modern (distributed) analytical libraries. Namely, the high-level analytical functionality (ML, DNN, optimisation, regression, and etc.) should be "glued" to the classic numerical functions via algorithmic differentiation, and the computation should be distributed via a specialised engine providing several well-defined distribution abstractions.

In the end, I only used less than 3.5k lines of code to implement a quite full-featured neural network module. Now let's go through what this module offers.

Module Structure

The Owl.Neural provides two submodules S and D for both single precision and double precision neural networks. In each submodule, it contains the following modules to allow you to work with the structure of the network and fine-tune the training.

• Graph : create and manipulate the neural network structure.
• Parallel : provide parallel computation capability, need to compose with Actor engine.
• Init : control the initialisation of the weights in the network.
• Activation : provide a set of frequently used activation functions.
• Params : maintains a set of training parameters.
• Batch : the batch parameter of training.
• Learning_Rate : the learning rate parameter of training.
• Loss : the loss function parameter of training.
• Gradient : the gradient method parameter of training.
• Momentum : the momentum parameter of training.
• Regularisation : the regularisation parameter of training.
• Clipping : the gradient clipping parameter of training.
• Checkpoint : the checkpoint parameter of training.

Types of Neuron

I have implemented a set of commonly used neurons in Owl.Neural.Neuron. Each neuron is a standalong module and adding a new type of neuron is much easier than adding a new one in Tensorflow or other framework thanks to Owl's Algodiff module.

Algodiff is the most powerful part of Owl and offers great benefits to the modules built atop of it. In neural network case, we only need to describe the logic of the forward pass without worrying about the backward propagation at all, because the Algodiff figures it out automatically for us thus reduces the potential errors. This explains why a full-featured neural network module only requires less than 3.5k lines of code.

In practice, you do not need to use the modules in Owl.Neural.Neuron directly. Instead, you should call the functions in Graph module to create a new neuron and add it to the network. Currently, Graph contains the following neurons.

• input
• activation
• linear
• linear_nobias
• embedding
• recurrent
• lstm
• gru
• conv1d
• conv2d
• conv3d
• max_pool1d
• max_pool2d
• avg_pool1d
• avg_pool2d
• global_max_pool1d
• global_max_pool2d
• global_avg_pool1d
• global_avg_pool2d
• fully_connected
• dropout
• gaussian_noise
• gaussian_dropout
• alpha_dropout
• normalisation
• reshape
• flatten
• lambda
• add
• mul
• dot
• max
• average
• concatenate

These neurons should be sufficient for creating from simple MLP to the most complicated Google's Inception network.

Train & Inference

Examples