neural network

This is a maven library for neural networks in java 8.
Different visualization options are provided with javafx.

Table of Contents

1. About
2. Scope
   2.1 Architecture
   2.2 Supported algorithms
   2.3 Initializers
   2.4 Rectifiers
   2.5 Cost functions
   2.6 Optimizers
   2.7 Parametrization
   2.8 Persistence
   2.9 UI
3. Samples
   3.1 Overview
   3.2 Walkthrough
4. Implementation
5. Releases
6. References

About

The motivation for this library originated in the study project snakeML, where the game snake was to be solved by neural networks. The rudimentary approach was extracted to this library and improved over time to study both neural networks and play around with java.
The goal of this library is to make neural networks easily accessible in terms of 'how does it work' and to provide an easy-to-use and plug-and-play-tool for other projects.
While the first steps focused on functionality, later work focused on different approaches of visualization with javafx.

Scope

Architecture

• fully connected
• none to many hidden layers supported
• node count per layer is configurable

Supported algorithms

• Supervised learning
• Genetic algorithm

Initializers

• Static
• Random
• Xavier
• Kaiming

Rectifiers

Implemented are following rectifiers:

• Identity
• RELU
• Leaky RELU
• Sigmoid
• Sigmoid (accurate)
• SILU
• SILU (accurate)
• TANH
• ELU
• GELU
• Softplus
• Softmax

Cost functions

• MSE native
• MSE
• Cross entropy
• Exponential
• Hellinger distance
• KLD
• GKLD
• ISD

Optimizers

• none (static)
• stochastic gradient descent (applicable to learning rate and mutation rate)
• dropout

Parametrization

The parametrization of the hyperparameters of the neural network can be done as following:

• programmatically
• by neuralnetwork.properties in case default values are used constantly

Persistence

• neural network instances are fully serializable

UI

With the additional ui package, you may be able to visualize the neural network and additional metrics interactively based on the javafx framework.

Samples

Overview

In order to have an idea about the look and feel, see following samples which were created with this library.

Sample code for a minimal prediction task:

double[] in = {1,0.5};                                              // input values
NeuralNetwork neuralNetwork = new NeuralNetwork.Builder(2, 4, 1)
    .setDefaultRectifier(Rectifier.SIGMOID)
    .setLearningRate(0.8)
    .build();                                                       // initialization
List<Double> out = neuralNetwork.predict(in);                       // prediction

Live visualization of a predicting neural network:

Line charts of available rectifiers:

Visualization of weights of a layer per node and overall, trained on mnist:

Confusion matrix visualization:

Binary decision boundaries in 2D and 3D, manually refreshed while training on xor dataset:

Multiclass decision boundaries in 3D, animated:

Fully integrated sample ui for rudimentary analysis of tsp problem:

Walkthrough

Detailed examples are available here:

Topic	Description
basic usage	constructor, methods, basic features
supervised learning	implementation example
genetic algorithm	implementation example
visualization	charts, decision boundaries, confusion matrix, layer weights etc. in 2D and 3D

Implementation

This library can be either implemented by jar file or as maven dependency.
Detailed instructions are documented here.

Releases

As this project started 'fun project' and the concept of 'free time' is more a fairy tale than reality, there is not a proper version control (yet).
In general, the neural netowrk algorithm is quite stable, no big changes are to be expected soon.
Before new features are introduced, a stable, consistent realease will be made.

Release	Description
upcoming	stable, consistent release, focusing on consistency
3.1	mostly minor fixes and features added, currently treated as snapshot
3.0	introduction of charts and other visualizations, lot of refactoring
<= 2.5	multiple releases focusing on the neural network algorithm

References

Code:

• Sonawane, Suyash: JAvaNet, GitHub Repository, 2021. (first steps)
• https://stackoverflow.com/ (coding hints)

Literature:

• Bialas, Piotr: Implementation of artifical intelligence in Snake game using genetic algorithm and neural networks, CEUR 2468, 2019.
• Chollet, Francois: Deepl Learning mit Python und Kears. Das Praxis-Handbuch. mitp Verlags GmbH & Co. KG, Frechen, 2018. (978-3-95845-838-3)
• Geron, Aurelien: Hands.On Machine Learning with Scikit-Learn, Keras & TensorFlow. Concepts, Tools, and Techniques to Build Intelligent Systems. Second Edition, O'Reilly, Canada 2019. (978-1-492-03264-9)
• Hansen, Casper: Neural Networks: Feedforward and Backpropagation Explained & Optimization. Deep Learning, mlfromscratch, 5.08.2019.
• Hansen, Casper: Activation Functions Explained - GELU, SELU, ELU, ReLU and more. Deep Learning, mlfromscratch, 22.08.2019.
• Karpathy, Andrej: CS231n Winter 2016, Stanford Universy, Youtube, 2016.
• Lapan, Maxim: Deep Reinforcement Learning Hands-On. Second Edition, Packt Publishing, Birmingham, 2020. (978-1-83882-699-4)
• Steinwendner, Joachim et. al: Neuronale Netzw programmieren mit Python. 2. Auflage, Rheinwerk Verlag, 2020. (978-3-8362-7450-0)