Author: David Wolf
last changed: 14.03.2024
This is a simple neural network that recognizes handwritten digits of the mnist dataset. The mnist dataset provides test and training csv files of digits, wich each consist of 28x28 pixels. The pixels values range between 0 and 255 representing a different shade of white.
The neural network consist of three layers. An input layer, an hidden layer and an output layer. The input layer takes all 784 color values as an input. The output layer calculates a probability for each number 0 to 9.
The input Vector
Of course not only one digit gets fed into the network. That's why there are
The unactivated Matrix
The input as now been converted into a matrix of
To calculate the values of the output layer another weights matrix
The unactivated Matrix
Now
Backwards Propagation is used to evaluate the impact each weight and bias has on the result of the forward propagation.
To be able to evalute how good the predication was in comparison to the optimal solution we need to calculate a cost for each dataset. The cost matrix
The cost matrix
To evaluate the impact of
Now the same priciple as before can be applied to the second cost matrix to calclate the impcat of
For our neural network to be able to improve and "learn" we can let it try and minimize our cost matrix. An alogrithm which is able to find a local minimum is called gradient descend. It works by following the steepest descend at a given point inside our cost matrix. The size of the steps it takes is defined by a leraning rate
The application of this alogorith is quite simple, we just need to update our weights and biases according to their impact each iteration we run the model, muliplied by the learing rate
Results where highly depended on the applied learing rate.
A learing rate
A learing rate
To improve the starting point and therefore the result of the learning process I changed the random number generation.
Generating random float numbers based on the gauss normal distribution with an expectancy of 0 and a variance of 1 as well as subtracting a spicific value seems to have a lot of potential and improved the results drastically.
To garantee numerical stability and avoid overflow errors a modification to the softmax activation function had to be done. By subtracting the maximum value of each dataset from both exponents of the fraction the numbers get substantially smaller.
The accuracy fluctuates between 60% and 85% depending on iterations and the random initial starting point. Further improvements could be made regarding the learing rate or the data set. Also more nodes or layers could be added to improve accuracy.