Simple MLP - NeuralNetwork Library For Microcontrollers

Nothing "Import ant", just a simple library for implementing Neural-Networks(NNs) easily and effectively on any Arduino board and other microcontrollers.

📚 Summary

NN Functions	Input Type (x)	Output Type (Y)	Action
BackProp(x)	DFLOAT Array	-	Trains the Neural-network "Tells" to the NN if the output was correct/the-expected/X-inputs and then, "teaches" it.
*FeedForward(x)	DFLOAT Array	DFLOAT Array	Returns the output of it "Feeds" the NN with X-input values and returns Y-Output Values, If needed.
getMeanSqrdError(x)	Unsigned Int	DFLOAT	Returns the Mean Squared Error MSE, is SSE (Sum Squared Error) divided by the Product of number-οf-οutputs and inputs-per-epoch aka batch-size.

Understanding the Basics of a Neural Network:
EXM 0 1 2 3 4 5 6 7 8 9 10 11

📦 Features

• Current

• • + Use of activation-functions per layer-to-layer.
• • + Optimizations based on user's preference.
• • + Support for custom activation functions.
• • + Basic ESP32-S3 SIMD acceleration.
• • + Both 16 and 8 bit, int quantization.
• • + MSE/BCE/CCE loss-functions.
• • + Support for double precision.
• • + Many activation-functions.
• • + Use of storage medias.
• • + Exceptional solutions.
• • + Simplicity!

• In the future, i want

• • - Your support.
• • - Better overall code.
• • - Other training methods.
• • - More Activation Functions.
• • - Training with int quantization.
• • - Support for convolutional layers.
• • - Support for fixed-point arithmetics.
• • - Different weight initialization methods.
• • - Support for external EEPROM and FRAM.
• • - Even more properties, for many different needs.

✏️ Examples

✨ (See also): training with Tensorflow section)

• 🔤 Basic:
• • Using a NN inside the loop
• • Pre-trained NN behaving as a 3-input-xor circuit
• • Training a NN to behave as a 3-input-xor circuit
• • Training a NN to behave as a single xor-gate
• • Using multiple Activ.-functions per layer-to-layer
• • Using multiple biases per layer-to-layer
• 💾 Media:
• • Save NN into SD and load it into RAM after restart
• • Saving a NN into the internal EEPROM for later use
• • Running NN mostly via internal EEPROM (not RAM)
• • Running NN mostly via programmable memmory
• 🎲 Other:
• • Pre-trained int8_t-quantized NN ✨
• • Using a custom function made by you
• • Support for 8Byte "double" instead of "float"
• • Recognizing handwritten digits (MNIST) ✨

⚠️ Important

1. I am NOT a professional in any of those fields...
2. In case of error with 'POINTER_REGS' click here
3. bias means biases if MULTIPLE_BIASES_PER_LAYER is enabled
4. Ensure you use (32-bit) floats during training unless you USE_64_BIT_DOUBLE.
5. And most important, destructor wont free last-layer's outputs!

   By design, the destructor won't free\deallocate the last layer's outputs, allowing you to continue using these outputs through the pointer in your sketch. To fully delete the neural-network and free the associated resources, it's your responsibility to: either delete[] outputs or delete[] NN.layers[NN.numberOflayers - 1].outputs; at the end of the scope. Additionally, with NN.load(file): ensure you deleted last-layer's *outputs in your sketch, in case you plan to re-use the same pointer for capturing the outputs of the newly-loaded-NN's feedforward.

🔬 Tested on

Arduino UNO

• Don't USE_64_BIT_DOUBLE: double is the same as float on the UNO

ESP32-C3

• You may need to increase serial baud-rate from 9600 to 115200
• Uses software-emulated EEPROM, so don't expect EEPROM-examples\functionalities to work on it

ATtiny85

• NN.print() Function is disabled!
• Doesn't have FPU that makes Maths on it, "difficult" for the SRAM (i think..?)
• If you want to use "Serial" on an ATtiny85 Click Here (Be Careful SoftwareSerial Uses A lot of SRAM)
• Backprop maths on an ATtiny85 won't work properly (due to SRAM limitations, unless NN too small), though Feed Forword maths will Work! [...] ^{_{(since the first release I haven't tested it again on the ATtiny85 at least yet, so I am not 100% sure)}}

⚙️ Functions, Variables ...

Note that DFLOAT means float, unless you USE_64_BIT_DOUBLE, then it means double. IDFLOAT equals DFLOAT unless you USE_INT_QUANTIZATION, then it either means int16_t or int8_t. IS_CONST means nothing, unless you USE_PROGMEM, then it means const.

(NN) Neural-Network's Constructors
NeuralNetwork() Default Constructors
NeuralNetwork(String file) Available if #include <SD.h>, lets you load NN from SD. Usefull\Important note: moving it bellow #include <NeuralNetwork.h> will disable the support.
NeuralNetwork(unsigned int address) Available if defined _1_OPTIMIZE 0B10000000-(USE_INTERNAL_EEPROM)
NeuralNetwork(*layer_, &NumberOflayers, *_ActFunctionPerLayer) Available if backpropagation is available (! NO_BACKPROP)
NeuralNetwork(*layer_, &NumberOflayers, &LRw, &LRb, *_ActFunctionPerLayer) Available if backpropagation is available (! NO_BACKPROP)
NeuralNetwork(*layer_, *default_Weights, &NumberOflayers, *_ActFunctionPerLayer) Available if NO_BIAS enabled
NeuralNetwork(*layer_, *default_Weights, *default_Bias, &NumberOflayers, *_ActFunctionPerLayer) (:

 IS_CONST IDFLOAT *default_Bias
 IS_CONST IDFLOAT *default_Weights
 byte *_ActFunctionPerLayer = NULL
 const unsigned int *layer_
 const unsigned int &NumberOflayers
 const DFLOAT &LRw
 const DFLOAT &LRb

Type Main Functions

NN Functions	Input Type (x)	Output Type (Y)	Action
FeedForward_Individual(x)	DFLOAT	DFLOAT Array	RAM Optimized FeedForward "Feeds" the NN with each one X-input Individually until it returns Y-Output Values, If needed. Important note: You can't train with it. (Almost no RAM usage for input layer, see also: example)
*FeedForward(x)	DFLOAT Array	DFLOAT Array	Returns the output of the NN "Feeds" the NN with X-input values and returns Y-Output Values, If needed.
BackProp(x)	DFLOAT Array	-	Trains the NN "Tells" to the NN if the output was correct/the-expected/X-inputs and then, "teaches" it.
load(x)	String	bool	Loads NN from SD Available if #include <SD.h>. Usefull\Important note: moving it bellow #include <NeuralNetwork.h> will disable the support.
save(x)	String \ int	bool \ int	Saves NN to storage media SD or internal-EEPROM
print()	-	String	Prints the specs of the NN (If _1_OPTIMIZE 0B10000000 prints from PROGMEM)

DFLOAT Loss Functions

No need for #define MEAN_SQUARED_ERROR, MSE is the default loss and it is always enabled. The only case in which you will also need to define the MSE in your sketch, is only if you want to use it in relation with another loss-function. To use any other loss-function except from MSE just define it as seen below.

Loss Functions	Enabling MACRO
NN.getMeanSqrdError(unsigned int batch_size)	#define MEAN_SQUARED_ERROR
NN.getBinaryCrossEntropy(unsigned int batch_size)	#define BINARY_CROSS_ENTROPY
NN.getCategoricalCrossEntropy(unsigned int batch_size)	#define CATEGORICAL_CROSS_ENTROPY

DFLOAT Loss Variables

To use any of the variables below, you first need to #define a loss function as said above too.

Loss variables	Sum variables
NN.MeanSqrdError	NN.sumSquaredError
NN.BinaryCrossEntropy	NN.sumOfBinaryCrossEntropy
NN.CategoricalCrossEntropy	NN.sumOfCategoricalCrossEntropy

DFLOAT Activation Functions

Due to (my uncertainty and) the strict RAM optimization that allows the library to use one array that stores only the values after the activation instead of two arrays storing values before and after the activation, the use of some derivative functions in backpropagation are not supported by this library at this moment, as also seen by the MACRO NO_BACKPROP below. This means that if you want to use and #define any function from 8-13 under the section "NO_BACKPROP support" , you won't be able to use backpropagation.

	Enabling MACRO	Activation Functions	Returns
0	#define Sigmoid	NN.layers->Sigmoid(&x)	1/(1+e^(-x))
1	#define Tanh	NN.layers->Tanh(&x)	(e^(2*x)-1)/(e^(2*x)+1)
2	#define ReLU	NN.layers->ReLU(&x)	(x>0)?x:0
3	#define LeakyELU	NN.layers->LeakyELU(&x)	(x>0)?x:AlphaLeaky*x
4	#define ELU	NN.layers->ELU(&x)	(x>0)?x:AlphaELU*(e^(x)-1)
5	#define SELU	NN.layers->SELU(&x)	(x>0)?x:AlphaSELU*(e^(x)-1)
6	#define Softmax	NN.layers->Softmax(&x)	void "complicated implementation"
7	#define Identity	NN.layers->Identity(&x)	x
		NO_BACKPROP SUPPORT
8	#define BinaryStep	NN.layers->BinaryStep(&x)	(x < 0) ? 0 : 1
9	#define Softplus	NN.layers->Softplus(&x)	log(1 + exp(x))
10	#define SiLU	NN.layers->SiLU(&x)	x / (1 + exp(-x))
11	#define GELU	NN.layers->GELU(&x)	(1/2) * x * (1 + erf(x / sqrt(x)))
12	#define Mish	NN.layers->Mish(&x)	x * Tanh(log(1 + exp(x)))
13	#define Gaussian	NN.layers->Gaussian(&x)	exp(-(x*x))
		Derivative Functions
0	#define Sigmoid	NN.layers->SigmoidDer(&fx)	fx-fx*fx
1	#define Tanh	NN.layers->TanhDer(&fx)	1-fx*fx
2	#define ReLU	NN.layers->ReLUDer(&fx)	(fx>0)?1:0
3	#define LeakyELU	NN.layers->LeakyELUDer(&fx)	(fx>0)?1:AlphaLeaky
4	#define ELU	NN.layers->ELUDer(&fx)	(fx>0)?1:fx+AlphaELU
5	#define SELU	NN.layers->SELUDer(&fx)	(fx>0)?LamdaSELU:fx+AlphaSELU*LamdaSELU
6	#define Softmax	NN.layers->SoftmaxDer(&fx)	fx * (1 - fx)
7	#define Identity	NN.layers->IdentityDer(&x)	x

if you want to use other activation function from the default one, just define one other:

#define Sigmoid //[default] No need definition, for single activation across network
#define Tanh
#define ReLU
#define LeakyELU
#define ELU
#define SELU
...

Use any activation function per layer-to-layer, like :

#define ACTIVATION__PER_LAYER
#include <NeuralNetwork.h>

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   1, ..., 2, 0};

// Tanh > ... > ReLU > Sigmoid

If you want to drastically reduce ROM & slightly RAM size you can Define which Functions to use/compile, like:

#define ACTIVATION__PER_LAYER
        #define Sigmoid // 0
        //#define Tanh
        //#define ReLU
        //#define LeakyELU
        #define ELU     // 1
        #define SELU    // 2
        ...
#include <NeuralNetwork.h>

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   1, ..., 2, 0};

// ELU > ... > SELU > Sigmoid

^{⚠️ have in mind that because I used f(x) for derivatives there might be chances of mistakes (?)}

#define Custom Functions

(See also example) You can define up to 5. Every custom function, comes after every each non-custom one (numerically) eg:

#define ACTIVATION__PER_LAYER
        #define Sigmoid // 0
        //#define Tanh
        //#define ReLU
        //#define LeakyELU
        #define ELU  // 1
        #define SELU // 2
        #define CUSTOM_AF1 my_act_fun1 // 3
        #define CUSTOM_AF2 my_act_fun2 // 4
        ...

Define derivative-functions, by just definening ..._DFX:

        #define CUSTOM_AF1 my_act_fun1 
        #define CUSTOM_DF1 

And then use them in your sketch like:

// CUSTOM_DF1 is optional ...
#define ACTIVATION__PER_LAYER
        #define Tanh
        #define CUSTOM_AF1 my_sigmoid 
        #define CUSTOM_DF1

#include <NeuralNetwork.h>

// derivative function must end in "Der" | Limited to f(x), due to optimization reasons
float NeuralNetwork::Layer::my_sigmoidDer(const float &fx){ return fx - fx * fx;      } 
float NeuralNetwork::Layer::my_sigmoid   (const float &x ){ return 1 / (1 + exp(-x)); }


unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   0, ..., 0, 1};

// Tanh > ... > Tanh > my_sigmoid

IMPORTANT NOTE: Be careful commenting in front of #define, see issue #29

DFLOAT Variables Of Activation Functions

Enabling MACRO	Activation Variables	Default	Explenation
#define LeakyELU	NN.AlphaLeaky	0.01	the α of Leaky
#define ELU	NN.AlphaELU	1	the α of ELU
#define SELU	NN.AlphaSELU	1.6733	the α of SELU
#define SELU	NN.LamdaSELU	1.0507	the λ of SELU

Type Other Variables

Note that except from _numberOfInputs and _numberOfOutputs everything else is not valid when you USE_INTERNAL_EEPROM

Type	NN's Variables	Explenation
byte*	NN.ActFunctionPerLayer	if ACTIVATION__PER_LAYER defined
DFLOAT	NN.LearningRateOfWeights	The Learning-Rate-Of-Weights
DFLOAT	NN.LearningRateOfBiases	The Learning-Rate-Of-Biases
IDFLOAT*	NN.weights	If REDUCE_RAM_WEIGHTS_LVL2
Layer*	NN.layers	Layers of NN
	Layer's Variables
IDFLOAT*	NN.layers[i].bias	The bias of an individual layer[i], unless... NO_BIAS or MULTIPLE_BIASES_PER_LAYER is enabled.
DFLOAT*	NN.layers[i].outputs[]	The Output array of an individual layer[i]
IDFLOAT**	NN.layers[i].weights[][]	if not REDUCE_RAM_WEIGHTS_LVL2
DFLOAT*	NN.layers[i].preLgamma[]	The γ-error of previous layer[i-1]
unsigned int	NN.layers[i]._numberOfInputs	The Layer[i]'s Number Of inputs\nodes
unsigned int	NN.layers[i]._numberOfOutputs	The number-Of-Outputs for an individual layer[i]

#define MACRO Properties

#define _1_OPTIMIZE 0B00000000

_1_OPTIMIZE		Action	Keyword
0B00000000		Nothing
0B10000000	⚠️	Use PROGMEM instead of RAM Enables the use of programmable-memmory instead of RAM, to store and use weights and biases	USE_PROGMEM
0B01000000	⚠️📌	Deletes previous layer's Outputs Highly-Recommended because: for each layer-to-layer input-to-ouput operation of internal feedforward, it deletes the previous layer's outputs. Important note: in case you want to delete[] NN->layers[NN->numberOflayers - 1].outputs; make sure afterwards to ...outputs = NULL (if you plan to feedforward again later in your sketch). Reducing RAM by a factor of ((the_sum_of_each_layer'_s _numberOfOutputs) - (_numberOfOutputs of_biggest_layer) *(4[float] or 8[double])Bytes ) approximately i think ?	REDUCE_RAM_DELETE_OUTPUTS
0B00100000	❌	Reduces RAM for Weights, level 1 (Partially reduce) Not yet implimented	REDUCE_RAM_WEIGHTS_LVL1
0B00010000	📌	Reduces RAM for Weights, level 2 by a factor of (number_of_layers-1)*2 Bytes	REDUCE_RAM_WEIGHTS_LVL2
0B00001000	🟢	Deletes previous layer's Gamma Always enabled (not switchable yet.)	REDUCE_RAM_..._LAYER_GAMMA
0B00000100	ⓘ	Reduces RAM using static reference ... to the NN-object (for layers) | by a factor of 2*(number_of_layers - 1 or 2)bytes. (With this optimization) Note that, when you are using multiple NN-objects interchangeably in your sketch, you should either update NN.me manually before using the next one like NN.me = &NN2 or just use _2_OPTIMIZE 0B00000010 instead	REDUCE_RAM_STATIC_REFERENCE
0B00000010	📌	Disables MSE function Disables the default loss function | Reduces ROM, RAM & CPU consumption, althought usually needed for backpropagation	DISABLE_MSE
0B00000001	ⓘ	Use 8-Byte double instead of float This will work only if your MCU supports 8byte doubles eg. Arduino UNO DOESN'T (see also example)	USE_64_BIT_DOUBLE
_2_OPTIMIZE
0B10000000	⚠️	Use internal EEPROM instead of RAM Weights, biases, and activation functions stored-into and used-from the internal EEPROM of the MCU. Additionally, this means REDUCE_RAM_WEIGHTS_LVLX has no effect. see also: example	USE_INTERNAL_EEPROM
0B01000000		Use NN without biases It disables the use of biases in the entire NN	NO_BIAS
0B00100000		Use more than 1 bias, layer-to-layer Enables the use of a unique bias for each unit\neuron of each layer-to-layer	MULTIPLE_BIASES_PER_LAYER
0B00010000		Use F() macro for print function Serial.print(...) strings, normally saved in RAM. This ensures strings are stored in PROGMEM (At least for Arduino boards)	MULTIPLE_BIASES_PER_LAYER
0B00001000	📌	Use int16_t quantization Weights and biases are stored as int16_t (2-bytes each). During the proccess of feedforward each individual weight or bias: temporarily converts back to it's equivalent float [...] Reduces memmory-footprint by a factor of half the size of the "equivalent" float weights and biases. Slightly CPU intensive. (See also: Training > int-quantization + details)	USE_INT_QUANTIZATION
0B00000100		Use int8_t quantization Weights and biases are stored as int8_t (1-byte each). During the proccess of feedforward each individual weight or bias: temporarily converts back to it's equivalent float [...] Reduces memmory-footprint by a factor of half the size of the "equivalent" int16_t weights and biases. Slightly CPU intensive. (See also: Training > int-quantization + details)	USE_INT_QUANTIZATION
0B00000010		REDUCE_RAM_STATIC_REFERENCE for multiple NN It does the same thing as REDUCE_RAM_STATIC_REFERENCE but for multiple NN objects (instead for just one). If you use FeedForward_Individual It is recommended to fallback to the original _1_OPTIMIZE 0B00000100 and manually change NN.me	...FOR_MULTIPLE_NN_OBJECTS
0B00000001	⚠️	Disables backpropagation Disabling backpropagation when it's not automatically disabled, helps reduce the size of your sketch.	NO_BACKPROP

Don't use keywords to define optimizations, it won't work, use _X_OPTIMIZE

• ⚠️ = Backpropagation is not allowed
• 🟢 = Always enabled _{^{(not switchable yet.)}}
• ❌ = Not yet implimented
• 📌 = Recommended

👨‍💻 Training

To train a neural-network, you can use Tensorflow to do so. Here's a basic python example:

# pip install tensorflow
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import LearningRateScheduler
import tensorflow as tf
import numpy as np


# Define if you want to use biases
IS_BIASED = True

# Enable 32-bit floating-point precision
tf.keras.backend.set_floatx('float32')

# Define the XOR gate inputs and outputs
inputs  = np.array([
    [ 0, 0, 0 ], 
    [ 0, 0, 1 ], 
    [ 0, 1, 0 ], 
    [ 0, 1, 1 ], 
    [ 1, 0, 0 ], 
    [ 1, 0, 1 ], 
    [ 1, 1, 0 ], 
    [ 1, 1, 1 ]
], dtype = np.float32)
outputs = np.array([[0], [1], [1], [0], [1], [0], [0], [1]], dtype = np.float32)
input_size = 3

# Create a simple convolutional neural network
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(input_size,)), # Input layer (no bias) 
    tf.keras.layers.Dense(3, activation='sigmoid', use_bias=IS_BIASED), # Dense  3 units 
    tf.keras.layers.Dense(1, activation='sigmoid', use_bias=IS_BIASED)  # Output 1 unit 
])

# Compile the model
optimizer = Adam(learning_rate=0.031)
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

# Train the model
model.fit(inputs, outputs, epochs=900, verbose=0)

# Evaluate the model on the training data
loss, accuracy = model.evaluate(inputs, outputs)
print(f"Model accuracy: {accuracy * 100:.2f}%")

# Predict XOR gate outputs
predictions = model.predict(inputs)
print("Predictions:")
for i in range(len(inputs)):
    print(f"Input: {inputs[i]}, Predicted Output: {predictions[i][0]:.7f}")

# Print biases and weights
# (IMPORTANT NOTE! they are printed as w[i][j] not w[j][i] | outputs * inputs)
print()
weights_biases = model.get_weights()

print("#define _1_OPTIMIZE 0B01000000 // Highly-Recommended Optimization For RAM")
if IS_BIASED:
    print("#define _2_OPTIMIZE 0B00100000 // MULTIPLE_BIASES_PER_LAYER \n")
    print('float biases[] = {')
    for l, (w, b) in enumerate(zip(weights_biases[::2], weights_biases[1::2])):
        print('  ', end='')
        for j in range(0, w.shape[1]):
            print(b[j], end=', ')
        print()
    print('};\n')
else:
    print("#define _2_OPTIMIZE 0B01000000 // NO_BIAS \n")

print('float weights[] = {', end="")
for l, (w, b) in enumerate(zip(weights_biases[::2], weights_biases[1::2])):
    print()
    for j in range(0, w.shape[1]):
        print('  ', end='')
        for i in range(0, w.shape[0]):
            print(w[i][j], end=', ')
        print()
print('};\n')

Int quantization

Assuming you already have either enabled int16_t or int8_t... before proceeding with the example, you should know that the default range of weights (that maps floats to ints) , is set to 200.0 for int16_t and 51.0 for int8_t via this simple formula:

// FLOAT RANGE FOR INT16 = (100.0) - (-100.0) = 200.0 | MAX - MIN
// FLOAT RANGE FOR INT8  = ( 25.5) - (- 25.5) =  51.0 | MAX - MIN

You can change that by defining your own value in your sketch, like:

#define Q_FLOAT_RANGE 40.0 // (20.0) - (-20.0) = 40.0 | Ensure you used a dot!

Now click here to see the full int-quantization training example

# pip install tensorflow
from tensorflow.keras.optimizers import Adam
import tensorflow as tf
import numpy as np


IS_BIASED  = True      # Define if you want to use biases
IS_PROGMEM = True      # Use PROGMEM or not
INT_RANGE  = 65535     # Range of int16_t -32768 to 32767 | for int8_t use 255 
FP__RANGE  = 200.0     # Range of weights -100   to 100   | for int8_t use 51
TYPE_NAME  = 'int16_t' # or 'int8_t'


def quantize_float32_to_int(w):
    S = (FP__RANGE) / (INT_RANGE)
    return round(w / S) # + Z

def int_to_float32(q):
    return np.float32(np.float32(FP__RANGE) / (INT_RANGE)) * np.float32(q)


# Enable 32-bit floating-point precision
tf.keras.backend.set_floatx('float32')

# Define the XOR gate inputs and outputs
inputs = np.array([
    [0, 0, 0], 
    [0, 0, 1], 
    [0, 1, 0], 
    [0, 1, 1], 
    [1, 0, 0], 
    [1, 0, 1], 
    [1, 1, 0], 
    [1, 1, 1]
], dtype=np.float32)
outputs = np.array([[0], [1], [1], [0], [1], [0], [0], [1]], dtype=np.float32)
input_size = 3

# Create a simple convolutional neural network
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(input_size,)),  # Input layer (no bias)
    tf.keras.layers.Dense(3, activation='sigmoid', use_bias=IS_BIASED),  # Dense 3 units
    tf.keras.layers.Dense(1, activation='sigmoid', use_bias=IS_BIASED)  # Output 1 unit
])

# Compile the model
optimizer = Adam(learning_rate=0.031)
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

# Train the model
model.fit(inputs, outputs, epochs=1000, verbose=0)

# Evaluate the model on the training data
loss, accuracy = model.evaluate(inputs, outputs)
print(f"original Model accuracy: {accuracy * 100:.2f}%")

weights_biases = model.get_weights()

if IS_PROGMEM:
    print("\n#define _1_OPTIMIZE 0B11000000 // PROGMEM + Highly-Recommended Optimization For RAM")
else:
    print("\n#define _1_OPTIMIZE 0B01000000 // Highly-Recommended Optimization For RAM")

# Quantize float32 biases to intX_t, print and then back to float32
# (IMPORTANT NOTE! they are printed as w[i][j] not w[j][i] | outputs * inputs)
if IS_BIASED:
    if TYPE_NAME == 'int16_t':
        print("#define _2_OPTIMIZE 0B00101000 // MULTIPLE_BIASES_PER_LAYER + int16_t quantization \n")
    else:
        print("#define _2_OPTIMIZE 0B00100100 // MULTIPLE_BIASES_PER_LAYER + int8_t quantization \n")
    print(('const PROGMEM ' if IS_PROGMEM else '') + TYPE_NAME + ' biases[] = {')
    for l, (w, b) in enumerate(zip(weights_biases[::2], weights_biases[1::2])):
        print('  ', end='')
        for j in range(0, w.shape[1]):
            print(quantize_float32_to_int(b[j]), end=', ')
            b[j] = int_to_float32(quantize_float32_to_int(b[j]))
        print()
    print('};\n')
else:
    if TYPE_NAME == 'int16_t':
        print("#define _2_OPTIMIZE 0B01001000 // NO_BIAS + int16_t quantization \n")
    else:
        print("#define _2_OPTIMIZE 0B01000100 // NO_BIAS + int8_t quantization \n")

# Quantize float32 weights to intX_t, print and then back to float32
print(('const PROGMEM ' if IS_PROGMEM else '') + TYPE_NAME + ' weights[] = {', end="")
for l, (w, b) in enumerate(zip(weights_biases[::2], weights_biases[1::2])):
    print()
    for j in range(0, w.shape[1]):
        print('  ', end='')
        for i in range(0, w.shape[0]):
            print(quantize_float32_to_int(w[i][j]), end=', ')
            w[i][j] = int_to_float32(quantize_float32_to_int(w[i][j]))
        print()
print('};\n')

# Load quantized weights for NN evaluation
model.set_weights(weights_biases)

# Evaluate the model on the training data
loss, accuracy = model.evaluate(inputs, outputs)
print(f"{TYPE_NAME} Model accuracy: {accuracy * 100:.2f}%")

# Print predictions
print(f"{TYPE_NAME} Predictions:")
predictions = model.predict(inputs)
for i in range(len(inputs)):
    print(f"Input: {inputs[i]}, Predicted Output: {predictions[i][0]:.7f}")

(See also: pretrained-quantized-ino-example)

IMPORTANT NOTE: See how weights and biases are printed at the end of the script and make sure you have (on top of your sketch) enabled\defined _2_OPTIMIZE 0B00100000 // MULTIPLE_BIASES_PER_LAYER or _2_OPTIMIZE 0B01000000 // NO_BIAS depending on your needs of use. Additionally, if you want to use just 1 bias per layer-to-layer don't use any of those 2 optimizations (Althought, just so you know... Tensorflow doesn't seem to support 1 bias per layer-to-layer). Finally make sure to use float32 unless your MCU is compatible and you want to USE_64_BIT_DOUBLE-optimization

(see also examples on how to train a NN directly on an MCU)

A HUGE THANK YOU!

I want to really thanks Underpower Jet for his amazing tutorial, by bringing it more to the surface. Because after all the videos and links I came across, he was the one that made the most significant difference to my understanding of backpropagation in neural networks. Plus, I would like to thanks: giant_neural_network for this and this, 3Blue1Brown for this, the authors of ✨ this scientific article for referencing me, Ivo Ljubičić for using my library for his ✨ master thesis, Arduino community and everyone else who gave me the oportunity to learn and make this library possible to exist [...]

🌐 Research

Here most of the resources I came across the internet, I recomend you to have a look if you want to (but please stay aware of the fact that for some of those sites, I had only opened them checked something and then closed them in a matter of seconds [so, please don't get them all seriously])

22\11\2023

• Code Related:
• • Handy C++ "cheat sheet"
• • Macros:
• • • Do not put comments in front of #define whatever
• • • is #ifdef __SD_H__ considered a bad practice?
• • • C++11-style [[unused]] attribute in gcc?
• • • comments in c macro definition
• • • Using #undef before #define
• • Arduino:
• • • StackOverflow:
• • • • PROGMEM : pgm_read_float_near() equivalent for double? (Arduino Due)
• • • • warning: inline variables are only available with ‘-std=c++17’ or ‘-std=gnu++17’ how to suppress it? [closed]
• Neural Network Related
• • General:
• • • Large values of weights in neural network

12\08\2021

• Neural Network Related
• • Videos:
• • • How convolutional neural networks work, in depth
• • • How to use the loss function categorical crossentropy
• • • Tutorial 4: How to train Neural Network with BackPropogation
• • • Neural Networks Part 8: Image Classification with Convolutional Neural Networks
• • Softmax:
• • • The Softmax : Data Science Basics
• • • Softmax Function in Deep Learning
• • • The SoftMax Derivative, Step-by-Step!!!
• • • Derivative of Sigmoid and Softmax Explained Visually
• • • Softmax and Cross Entropy Gradients for Backpropagation
• • StackOverflow\Exchange:
• • • What is batch size in neural network?
• • • how to do weight clipping after gradient updates?
• • • Cost function turning into nan after a certain number of iterations
• • • Why use softmax only in the output layer and not in hidden layers?
• • • Why is step function not used in activation functions in machine learning?
• • • if softmax is used as an activation function for output layer, must the number of nodes in the last hidden layer equal the number of output nodes?
• • General:
• • • Is Sigmoid the only activation function compatible with the binary crossentropy loss function
• • • Understanding binary cross-entropy / log loss: a visual explanation
• • • Understanding RMSprop
• • • MLP_C Arduino Library
• Code Related:
• • Tools:
• • • Compiler Explorer
• • Macros:
• • • 3.7.1 Standard Predefined Macros
• • • Diagnostic Pragmas
• • • PREPROCESSOR DIRECTIVES 2
• • Arduino:
• • • StackOverflow:
• • • • Why there's no erf() function in Arduino's math.h library?
• • • • How to Serial.println text and variable's values in the same line
• • • • Why AVR-GCC compiler throws an error when overloading with the same variables but as PROGMEM?
• • • General:
• • • • Natural Log of a Number
• General:
• • Math:
• • • What the cap over letters in math means
• • • When log is written without a base, is the equation normally referring to log base 10 or natural log?
• • Grammar:
• • • Plural of feedforward
• • Just "Random":
• • • Delving Deep into Rectifiers
• • • Linear & Identity Activation Functions
• • • “softmax derivative c++” Code Answer
• • • The Softmax function and its derivative
• • • -

xx\xx\202x

• Neural Network Related

• • kind of Intresting To me
• • • Training Deep Neural Nets
• • • Activation Functions - EXPLAINED!
• • • (Of course wiki lol) Activation function
• • • MNIST functions Intresting Simple Paper
• • General
• • • How many times is backprop used in epoch?
• • • Back-propagation in Neural Nets with >2 hidden layers
• • • Normalized Function, Normalized Data and Normalization
• • Activation Function Related
• • • Why the 6 in relu6?
• • • Are relus incapable of solving certain problems
• • • (Seems Intresting!) Modern Activations
• • • Machine learning using ReLu return NaN
• • • Machine learning using ReLu return NaN
• • • NaN loss when training regression network
• • • How to avoid NaN in using ReLU + Cross-Entropy? duplicate
• • • Activation Functions Explained - GELU, SELU, ELU, ReLU and more
• • • Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML
• • • For deep learning, With activation relu the output becomes NAN during training while is normal with tanh
• • Gradient Explosion and clipping Related
• • • A clipped ReLU
• • • (Seems Intersting) Paper
• • • Complete Guide To Exploding Gradient Problem
• • • Understanding Gradient Clipping (and How It Can Fix Exploding Gradients Problem)
• • • Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?
• • • (Seems Intersting) Keras ML library: how to do weight clipping after gradient updates? TensorFlow backend
• • MNIST Related
• • • MNIST-Visualizer
• • • THE MNIST DATABASE
• • • ReLU vs sigmoid in mnist example
• • • Why not use more than 3 hidden layers for MNIST classification?
• • • Neural network classificator stuck at 85% accuracy for mnist dataset
• • • (Arduino Project) MNIST - (approach) downsampled to 9x9 - 170 neurons
• • • MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)
• • • What is the smallest model in terms of number of parameters that can achieve results over 99% on MNIST dataset?

• Related to Programming

• • to C-type Languages
• • • c++ making a function that always runs when any other function of a class is called
• • • int.. = {1,2,}; Weird comma allowed. Any particular reason?
• • • Declare an array of pointers to functions in Visual C++
• • • Function Pointer Array to a Method within a Class
• • • Inability to overload Dot '.' operator in c++
• • • How to get address of a pointer in c/c++?
• • • CSharp 16 bit float conversions
• • • Understanding * oppperator
• • • Array of function pointers
• • • Issue I came across c++
• • • Issue I came across c++
• • • c++ constexpr
• • • c++
• • to Python
• • • Python Floating Point Precision numpy
• • • Python Floating Point Precision Tensorflow
• • Other
• • • Aspect-oriented programming
• • • Metaprogramming

• Arduino Related

• • Reading and Writing Data to External EEPROM Using Arduino
• • avrdude: verification error, first mismatch at byte 0x7800
• • avrdude: verification error, first mismatch at byte 0x7800
• • KEYWORDS.TXT

• #MACROS / pre-processor directives

• • Is it possible to define same macro more than once?
• • (Very Useful) Print Content of Defined Variable
• • How to remove gcc warning on pragma region
• • (Very Useful) Generate Functions
• • (seems intresting) Add numbers
• • 3.8 Undefining and Redefining Macros
• • (Seems Intresting) Iterate through #defines
• • (Seems Intresting) Checking type of a variable
• • Macro functions
• • (Random)

Old Searches:

| | Intresting |NN.| Neural Network(s) |A.| Arduino etc. |-| Mostly .NET & Other |*| Maybe Intresting?

NNs	PROGMEM	Define directive	Other & "Random"
Playlist	Arduino	Tutorial	A. Initialize Array Values
Playlist	Manual	Arduino Define	A. Inheritance,destructors?
BackPropagation	Examples	Determining board	A. Identifying Arduino type?
Math Chain Rule	+ Post	define extern?	A. Create compile error?
Getting Started	+ Pointers	.ino Determining Board	A. Measuring Memory Usage
+ BackProp Tutorial	+ Double Info	Understanding #if ?	A. External Memory
+ BackProp Tutorial	read-only?	Random Defined Site?	A. ATtiny85 Math Issues?
+ Complete NN chart !	flash to RAM?	Loading local libraries?	A. Attiny85 External Mem.?
+ MIT Deep RL	Info Near Far?		A. Splitting Array?
+ MIT Deep Learning	Example		A. Importing Loads Everything?
- .NET Framework	What is PGM_P?		NN. Backprop. For Dummies YT
- .NET ! 1,2,3,4,5,6	Passing Array?		NN. Convolutional (Math- code)
- C# Implementation, 2	Passing Array?		NN. In 4 minutes - Python
- .NET Deep NN	Reading?		NN. Quick Intro
* Predicting Languages	Easy data handling?		NN. -
* MIT Recurrent NNs	Reading Long?		NN. PyConSG 2016
- 2007 .NET Img Recognition	Multi-D Array?		NN. Simple 9 lines of Py-code
- C# Handwritten Recognition	Attiny85 mem?		NN. MIT Press book
Youtube Chanel	Prob Array use?		NN. A Beginner's Guide
+ Recurrent Explained	Double or float?		NN. MIT courses?
- .NET 1,2,3			NN.Back Propagation
- C# Handwritten Recognition			NN. MLP Maths?
+ Python NN From Scratch			NN. Math. Notations Into Code
* How Backpropagation!			NN. Maths Into Code
linear Regression			NN. (TAA),(BDI) Architecture
+ MLP			NN. fast.ai ?
+ MLP			NN. deeplearning.net
🇬🇷 Νευρωνικά Δίκτυα			NN. BackProp Python
🇬🇷 Γενετικός Αλγόριθμος			NN. C# Code
- MS NN Classification			NN. Implement BackProp Python
- VB ML Q-Learning			NN. Java Tut Playlist ?
* handwritten Recognition			NN. BackProp for Dummies
* Deep Learning in 6 weeks?			NN. Wiki Backprop Math
* Playlist			NN. Looking Inside
- SciBasic

💗 Donation

Please consider donating something, even the least amount would be really appreciated. I need it. | Monero address: ^{87PVyQ8Vt768hnvVyR9Qw1NyGzDea4q9Zd6AuwHb8tQBU9VdRYjRoBL7Ya8yRPVQakW2pjt2UWEtzYoxiRd7xpuB4XSJVAW}

🕳️ Outro

• Forgive me if I've made any mistakes and please don't take me seriously with every claim i make, I am mainly "self taught" in this field of NeuralNetworks, I am not a professional programmer nor do I have a good knowledge in many of the fields used to create this library, I just make things because I love to [...]

• Also looking for jobs, if you are interested let me know, I really like working with embeded systems, C\C++, python, CLIs and etc.

if you want to help me&others to educate ourselves better and if you have a love and passion for sharing and helping, then I suggest you to join our discord server 🤍

My Instagram account is: giorgos.xou ;) feel free to ask me anything