LC Part 1

Part 1 - Neuron

Every neural network - artificial or otherwise - is created with the simplest of building blocks, the neuron. In this tutorial we will show you how to create your very own neural network library in JavaScript.

neuron.js

Step 1 - Creating the module

function Neuron() {
  // Code here...
}

module.exports = Neuron;

Every JavaScript package can be bundled by with tools like webpack or browserify by exporting them as a module (i.e. as shown above). This allows you to re-use your package in Node.js (server) and Vanilla JavaScript (Browser); since the browser and Node.js have two different import package/module import systems, it's important that we account for them when creating our library.

Step 2 - Importing dependencies

const uid = require("cuid");

function Neuron() {
  // Code here...
}

module.exports = Neuron;

When creating a new package it's often a lot easier to not start from scratch; if you can find an awesome package that can you speed some things along - great! Now you can build cool things faster; that is the awesomeness of open-source and tools like GitHub and NPM.

For this project, I wanted to import a universally unique ID generator so we can refer to neurons across threads and computers in the future; ID's will also allow us to have some cool optimizations in the next few parts.

Step 3 - Adding basic properties

...

function Neuron(bias) {
  this.id = uid();
  this.bias = bias == undefined ? Math.random() * 2 - 1 : bias; // this.bias ∈ ℝ && -1 < this.bias < 1

  this.incoming = {
    neurons: new Map(),
    weights: new Map()
  }
  this.outgoing = {
    neurons: new Map(),
    weights: new Map()
  }

  this._output; // f'(x)
  this.output; // f(x)
  this.error; // E'(f(x))

  // Code here...
}

...

Though it may seem like a lot, these simple properties are all we're going to need to create our first simple neural network library in JavaScript. Here's a brief explanation of each and why we're using them.

An important part of creating neural networks is understanding the relationships and connections between neurons; we can store this information in this.outgoing & this.incoming. The most basic neural network - which is very similar to what we're going to build here - works very similar to networks of people; neurons can be thought of as people working together towards some common goal and relaying information and weights can be thought of the trust these people have in each others information - so it's important that we store these too (this.[incoming/outgoing].[neuron/weight]) so we can edit, update, and optimize these later. We store these values in an Object because of the advantage of searching through objects (O(1)) vs arrays (O(n)) in JavaScript.

Again, much like people, we can give each neuron a bias and a unique perspective on the information that they process, which can be update as the neuron, or group thereof, begin training on real data and learning (updating their bias and weights).

Lastly, to effectively learn like people we have to give the neuron the ability to remember information and learn from its mistakes by reflecting on the past. For this simple example, we will be giving VERY little memory to each neuron and they will only have to the ability to reflect on the action immediately just taken when updating their weights and biases; we will store this information in this._output & this.output. Once the network is done updating itself, it will also help all the other neurons that gave it information and helped it create its decision feedback - think of it of like sharing positive/negative criticisms; we will store this information in this.error.

Step 4 - Connecting neurons

...

function Neuron() {
  ...

  this.connect = function(neuron, weight) {
    this.outgoing.neurons[neuron.id] = neuron;
    neuron.incoming.neurons[this.id] = this;
    this.outgoing.weights[neuron.id] = neuron.incoming.weights[this.id] = weight == undefined ? Math.random() * 2 - 1 : weight; // weight ∈ ℝ && -1 < weight < 1
  }

  // Code here...
}

...

Alone neurons are much weaker than they are together, with neuron.connect() we can begin to build small networks of neurons and enable the creation of larger networks as well. neuron.connect() allows us to track the activity of neighboring neurons and pass information to/from them.

Step 5 - Using a neuron

...

function Neuron() {
  ...

  this.activate = function(input) {
    const self = this;
    
    function sigmoid(x) { return 1 / (1 + Math.exp(-x)) } // f(x)
    function _sigmoid(x) { return sigmoid(x) * (1 - sigmoid(x)) } f'(x)
    
    if(input) {
      this._output = 1; // f'(x)
      this.output = input; // f(x)
    } else {
      // Σ (x • w)
      const sum = Object.keys(this.incoming.targets).reduce(function(total, target, index) {
        return total += self.incoming.targets[target].output * self.incoming.weights[target];
      }, this.bias);
      
      this._output = _sigmoid(sum); // f'(x)
      this.output = sigmoid(sum); // f(x)
    }
    
    return this.output;
  }

  // Code here...
}

...

Step 6 - Teaching a neuron

...

function Neuron() {
  ...

  this.propagate = function(target, rate=0.3) {
    const self = this;
    
    //𝛿E /𝛿squash
    const sum = target == undefined ? Object.keys(this.outgoing.targets).reduce(function(total, target, index) {
        // Δweight
        self.outgoing.targets[target].incoming.weights[self.id] = self.outgoing.weights[target] -= rate * self.outgoing.targets[target].error * self.output;
        
        return total += self.outgoing.targets[target].error * self.outgoing.weights[target];
      }, 0) : this.output - target;
    
    // 𝛿squash/𝛿sum
    this.error = sum * this._output
    
    // Δbias
    this.bias -= rate * this.error;
    
    return this.error;
  }

  // Code here...
}

...