ml/perceptron.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPLv4Acab0ByD/5WRBuzUXa",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/werowe/HypatiaAcademy/blob/master/ml/perceptron.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#A Perceptron Illustrated\n",
        "\n",
        "\n",
        "\n",
        "#Neural Network\n",
        "\n",
        "Each circle is a **neuron**.  Read from left to right we have the input layer, which is the training data.  The middle layers are the **hidden layers**.  The rightmost layer is the **output** layer. The output are scalara or probabilities. In this case here are two.  This network would work for a binary classification problem.  An example of that is what we describe below.\n",
        "\n",
        "![](https://miro.medium.com/v2/resize:fit:1100/format:webp/1*v1ohAG82xmU6WGsG2hoE8g.png)\n",
        "\n",
        "(**Credit**: Image source https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf. )\n",
        "\n",
        "\n",
        "#Single Layer Single Neuron Neural Network\n",
        "\n",
        "![](https://raw.githubusercontent.com/werowe/HypatiaAcademy/d760406854bf85214e87b281d91badcb66974fde/images/single-layer-perceptron.png)\n",
        "\n",
        "\n",
        "\n",
        "#The AND Problem\n",
        "The **AND** problem is a class problem in neural networks.  \n",
        "\n",
        "We have four possibilities, where 1 is true and 0 is false.\n",
        "\n",
        "| A      | B      | AND |\n",
        "| ------------- | ------------- |  ------------- |\n",
        "| 1| 1 | 1 |\n",
        "| 0| 1| 0 |\n",
        "| 1 | 0 | 0 |\n",
        "| 0| 0 | 0 |\n",
        "\n",
        "\n",
        "# Recall\n",
        "\n",
        "X means input, like true (1) false (0).  before it was colors in a 28 x 28 pixel hand writing writing image\n",
        "\n",
        "what the neural network does it is calculates m\n",
        "\n",
        "remember with linear and logistic regression when we calculated m we use the least squared errors technique.  that was:\n",
        "\n",
        "1.  pick an m, pick an (but not always pick an m)\n",
        "2.  calculate xm + b\n",
        "3.  calculate error\n",
        "4.  change\n",
        "5.  repeat until done\n",
        "\n",
        "y = mx + b\n",
        "\n",
        "m is a matrix, [1,2,3]\n",
        "x is a matrix, [1,2,3]\n",
        "b is still a scalar, a single number\n",
        "\n",
        "M=[1,2,3]\n",
        "X=[4,5,6]\n",
        "b = 2\n",
        "\n",
        "y = M * X + b\n",
        "\n",
        "How to mutiply two matrices.\n",
        "\n",
        "[1,2,3] [4,\n",
        "         5,\n",
        "         6]\n",
        "\n",
        "1 * 4 + 2 * 5 + 3 * 6 - 2\n",
        "\n",
        "# How does it calculate m\n",
        "\n",
        "1. guess.  so first it will pick W=(random, random, random)\n",
        "\n",
        "2. mutiply [x1, x2, b] * [w1, w2, w3]\n",
        "\n",
        "3.  check and see if the answer matches x3.  it if does you are done.\n",
        "\n",
        "4.  it is does not then try a new value for [w1, w2, w3].\n",
        "\n",
        "5.  if we picked random numbers for [w1, w2, w3] and they were wrong, then how do we pick new numbers?\n",
        "\n",
        "w1 + (add something)\n",
        "\n",
        "Gregory Hinton (university of montreal), then he worked for google, then he quit.  why did Gregory Hinton quit.  Because he believe AI is going be dangerous one day.  For example, in Ukraine already they have AI weapons.  The Matrix could become real.  \n",
        "\n",
        "What he did was came up with the algorithm to adjust the weights.  Multi variable calculus .  He used stochastic gradient descent.  It uses what is called a partial derivative.\n",
        "\n",
        "If you just guess then the computer could take years to converge.  Converge means find the solution.  In calculus to find minimum values.  For example the minimum value.  \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# Neuron\n",
        "You can think of the neural network of the solution to the Y = W X + b problem where W are the coefficient in the model.  Except W in this model, unlike linear regression, are functions.  It is easiest to this of them as compound functions like g(f(x)) where each hidden layer is one more level in this nesting.\n",
        "\n",
        "It solves the network by muliplying the inputs x1, x2, x3 by the weights w1 and w2 and w3.  w1 and w2 are bias and we is weight.m x1 and x2 are inputs are x3 is observed output (i.e., y)\n",
        "\n",
        " Initially w1 , w2, w3 are just guesses.  This process repeats itself (called **backprogation** by adjusting the weights until they reach their minimum value.  This is usally dones by taking a partial derivative and looking for where the derivative is 0 or closest to zero, which is the **stochastic gradient solution** technique.  Each pass through the network moves the value of the weight in the opposite direction of the gradient.\n",
        "\n",
        "$$\\begin{bmatrix} 1 & 1 & 0 \\end{bmatrix} \\cdot \\begin{bmatrix} w_1 \\\\ w_2 \\\\ w_3 \\end{bmatrix} = 1 \\cdot w_1 + 1 \\cdot w_2 + 0 \\cdot w_3 = w_1 + w_2 $$\n",
        "\n",
        "\n",
        "![](https://miro.medium.com/v2/resize:fit:1302/format:webp/1*UA30b0mJUPYoPvN8yJr2iQ.jpeg)\n",
        "\n",
        "(**Credit**: Image source https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf. )\n",
        "\n",
        "#Activation Function\n",
        "This is a function that takes the value above and turns it into a number between 0 and 1.  \n",
        "\n",
        "Frabcois Chollet, wrote wrote Keras, says, \"It introduces non-linearity into the model, allowing it to learn complex patterns. Common activation functions include ReLU, sigmoid, and softmax.\"\n",
        "\n",
        "\n",
        "\n",
        "![](https://miro.medium.com/v2/resize:fit:1400/format:webp/1*JHWL_71qml0kP_Imyx4zBg.png)\n",
        "\n",
        "(**Credit**: Image source https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf. )\n",
        "\n",
        "#Loss Function\n",
        "\n",
        "Frabcois Chollet:  A **loss function** is a mathematical function used in machine learning to measure the difference between the predicted output of a model and the actual target values. It quantifies how well or poorly the model is performing. The goal of training a model is to minimize the loss function. The **sparse_categorical_crossentropy loss function** is used in machine learning for multi-class classification problems where the target labels are integers (i.e., not one-hot encoded).\n",
        "\n",
        "#Learning Rate\n",
        "This means how much to adjust the weights based upon the loss.  A larger number will make the model conver quicker, but it that has problems as it could skip over function that have more than one perceived minimum this picking the wrong minimum thus leading to the wrong answer.  The very small learing rate would mean the model takes longer to converge.  \n",
        "\n",
        "#Make a Single Perceptron\n",
        "\n",
        "Here we take Arthur Arnx's example solution to the XOR problem and add explanations to make it easier to understand.  It has three neurons and no hidden layer. So it's 1 or 0 going in and 1 or 0 going out.  \n",
        "\n"
      ],
      "metadata": {
        "id": "KlI5xkSy4UTL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The And Problem\n",
        "\n",
        "| A      | B      | AND |\n",
        "| ------------- | ------------- |  ------------- |\n",
        "| 1| 1 | 1 |\n",
        "| 0| 1| 0 |\n",
        "| 1 | 0 | 0 |\n",
        "| 0| 0 | 0 |\n",
        "\n",
        "\n",
        "deductive logic.  philopsophy.  In order for a statement using AND to be true then every on both sides of AND must be true\n",
        "\n",
        "TRUE (1) and TRUE (1) = TRUE (1)\n",
        "\n",
        "FALSE (0) and TRUE(1)=FALSE\n",
        "\n",
        "1 AND 0 = 0\n",
        "\n",
        "\n",
        "0 AND\t0\t= 0\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "QndR0a43StDF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Perceptron\n",
        "\n",
        "A perceptron is a type of artificial neuron used in machine learning and artificial intelligence. It is a simple model of a biological neuron and serves as a fundamental building block for more complex neural networks. Here's a breakdown of how a perceptron works:\n",
        "\n",
        "1. **Inputs**: A perceptron receives multiple input signals, each of which is associated with a weight. These inputs are typically denoted as \\( x_1, x_2, \\ldots, x_n \\).\n",
        "2. **Weights**: Each input has a corresponding weight (\\( w_1, w_2, \\ldots, w_n \\)), which determines the importance of the input in the final decision. The weights are adjusted during the training process to improve the performance of the perceptron.\n",
        "3. **Summation**: The perceptron computes a weighted sum of its inputs, which is often written as \\( \\sum_{i=1}^n w_i x_i \\).\n",
        "4. **Activation Function**: This weighted sum is then passed through an activation function, which determines the output of the perceptron. In the simplest form of a perceptron, a step function is used, which outputs 1 if the weighted sum is above a certain threshold and 0 otherwise. 5. **Output**: Based on the result of the activation function, the perceptron produces an output, which can be used for binary classification tasks. Mathematically, the output \\( y \\) of a perceptron can be represented as:\n",
        "\n",
        "$$y = \\begin{cases} 1 & \\text{if } \\sum_{i=1}^n w_i x_i + b > 0 \\\\ 0 & \\text{otherwise} \\end{cases} $$\n",
        "\n",
        "where $ b $ is a bias term that shifts the decision boundary. The perceptron learning algorithm involves adjusting the weights and bias based on the error between the predicted output and the actual output for a set of training examples. This adjustment process is typically done using a rule such as the delta rule, which updates the weights in proportion to the error and the input values. Perceptrons can be used to solve linearly separable problems, but they are limited in their capacity to solve more complex problems. To handle non-linear problems, multiple perceptrons can be combined into multi-layer networks, which are the basis for more advanced neural network architectures."
      ],
      "metadata": {
        "id": "dA-4EZcUxJTr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "For a single-layer perceptron to correctly solve the XOR problem, it must be capable of learning non-linearly separable functions, which a single-layer perceptron cannot do. However, for the AND and OR functions, a single-layer perceptron can solve these problems."
      ],
      "metadata": {
        "id": "OlwTNN52tl3n"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Explanation\n",
        "\n",
        "1. AND Gate Training Data:\n",
        "\n",
        "Inputs and outputs are defined for the AND problem.\n",
        "\n",
        "2. Parameters:\n",
        "\n",
        "* Learning rate, bias, and number of epochs are set.\n",
        "* Weights Initialization:\n",
        "\n",
        "Weights are initialized randomly.\n",
        "3. Training Loop:\n",
        "\n",
        "* For each epoch:\n",
        "  * Iterate over all training examples.\n",
        "  * Compute the weighted sum and apply the activation function.\n",
        "  * Calculate the error and update the weights.\n",
        "  * Track the total error for the epoch.\n",
        "* Print the error every 100 epochs to monitor training progress.\n",
        "* Stop training if the total error becomes zero.\n",
        "\n",
        "5. Testing Function:\n",
        "\n",
        "* test_perceptron(input1, input2): Performs a forward pass using the learned weights and applies the Heaviside step function to predict the output.\n",
        "\n",
        "6. Testing the Model:\n",
        "\n",
        "* The code tests the model with various inputs and compares the predicted output with the expected output.\n",
        "* It prints the test input, expected output, and predicted output, and checks if the prediction is correct.\n",
        "\n",
        "##Running the Test\n",
        "By running the complete code, you can train the perceptron and test it with different inputs for the AND gate problem. The output will indicate whether the model correctly predicts the output for each test input. This way, you can verify the functionality of the perceptron with the AND gate."
      ],
      "metadata": {
        "id": "l98i1M61valL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "import numpy as np\n",
        "\n",
        "# Define the inputs and outputs for the AND problem\n",
        "inputs = np.array([[0, 0],\n",
        "                   [0, 1],\n",
        "                   [1, 0],\n",
        "                   [1, 1]])\n",
        "\n",
        "outputs = np.array([0, 0, 0, 1])\n",
        "\n",
        "# Initialize parameters\n",
        "lr = 0.1  # Learning rate\n",
        "bias = 1\n",
        "epochs = 1000\n",
        "\n",
        "# Initialize weights randomly\n",
        "weights = [random.random(), random.random(), random.random()]\n",
        "\n",
        "# Training loop\n",
        "for epoch in range(epochs):\n",
        "    total_error = 0\n",
        "    for i in range(len(inputs)):\n",
        "        input1 = inputs[i][0]\n",
        "        input2 = inputs[i][1]\n",
        "        output = outputs[i]\n",
        "\n",
        "        # Forward pass: calculate weighted sum of inputs and bias\n",
        "        outputP = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "\n",
        "        # Activation function: Heaviside step function\n",
        "        if outputP > 0:\n",
        "            outputP = 1\n",
        "        else:\n",
        "            outputP = 0\n",
        "\n",
        "        # Calculate error\n",
        "        error = output - outputP\n",
        "        total_error += abs(error)\n",
        "\n",
        "        # Update weights\n",
        "        weights[0] += error * input1 * lr\n",
        "        weights[1] += error * input2 * lr\n",
        "        weights[2] += error * bias * lr\n",
        "\n",
        "    # Print error at each epoch\n",
        "    if epoch % 100 == 0:\n",
        "        print(f\"Epoch {epoch+1} \\t Error: {total_error}\")\n",
        "\n",
        "    # Check for convergence\n",
        "    if total_error == 0:\n",
        "        print(\"Converged!\")\n",
        "        break\n",
        "\n",
        "print(\"Final weights:\", weights)\n",
        "\n",
        "# Define a function to test the model with a given input\n",
        "def test_perceptron(input1, input2):\n",
        "    # Forward pass: calculate weighted sum of inputs and bias\n",
        "    outputP = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "\n",
        "    # Activation function: Heaviside step function\n",
        "    if outputP > 0:\n",
        "        outputP = 1\n",
        "    else:\n",
        "        outputP = 0\n",
        "\n",
        "    return outputP\n",
        "\n",
        "# Test the model with various inputs\n",
        "test_cases = [\n",
        "    (0, 0, 0),\n",
        "    (0, 1, 0),\n",
        "    (1, 0, 0),\n",
        "    (1, 1, 1)\n",
        "]\n",
        "\n",
        "for test_input1, test_input2, expected_output in test_cases:\n",
        "    predicted_output = test_perceptron(test_input1, test_input2)\n",
        "    print(f\"Test input: [{test_input1}, {test_input2}]\")\n",
        "    print(f\"Expected output: {expected_output}\")\n",
        "    print(f\"Predicted output: {predicted_output}\")\n",
        "    if predicted_output == expected_output:\n",
        "        print(\"The model correctly predicts the output.\\n\")\n",
        "    else:\n",
        "        print(\"The model does not correctly predict the output.\\n\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1rLcyoYBuRbC",
        "outputId": "ea869fd0-b3b6-4471-b336-9ee263c61386"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1 \t Error: 3\n",
            "Converged!\n",
            "Final weights: [0.17786938446498032, 0.11912766429193872, -0.25397049879474204]\n",
            "Test input: [0, 0]\n",
            "Expected output: 0\n",
            "Predicted output: 0\n",
            "The model correctly predicts the output.\n",
            "\n",
            "Test input: [0, 1]\n",
            "Expected output: 0\n",
            "Predicted output: 0\n",
            "The model correctly predicts the output.\n",
            "\n",
            "Test input: [1, 0]\n",
            "Expected output: 0\n",
            "Predicted output: 0\n",
            "The model correctly predicts the output.\n",
            "\n",
            "Test input: [1, 1]\n",
            "Expected output: 1\n",
            "Predicted output: 1\n",
            "The model correctly predicts the output.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# break into pieces\n",
        "\n",
        "import random\n",
        "import numpy as np\n",
        "\n",
        "# Define the inputs and outputs for the AND problem\n",
        "\n",
        "# this is all the combinations FALSE, FALSE; FALSE, TRUE, ...\n",
        "inputs = np.array([[0, 0],\n",
        "                   [0, 1],\n",
        "                   [1, 0],\n",
        "                   [1, 1]])\n",
        "\n",
        "# remember this is supervisor model.  that means it has features and labels.  Like the handwriting recognition\n",
        "# that showed pixel and it gave the answer.  we have that with linear regression too.  So this is why it's\n",
        "# called \"training\" the model.  because give it the answers.\n",
        "\n",
        "# here are the answers\n",
        "\n",
        "outputs = np.array([0, 0, 0, 1])\n",
        "\n",
        "# Initialize parameters\n",
        "lr = 0.1  # Learning rate\n",
        "bias = 1\n",
        "epochs = 1000\n",
        "\n",
        "# Initialize weights randomly\n",
        "\n",
        "# guess\n",
        "weights = [random.random(), random.random(), random.random()]\n",
        "\n",
        "bias = 1\n"
      ],
      "metadata": {
        "id": "wKLLPm2BT1s_"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "inputs"
      ],
      "metadata": {
        "id": "-dm96zvFUk9F",
        "outputId": "22b67b50-f0f6-481a-fe82-44a028ce3b68",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0],\n",
              "       [0, 1],\n",
              "       [1, 0],\n",
              "       [1, 1]])"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "weights"
      ],
      "metadata": {
        "id": "z7GOlQsWUrd1",
        "outputId": "6f33905a-ad56-4df8-d540-b0a916c49ad0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.5658904041960958, 0.7083850733003952, 0.5373989793688908]"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "inputs[1,0]"
      ],
      "metadata": {
        "id": "Uzq4H7ovVd7N",
        "outputId": "848cc695-c714-4768-e921-73e41814212e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#y = M * X + b\n",
        "\n",
        "#y = M * X\n",
        "\n",
        "inputs[0,0] * weights[0] + inputs[1,0] * weights[1] + bias * weights[2]\n",
        "\n"
      ],
      "metadata": {
        "id": "LnQotx_JUw_a",
        "outputId": "6ea3eb21-8482-4186-dce2-47f32c690ef6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5373989793688908"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(inputs)"
      ],
      "metadata": {
        "id": "UlYWlkT3WMtM",
        "outputId": "c1e56f7d-196d-4746-bf53-1cf2e9bfd260",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(len(inputs)):\n",
        "        input1 = inputs[i][0]\n",
        "        input2 = inputs[i][1]\n",
        "        output = outputs[i]\n",
        "\n",
        "        # Forward pass: calculate weighted sum of inputs and bias\n",
        "        outputP = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "        print(input1, input2, output, outputP)\n",
        "\n",
        "\n",
        "# the next thing it will do is adjust the weights.  it will not us Gregory Hinton's algorithm\n",
        "# because this is simple problem\n",
        "\n",
        "\n",
        "\n",
        "# outputP is what is called an activation function.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "        if outputP > 0:\n",
        "            outputP = 1\n",
        "        else:\n",
        "            outputP = 0\n",
        "\n",
        "        print(\"outputP=\", outputP)\n",
        "\n",
        "        # Calculate error\n",
        "        error = output - outputP\n",
        "\n",
        "        print(\"error=\", error)\n",
        "\n",
        "        total_error += abs(error)\n"
      ],
      "metadata": {
        "id": "08NrLedIWHJ5",
        "outputId": "78a3ea43-e611-4143-fbce-3e6700460a43",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 0 0 0.5373989793688908\n",
            "outputP= 1\n",
            "error= -1\n",
            "0 1 0 1.245784052669286\n",
            "outputP= 1\n",
            "error= -1\n",
            "1 0 0 1.1032893835649866\n",
            "outputP= 1\n",
            "error= -1\n",
            "1 1 1 1.811674456865382\n",
            "outputP= 1\n",
            "error= 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Heavside Activation Function\n",
        "\n",
        "```\n",
        "if outputP > 0:\n",
        "            outputP = 1\n",
        "        else:\n",
        "            outputP = 0\n",
        "```\n",
        "\n",
        "H(x) =\n",
        "\\begin{cases}\n",
        "0 & \\text{if } x < 0 \\\\\n",
        "1 & \\text{if } x \\ge 0\n",
        "\\end{cases}\n"
      ],
      "metadata": {
        "id": "fupK6Tb4X_IE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Training loop\n",
        "for epoch in range(epochs):\n",
        "    total_error = 0\n",
        "    for i in range(len(inputs)):\n",
        "        input1 = inputs[i][0]\n",
        "        input2 = inputs[i][1]\n",
        "        output = outputs[i]\n",
        "\n",
        "        # Forward pass: calculate weighted sum of inputs and bias\n",
        "        outputP = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "\n",
        "        # Activation function: Heaviside step function\n",
        "        if outputP > 0:\n",
        "            outputP = 1\n",
        "        else:\n",
        "            outputP = 0\n",
        "\n",
        "        # Calculate error\n",
        "        error = output - outputP\n",
        "        total_error += abs(error)\n",
        "\n",
        "        # Update weights\n",
        "        weights[0] += error * input1 * lr\n",
        "        weights[1] += error * input2 * lr\n",
        "        weights[2] += error * bias * lr\n",
        "\n",
        "    # Print error at each epoch\n",
        "    if epoch % 100 == 0:\n",
        "        print(f\"Epoch {epoch+1} \\t Error: {total_error}\")\n",
        "\n",
        "    # Check for convergence\n",
        "    if total_error == 0:\n",
        "        print(\"Converged!\")\n",
        "        break\n",
        "\n",
        "print(\"Final weights:\", weights)\n",
        "\n",
        "# Define a function to test the model with a given input\n",
        "def test_perceptron(input1, input2):\n",
        "    # Forward pass: calculate weighted sum of inputs and bias\n",
        "    outputP = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "\n",
        "    # Activation function: Heaviside step function\n",
        "    if outputP > 0:\n",
        "        outputP = 1\n",
        "    else:\n",
        "        outputP = 0\n",
        "\n",
        "    return outputP\n",
        "\n",
        "# Test the model with various inputs\n",
        "test_cases = [\n",
        "    (0, 0, 0),\n",
        "    (0, 1, 0),\n",
        "    (1, 0, 0),\n",
        "    (1, 1, 1)\n",
        "]\n",
        "\n",
        "for test_input1, test_input2, expected_output in test_cases:\n",
        "    predicted_output = test_perceptron(test_input1, test_input2)\n",
        "    print(f\"Test input: [{test_input1}, {test_input2}]\")\n",
        "    print(f\"Expected output: {expected_output}\")\n",
        "    print(f\"Predicted output: {predicted_output}\")\n",
        "    if predicted_output == expected_output:\n",
        "        print(\"The model correctly predicts the output.\\n\")\n",
        "    else:\n",
        "        print(\"The model does not correctly predict the output.\\n\")"
      ],
      "metadata": {
        "id": "TAqbH4OQUh_X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Use Sigmoid\n",
        "\n",
        "Using the sigmoid function as the activation function requires some adjustments to both the training and testing phases. The sigmoid function outputs values between 0 and 1, so we will need to use a threshold to decide whether the perceptron's output is closer to 0 or"
      ],
      "metadata": {
        "id": "02WghtgCvIy0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use sigmoid function\n",
        "\n",
        "import random\n",
        "import numpy as np\n",
        "\n",
        "# Define the sigmoid function\n",
        "def sigmoid(x):\n",
        "    return 1 / (1 + np.exp(-x))\n",
        "\n",
        "# Define the derivative of the sigmoid function\n",
        "def sigmoid_derivative(x):\n",
        "    return x * (1 - x)\n",
        "\n",
        "# Define the inputs and outputs for the AND problem\n",
        "inputs = np.array([[0, 0],\n",
        "                   [0, 1],\n",
        "                   [1, 0],\n",
        "                   [1, 1]])\n",
        "\n",
        "outputs = np.array([0, 0, 0, 1])\n",
        "\n",
        "# Initialize parameters\n",
        "lr = 0.1  # Learning rate\n",
        "bias = 1\n",
        "epochs = 10000\n",
        "\n",
        "# Initialize weights randomly\n",
        "weights = [random.random(), random.random(), random.random()]\n",
        "\n",
        "# Training loop\n",
        "for epoch in range(epochs):\n",
        "    total_error = 0\n",
        "    for i in range(len(inputs)):\n",
        "        input1 = inputs[i][0]\n",
        "        input2 = inputs[i][1]\n",
        "        output = outputs[i]\n",
        "\n",
        "        # Forward pass: calculate weighted sum of inputs and bias\n",
        "        weighted_sum = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "\n",
        "        # Apply sigmoid activation function\n",
        "        outputP = sigmoid(weighted_sum)\n",
        "\n",
        "        # Calculate error\n",
        "        error = output - outputP\n",
        "        total_error += abs(error)\n",
        "\n",
        "        # Calculate delta (error * sigmoid_derivative)\n",
        "        delta = error * sigmoid_derivative(outputP)\n",
        "\n",
        "        # Update weights\n",
        "        weights[0] += delta * input1 * lr\n",
        "        weights[1] += delta * input2 * lr\n",
        "        weights[2] += delta * bias * lr\n",
        "\n",
        "    # Print error at each epoch\n",
        "    if epoch % 1000 == 0:\n",
        "        print(f\"Epoch {epoch+1} \\t Error: {total_error}\")\n",
        "\n",
        "    # Check for convergence\n",
        "    if total_error < 1e-5:  # Convergence criteria\n",
        "        print(\"Converged!\")\n",
        "        break\n",
        "\n",
        "print(\"Final weights:\", weights)\n",
        "\n",
        "# Define a function to test the model with a given input\n",
        "def test_perceptron(input1, input2):\n",
        "    # Forward pass: calculate weighted sum of inputs and bias\n",
        "    weighted_sum = input1 * weights[0] + input2 * weights[1] + bias * weights[2]\n",
        "\n",
        "    # Apply sigmoid activation function\n",
        "    outputP = sigmoid(weighted_sum)\n",
        "\n",
        "    # Use a threshold to determine the final output (0 or 1)\n",
        "    if outputP > 0.5:\n",
        "        return 1\n",
        "    else:\n",
        "        return 0\n",
        "\n",
        "# Test the model with various inputs\n",
        "test_cases = [\n",
        "    (0, 0, 0),\n",
        "    (0, 1, 0),\n",
        "    (1, 0, 0),\n",
        "    (1, 1, 1)\n",
        "]\n",
        "\n",
        "for test_input1, test_input2, expected_output in test_cases:\n",
        "    predicted_output = test_perceptron(test_input1, test_input2)\n",
        "    print(f\"Test input: [{test_input1}, {test_input2}]\")\n",
        "    print(f\"Expected output: {expected_output}\")\n",
        "    print(f\"Predicted output: {predicted_output}\")\n",
        "    if predicted_output == expected_output:\n",
        "        print(\"The model correctly predicts the output.\\n\")\n",
        "    else:\n",
        "        print(\"The model does not correctly predict the output.\\n\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qdvfZAJcuzTo",
        "outputId": "df54bf78-7081-4d3a-c876-14b6aa65c78b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1 \t Error: 2.306970615438367\n",
            "Epoch 1001 \t Error: 0.6326231109893838\n",
            "Epoch 2001 \t Error: 0.4366496501047721\n",
            "Epoch 3001 \t Error: 0.34809436954910833\n",
            "Epoch 4001 \t Error: 0.2960515704203268\n",
            "Epoch 5001 \t Error: 0.2611466850237913\n",
            "Epoch 6001 \t Error: 0.23578996350539777\n",
            "Epoch 7001 \t Error: 0.2163576705900786\n",
            "Epoch 8001 \t Error: 0.2008843424656244\n",
            "Epoch 9001 \t Error: 0.18820404903893906\n",
            "Final weights: [5.481077066612058, 5.480712989220059, -8.313356824464995]\n",
            "Test input: [0, 0]\n",
            "Expected output: 0\n",
            "Predicted output: 0\n",
            "The model correctly predicts the output.\n",
            "\n",
            "Test input: [0, 1]\n",
            "Expected output: 0\n",
            "Predicted output: 0\n",
            "The model correctly predicts the output.\n",
            "\n",
            "Test input: [1, 0]\n",
            "Expected output: 0\n",
            "Predicted output: 0\n",
            "The model correctly predicts the output.\n",
            "\n",
            "Test input: [1, 1]\n",
            "Expected output: 1\n",
            "Predicted output: 1\n",
            "The model correctly predicts the output.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "# Homework\n",
        "\n",
        "Write a google document.  Answer these questions.  Look in chatgpt, perplexity, but try understand.  For example look for a tutorial (easy)\n",
        "\n",
        "\n",
        "1.  what is a neural network\n",
        "2.  what is a neuron in a neural network\n",
        "3.  what is a perceptron\n",
        "4. (difficult) what is an activation function\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "8hJumN91YYrn"
      }
    }
  ]
}