neuropathy_preprocessing.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/PandaPowell/ECG_DDA/blob/master/neuropathy_preprocessing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0yiS6O6yISgW"
      },
      "source": [
        "# Processing ECG data in Python\n",
        "\n",
        "\n",
        "1.   Setup\n",
        "  *   Install and import packages\n",
        "  *   Mount Google Drive and sync to Git repository\n",
        "  *   Copy the filtered ECG files from previous R workflow to Google Drive\n",
        "\n",
        "\n",
        "2.   Processing ECG data\n",
        "  *  Remove year string from ECG header files\n",
        "  *  Read signal channels and make sure ECG leads are in channels 1 & 2.\n",
        "  *  Split ECG data into 10 second snippets, and export plots as images\n",
        "\n",
        "3.   Next up:\n",
        "   * Script to train neuropathy classifier\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "huDCGyGpYR-i"
      },
      "source": [
        "## 1. Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "paLv8DbXKZ8z"
      },
      "source": [
        "### Install and import packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lm5wc13ZTKPj",
        "outputId": "f7ed2614-caa6-43fd-99ca-17e6ef55645f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: wfdb in /usr/local/lib/python3.7/dist-packages (3.4.1)\n",
            "Requirement already satisfied: matplotlib>=3.3.4 in /usr/local/lib/python3.7/dist-packages (from wfdb) (3.5.2)\n",
            "Requirement already satisfied: numpy>=1.10.1 in /usr/local/lib/python3.7/dist-packages (from wfdb) (1.21.6)\n",
            "Requirement already satisfied: requests>=2.8.1 in /usr/local/lib/python3.7/dist-packages (from wfdb) (2.23.0)\n",
            "Requirement already satisfied: pandas>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from wfdb) (1.3.5)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from wfdb) (1.4.1)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (7.1.2)\n",
            "Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (3.0.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (2.8.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (21.3)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (4.33.3)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (1.4.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.3.4->wfdb) (0.11.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib>=3.3.4->wfdb) (4.2.0)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.17.0->wfdb) (2022.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7->matplotlib>=3.3.4->wfdb) (1.15.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.8.1->wfdb) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.8.1->wfdb) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.8.1->wfdb) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.8.1->wfdb) (2021.10.8)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (3.5.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (21.3)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.21.6)\n",
            "Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (3.0.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (4.33.3)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (7.1.2)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.4.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib) (4.2.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7->matplotlib) (1.15.0)\n"
          ]
        }
      ],
      "source": [
        "# Installs\n",
        "!pip install wfdb\n",
        "!pip install -U matplotlib\n",
        "\n",
        "# Imports\n",
        "from IPython.display import display\n",
        "from IPython.display import clear_output\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import os\n",
        "import shutil\n",
        "import posixpath\n",
        "import wfdb\n",
        "import pandas as pd\n",
        "from scipy import signal\n",
        "import re\n",
        "from google.colab import drive\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "likvwR_8KeFT"
      },
      "source": [
        "### Mount Google Drive and sync to Git repository"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rnRS891LX1NQ",
        "outputId": "33798faf-c8fd-4c2b-e2f2-ea11f9655664"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/ecg_data\n",
            "/content/ecg_data/MyDrive/ecg_ai\n"
          ]
        }
      ],
      "source": [
        "## Mount google drive\n",
        "drive.mount('/content/ecg_data')\n",
        "\n",
        "# Clone git repo (uncomment if not already cloned)\n",
        "%cd '/content/ecg_data/MyDrive/ecg_ai/'\n",
        "#! git clone https://github.com/PandaPowell/ECG_DDA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zoO94lwk6p3A",
        "outputId": "3173bb38-1542-4406-81b9-e80b91edbced"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content/ecg_data/MyDrive/ecg_ai/ECG_DDA\n",
            "Already up to date.\n"
          ]
        }
      ],
      "source": [
        "# Pull updates:\n",
        "%cd '/content/ecg_data/MyDrive/ecg_ai/ECG_DDA'\n",
        "! git pull"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8qomEb4KtuK"
      },
      "source": [
        "### Copy the filtered ECG files from previous R workflow to Google Drive"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "67g6yyBmQutF"
      },
      "source": [
        "  * If you've previously run or knitted the `readme.Rmd` script from the *R* session, you can copy the contents of your local `/ecg_data/` folder to the corresponding folder in the repo cloned to your mounted google drive (in this case `/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_data/`).\n",
        "  * If you haven't run the *R* preprocessing script, you can grab the preprocessed ECG data from the [`/ecg_data/` folder here](https://drive.google.com/drive/folders/17AX33KuzP2nZwCQ9sWErY_0bSGI-ON1R?usp=sharing)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B4v0zpK9M9OL"
      },
      "source": [
        "## 2. Processing ECG data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ge7havT0NCMD"
      },
      "source": [
        "### Remove year string from ECG header files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CkFWkCRwNJyT"
      },
      "source": [
        "Can't read data otherwise (feature or bug?)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "yRanatQY3q2o"
      },
      "outputs": [],
      "source": [
        "# Function (Credit to Anders Askeland)\n",
        "def remove_year_from_hea(filename: str) -> None:\n",
        "    '''\n",
        "    Removes year from header file in wfdb (.hea) on line 1. Run before reading file with wfdb functions. Does not overwrite original date, wherein the function will rename the original file to: '***_original.hea'.\n",
        "\n",
        "    PS: Very rudimentary function: will always remove 4 digit strings (regex based) if they appear to the left of newline char (\\n).\n",
        "\n",
        "    Args:\n",
        "        filepath to .hea file (without .hea extension)\n",
        "    \n",
        "    Attributes:\n",
        "        None\n",
        "    '''\n",
        "    \n",
        "    # Rewrite hea file\n",
        "    with open(filename + \".hea\", \"r\", encoding='utf-8') as f:\n",
        "        \n",
        "        # Read lines\n",
        "        text = f.readlines()\n",
        "\n",
        "        # Modify line 1 (with regex)\n",
        "        pattern = '\\s\\d{4}(?=(\\\\n))'\n",
        "        if not re.search(pattern, text[0]):\n",
        "            print(\"Year is already removed. Doing nothing.\")\n",
        "            return(1)\n",
        "        else:\n",
        "            print(\"Removing year from line 1\")\n",
        "            modified_line = re.sub(pattern, '', text[0])\n",
        "            text[0] = modified_line\n",
        "    \n",
        "    # Rename original file (don't need)\n",
        "    # os.rename(filename + '.hea', filename + '_original.hea')\n",
        "    \n",
        "    # Create new file\n",
        "    with open(filename + \".hea\", \"w\", encoding='utf-8') as f:\n",
        "        f.writelines(text)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7ybm-CZ557qu"
      },
      "outputs": [],
      "source": [
        "# Apply function to all .hea files:\n",
        "ecg_root = '/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb/'\n",
        "\n",
        "hea_list = [val for sublist in [[os.path.join(i[0], j) for j in i[2] if j.endswith('.hea')] for i in os.walk(ecg_root)] for val in sublist]\n",
        "\n",
        "\n",
        "hea_list_2 = list()\n",
        "\n",
        "# Remove filename endings and feed to function:\n",
        "for filename in hea_list:\n",
        "    name = filename.split('.')[0]\n",
        "    hea_list_2.append(name)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q56DD8Fw59jO",
        "outputId": "b42ae27f-29e8-4cd1-eb41-ffe71157607d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Year is already removed. Doing nothing.\n",
            "Removing year from line 1\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Removing year from line 1\n",
            "Year is already removed. Doing nothing.\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n",
            "Year is already removed. Doing nothing.\n"
          ]
        }
      ],
      "source": [
        "for files in hea_list_2:\n",
        "  remove_year_from_hea(files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DXHEnBtJOgQL"
      },
      "source": [
        "### Read signal channels\n",
        "Make sure ECG leads are in channels 1 & 2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vQlu4tebbaE"
      },
      "source": [
        "Everything is in the right place at the start of monitoring. While that's a good start, wired connections will be prone to the patient unplugging and replugging them to wrong channels in a real-world setting. We'll ignore this issue, as we see no feasible way to address it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "86tPZf1vVgA0"
      },
      "outputs": [],
      "source": [
        "ecg_root = '/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb/'\n",
        "\n",
        "\n",
        "lst = [val for sublist in [[os.path.join(i[0], j) for j in i[2] if j.endswith(('.hea','.dat'))] for i in os.walk(ecg_root)] for val in sublist]\n",
        "\n",
        "lst2 = []\n",
        "\n",
        "for filename in lst:\n",
        "    name = filename.split('.')[0]\n",
        "    lst2.append(name)\n",
        "\n",
        "# Get unique list, sorted for keep indices reproducible in case of crashes\n",
        "lst3 = sorted(list(set(lst2))) \n",
        "\n",
        "# Remove image files (in case some splitting to images has been already done)\n",
        "lst3 = [x for x in lst3 if x.endswith('ECG')]\n",
        "\n",
        "# Manually check quality of ECGS\n",
        "# Split into chunks of 10 to avoid running out of memory:\n",
        "# for filename in lst3[0:9]:\n",
        "#     print(filename)\n",
        "#     # load a record using the 'rdrecord' function\n",
        "#     record = wfdb.rdrecord(filename)    \n",
        "#     sig1 = pd.DataFrame(record.p_signal)\n",
        "#     sig1.columns = ['ecg_0' , 'ecg_1', 'sensor_0' , 'sensor_1', 'emg_0', 'emg_1', 'accelerometer_0', 'accelerometer_1']\n",
        "#     figure, axis = plt.subplots(2, 2, figsize=(25, 8))\n",
        "#     L = record.fs*10\n",
        "    \n",
        "#     axis[0, 0].plot(sig1.iloc[1:L,0])\n",
        "#     axis[0, 0].set_title(record.record_name)\n",
        "#     axis[0, 1].plot(sig1.iloc[1:L,1])\n",
        "#     axis[0, 1].set_title(record.record_name)\n",
        "#     axis[1, 0].plot(sig1.iloc[2:L,2])\n",
        "#     axis[1, 0].set_title(record.record_name)\n",
        "#     axis[1, 1].plot(sig1.iloc[3:L,3]) \n",
        "#     axis[1, 1].set_title(record.record_name)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HSGiA_imcNpq"
      },
      "source": [
        "### Split ECG data into 10 second snippets, and export plots as images\n",
        "While we lose some precision in the data with this approach (depending on image resolution) and lose information contained in the header, this loss is beneficial as it prevents against over-fitting, and the aim is to be able to classify raw ECGs without any additional data.\n",
        "\n",
        "If Colab throws an error, you may need to restart the run-time and re-run the `Install and import packages` & `Read signal channels` chunks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ee0Y_H6RGvI8"
      },
      "source": [
        "#### Sample balance:\n",
        "Sample length varies a lot: 47 of the 60 individuals produce less than 200 10-second snippets, but 7 individuals produce more than 5000.\n",
        "\n",
        "To balance the dataset, each individual only contributes the first 2000 seconds (33 minutes, 200 snippets). Hopefully the wires remain in their right places for most of this interval."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TmA2JCbE_EaZ"
      },
      "outputs": [],
      "source": [
        "# Plot 10 second ECGs for all individuals, to check sample lengths\n",
        "# Length varies a lot: 47 of the 60 individuals produce less than 200 10-second snippets,\n",
        "# but 7 individuals produce more than 5000:\n",
        "\n",
        "for index, filename in enumerate(lst3):\n",
        "    \n",
        "    print('filename: ', filename)\n",
        "    print('index: ', index)\n",
        "    \n",
        "    # load a record using the 'rdrecord' function\n",
        "    record = wfdb.rdrecord(filename)\n",
        "    \n",
        "    sig1 = pd.DataFrame(record.p_signal)\n",
        "    \n",
        "    # record time\n",
        "    print('10-sec snippets: ', sig1.shape[0] / record.fs/ 10, '\\n')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4OnrRlwfv8ad"
      },
      "source": [
        "#### Export signal 1: lead V1/V2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fzBYFd2Aaa_A"
      },
      "outputs": [],
      "source": [
        "# Plot first 200 10 second ECGs for all individuals, both ECG signals\n",
        "\n",
        "### Signal 1 ###\n",
        "for index, filename in enumerate(lst3):\n",
        "    \n",
        "    \n",
        "    clear_output(wait=True)\n",
        "    print('index: ', index)\n",
        "    print(filename)\n",
        "    \n",
        "    # load a record using the 'rdrecord' function\n",
        "    record = wfdb.rdrecord(filename)\n",
        "    \n",
        "    sig1 = pd.DataFrame(record.p_signal)\n",
        "    \n",
        "    # record time\n",
        "    print('How many 10 second measurement intervals:', sig1.shape[0] / record.fs/ 10)\n",
        "    \n",
        "    # Define 10 sec interval via record freq times 10\n",
        "    ten = record.fs*10\n",
        "    # Create sequence of intervals \n",
        "    ten_int = np.arange(0, sig1.shape[0], ten)\n",
        "    \n",
        "    # Limit to first 200\n",
        "    if (len(ten_int) > 200):\n",
        "        len_sig = 200\n",
        "    else:\n",
        "        len_sig = len(ten_int)\n",
        "        \n",
        "    print(len_sig)\n",
        "        \n",
        "    for x in range(1,len_sig):\n",
        "        print(x)\n",
        "        plt.figure(figsize=(15, 2.5), dpi=100)\n",
        "        plt.plot(sig1.iloc[ten_int[x-1]:ten_int[x],0])\n",
        "        plt.ylim(-350, 750)\n",
        "        plt.ylabel('V1/V2  ', rotation=0)\n",
        "        plt.savefig(filename.replace('ecg_wfdb', 'ecg_img') + '_signal1_' + str(x) + '.png', bbox_inches='tight', dpi=100)\n",
        "        plt.cla()\n",
        "        plt.close()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-7XKlTHhwMhF"
      },
      "source": [
        "#### Export signal 2: lead V5/V6"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "5f9g4oa_2Ie1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "493ae184-88a8-4959-d2b4-7f6ecb46b225"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "index:  59\n",
            "/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb/valid/neuropathy/S0610ECG\n",
            "How many 10 second measurement intervals: 38.9202\n",
            "39\n",
            "1\n",
            "2\n",
            "3\n",
            "4\n",
            "5\n",
            "6\n",
            "7\n",
            "8\n",
            "9\n",
            "10\n",
            "11\n",
            "12\n",
            "13\n",
            "14\n",
            "15\n",
            "16\n",
            "17\n",
            "18\n",
            "19\n",
            "20\n",
            "21\n",
            "22\n",
            "23\n",
            "24\n",
            "25\n",
            "26\n",
            "27\n",
            "28\n",
            "29\n",
            "30\n",
            "31\n",
            "32\n",
            "33\n",
            "34\n",
            "35\n",
            "36\n",
            "37\n",
            "38\n"
          ]
        }
      ],
      "source": [
        "### Signal 2 ###\n",
        "for index, filename in enumerate(lst3):\n",
        "    \n",
        "    clear_output(wait=True)\n",
        "    print('index: ', index)\n",
        "    print(filename)\n",
        "    \n",
        "    # load a record using the 'rdrecord' function\n",
        "    record = wfdb.rdrecord(filename)\n",
        "    \n",
        "    sig1 = pd.DataFrame(record.p_signal)\n",
        "    \n",
        "    # record time\n",
        "    print('How many 10 second measurement intervals:', sig1.shape[0] / record.fs/ 10)\n",
        "    \n",
        "    # Define 10 sec interval via record freq times 10\n",
        "    ten = record.fs*10\n",
        "    # Create sequence of intervals \n",
        "    ten_int = np.arange(0, sig1.shape[0], ten)\n",
        "    \n",
        "    # Limit to first 200\n",
        "    if (len(ten_int) > 200):\n",
        "        len_sig = 200\n",
        "    else:\n",
        "        len_sig = len(ten_int)\n",
        "        \n",
        "    print(len_sig)\n",
        "        \n",
        "    for x in range(1,len_sig):\n",
        "        print(x)\n",
        "        plt.figure(figsize=(15, 2.5), dpi=100)\n",
        "        plt.plot(sig1.iloc[ten_int[x-1]:ten_int[x],1])\n",
        "        plt.ylim(-350, 750)\n",
        "        plt.ylabel('V5/V6  ', rotation=0)\n",
        "        plt.savefig(filename.replace('ecg_wfdb', 'ecg_img') + '_signal2_' + str(x) + '.png', bbox_inches='tight', dpi=100)\n",
        "        plt.cla()\n",
        "        plt.close()\n",
        "        "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N0XJY5aM01hH"
      },
      "source": [
        "## 3. Repeat for randomsplit samples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "Pp7UnU457HU3"
      },
      "outputs": [],
      "source": [
        "# Apply function to all .hea files:\n",
        "ecg_root = '/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb_randomsplit/'\n",
        "\n",
        "rand_hea_list = [val for sublist in [[os.path.join(i[0], j) for j in i[2] if j.endswith('.hea')] for i in os.walk(ecg_root)] for val in sublist]\n",
        "\n",
        "\n",
        "rand_hea_list_2 = list()\n",
        "\n",
        "# Remove filename endings and feed to function:\n",
        "for filename in rand_hea_list:\n",
        "    name = filename.split('.')[0]\n",
        "    rand_hea_list_2.append(name)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3a95aa2d-6020-419d-8029-9ba0c900f973",
        "id": "6Vuo6Uzj7S92"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n",
            "Removing year from line 1\n"
          ]
        }
      ],
      "source": [
        "for files in rand_hea_list_2:\n",
        "  remove_year_from_hea(files)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "Dlbiuv2i7oyk"
      },
      "outputs": [],
      "source": [
        "ecg_root = '/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb_randomsplit/'\n",
        "\n",
        "\n",
        "rand_lst = [val for sublist in [[os.path.join(i[0], j) for j in i[2] if j.endswith(('.hea','.dat'))] for i in os.walk(ecg_root)] for val in sublist]\n",
        "\n",
        "rand_lst2 = []\n",
        "\n",
        "for filename in rand_lst:\n",
        "    name = filename.split('.')[0]\n",
        "    rand_lst2.append(name)\n",
        "\n",
        "# Get unique list, sorted for keep indices reproducible in case of crashes\n",
        "rand_lst3 = sorted(list(set(rand_lst2))) \n",
        "\n",
        "# Remove image files (in case some splitting to images has been already done)\n",
        "rand_lst3 = [x for x in rand_lst3 if x.endswith('ECG')]\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wTFgV-y073x9",
        "outputId": "5603669c-bf8a-4e28-c9c6-207371dcc45f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "index:  42\n",
            "/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb_randomsplit/valid/neuropathy/S0381ECG\n",
            "How many 10 second measurement intervals: 139.47840000000002\n",
            "140\n",
            "1\n",
            "2\n",
            "3\n",
            "4\n",
            "5\n",
            "6\n",
            "7\n",
            "8\n",
            "9\n",
            "10\n",
            "11\n",
            "12\n",
            "13\n",
            "14\n",
            "15\n",
            "16\n",
            "17\n",
            "18\n",
            "19\n",
            "20\n",
            "21\n",
            "22\n",
            "23\n",
            "24\n",
            "25\n",
            "26\n",
            "27\n",
            "28\n",
            "29\n",
            "30\n",
            "31\n",
            "32\n",
            "33\n",
            "34\n",
            "35\n",
            "36\n",
            "37\n",
            "38\n",
            "39\n",
            "40\n",
            "41\n",
            "42\n",
            "43\n",
            "44\n",
            "45\n",
            "46\n",
            "47\n",
            "48\n",
            "49\n",
            "50\n",
            "51\n",
            "52\n",
            "53\n",
            "54\n",
            "55\n",
            "56\n",
            "57\n",
            "58\n",
            "59\n",
            "60\n",
            "61\n",
            "62\n",
            "63\n",
            "64\n",
            "65\n",
            "66\n",
            "67\n",
            "68\n",
            "69\n",
            "70\n",
            "71\n",
            "72\n",
            "73\n",
            "74\n",
            "75\n",
            "76\n",
            "77\n",
            "78\n",
            "79\n",
            "80\n",
            "81\n",
            "82\n",
            "83\n",
            "84\n",
            "85\n",
            "86\n",
            "87\n",
            "88\n",
            "89\n",
            "90\n",
            "91\n",
            "92\n",
            "93\n",
            "94\n",
            "95\n",
            "96\n",
            "97\n",
            "98\n",
            "99\n",
            "100\n",
            "101\n",
            "102\n",
            "103\n",
            "104\n",
            "105\n",
            "106\n",
            "107\n",
            "108\n",
            "109\n",
            "110\n",
            "111\n",
            "112\n",
            "113\n",
            "114\n",
            "115\n",
            "116\n",
            "117\n",
            "118\n",
            "119\n",
            "120\n",
            "121\n",
            "122\n",
            "123\n",
            "124\n",
            "125\n",
            "126\n",
            "127\n",
            "128\n",
            "129\n",
            "130\n",
            "131\n",
            "132\n",
            "133\n",
            "134\n",
            "135\n",
            "136\n",
            "137\n",
            "138\n",
            "139\n"
          ]
        }
      ],
      "source": [
        "# Plot first 200 10 second ECGs for all individuals, both ECG signals\n",
        "\n",
        "### Signal 1 ###\n",
        "for index, filename in enumerate(rand_lst3[17:]):\n",
        "    \n",
        "    \n",
        "    clear_output(wait=True)\n",
        "    print('index: ', index)\n",
        "    print(filename)\n",
        "    \n",
        "    # load a record using the 'rdrecord' function\n",
        "    record = wfdb.rdrecord(filename)\n",
        "    \n",
        "    sig1 = pd.DataFrame(record.p_signal)\n",
        "    \n",
        "    # record time\n",
        "    print('How many 10 second measurement intervals:', sig1.shape[0] / record.fs/ 10)\n",
        "    \n",
        "    # Define 10 sec interval via record freq times 10\n",
        "    ten = record.fs*10\n",
        "    # Create sequence of intervals \n",
        "    ten_int = np.arange(0, sig1.shape[0], ten)\n",
        "    \n",
        "    # Limit to first 200\n",
        "    if (len(ten_int) > 200):\n",
        "        len_sig = 200\n",
        "    else:\n",
        "        len_sig = len(ten_int)\n",
        "        \n",
        "    print(len_sig)\n",
        "        \n",
        "    for x in range(1,len_sig):\n",
        "        print(x)\n",
        "        plt.figure(figsize=(15, 2.5), dpi=100)\n",
        "        plt.plot(sig1.iloc[ten_int[x-1]:ten_int[x],0])\n",
        "        plt.ylim(-350, 750)\n",
        "        plt.ylabel('V1/V2  ', rotation=0)\n",
        "        plt.savefig(filename.replace('ecg_wfdb', 'ecg_img') + '_signal1_' + str(x) + '.png', bbox_inches='tight', dpi=100)\n",
        "        plt.cla()\n",
        "        plt.close()\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9e502b4-2794-408c-8590-ee2cab057024",
        "id": "fP_ddw3k8w3m"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "index:  59\n",
            "/content/ecg_data/MyDrive/ecg_ai/ECG_DDA/ecg_wfdb_randomsplit/valid/neuropathy/S0381ECG\n",
            "How many 10 second measurement intervals: 139.47840000000002\n",
            "140\n",
            "1\n",
            "2\n",
            "3\n",
            "4\n",
            "5\n",
            "6\n",
            "7\n",
            "8\n",
            "9\n",
            "10\n",
            "11\n",
            "12\n",
            "13\n",
            "14\n",
            "15\n",
            "16\n",
            "17\n",
            "18\n",
            "19\n",
            "20\n",
            "21\n",
            "22\n",
            "23\n",
            "24\n",
            "25\n",
            "26\n",
            "27\n",
            "28\n",
            "29\n",
            "30\n",
            "31\n",
            "32\n",
            "33\n",
            "34\n",
            "35\n",
            "36\n",
            "37\n",
            "38\n",
            "39\n",
            "40\n",
            "41\n",
            "42\n",
            "43\n",
            "44\n",
            "45\n",
            "46\n",
            "47\n",
            "48\n",
            "49\n",
            "50\n",
            "51\n",
            "52\n",
            "53\n",
            "54\n",
            "55\n",
            "56\n",
            "57\n",
            "58\n",
            "59\n",
            "60\n",
            "61\n",
            "62\n",
            "63\n",
            "64\n",
            "65\n",
            "66\n",
            "67\n",
            "68\n",
            "69\n",
            "70\n",
            "71\n",
            "72\n",
            "73\n",
            "74\n",
            "75\n",
            "76\n",
            "77\n",
            "78\n",
            "79\n",
            "80\n",
            "81\n",
            "82\n",
            "83\n",
            "84\n",
            "85\n",
            "86\n",
            "87\n",
            "88\n",
            "89\n",
            "90\n",
            "91\n",
            "92\n",
            "93\n",
            "94\n",
            "95\n",
            "96\n",
            "97\n",
            "98\n",
            "99\n",
            "100\n",
            "101\n",
            "102\n",
            "103\n",
            "104\n",
            "105\n",
            "106\n",
            "107\n",
            "108\n",
            "109\n",
            "110\n",
            "111\n",
            "112\n",
            "113\n",
            "114\n",
            "115\n",
            "116\n",
            "117\n",
            "118\n",
            "119\n",
            "120\n",
            "121\n",
            "122\n",
            "123\n",
            "124\n",
            "125\n",
            "126\n",
            "127\n",
            "128\n",
            "129\n",
            "130\n",
            "131\n",
            "132\n",
            "133\n",
            "134\n",
            "135\n",
            "136\n",
            "137\n",
            "138\n",
            "139\n"
          ]
        }
      ],
      "source": [
        "### Signal 2 ###\n",
        "for index, filename in enumerate(rand_lst3):\n",
        "    \n",
        "    clear_output(wait=True)\n",
        "    print('index: ', index)\n",
        "    print(filename)\n",
        "    \n",
        "    # load a record using the 'rdrecord' function\n",
        "    record = wfdb.rdrecord(filename)\n",
        "    \n",
        "    sig1 = pd.DataFrame(record.p_signal)\n",
        "    \n",
        "    # record time\n",
        "    print('How many 10 second measurement intervals:', sig1.shape[0] / record.fs/ 10)\n",
        "    \n",
        "    # Define 10 sec interval via record freq times 10\n",
        "    ten = record.fs*10\n",
        "    # Create sequence of intervals \n",
        "    ten_int = np.arange(0, sig1.shape[0], ten)\n",
        "    \n",
        "    # Limit to first 200\n",
        "    if (len(ten_int) > 200):\n",
        "        len_sig = 200\n",
        "    else:\n",
        "        len_sig = len(ten_int)\n",
        "        \n",
        "    print(len_sig)\n",
        "        \n",
        "    for x in range(1,len_sig):\n",
        "        print(x)\n",
        "        plt.figure(figsize=(15, 2.5), dpi=100)\n",
        "        plt.plot(sig1.iloc[ten_int[x-1]:ten_int[x],1])\n",
        "        plt.ylim(-350, 750)\n",
        "        plt.ylabel('V5/V6  ', rotation=0)\n",
        "        plt.savefig(filename.replace('ecg_wfdb', 'ecg_img') + '_signal2_' + str(x) + '.png', bbox_inches='tight', dpi=100)\n",
        "        plt.cla()\n",
        "        plt.close()\n",
        "        "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Udevv8IiwPvF"
      },
      "source": [
        "## 4. Next up: Training a model\n",
        "Model training is done in the [next notebook](https://colab.research.google.com/drive/1_b-j3hDzYTYbbqGdyUnoaGiPZyRMbcJE?usp=sharing)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "neuropathy_preprocessing.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPhttGzGRMzRxWuEOrxgtsH",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}