PPG&BP- Method 6- Data preparation (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1: Data preperation\n",
    "Extraction of normal PPG segments around a normal blood pressure measurement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Importing several libraries  \n",
    "These libraries are commonly used for data analysis, visualization, file handling, and signal processing tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A library for data manipulation and analysis. \n",
    "# It provides data structures like DataFrames that allow to work with structured data efficiently.\n",
    "import pandas as pd\n",
    "\n",
    "# A library for numerical computations.\n",
    "import numpy as np\n",
    "from numpy.ma.core import zeros_like\n",
    "\n",
    "# A plotting library that provides a wide range of functions for creating static, animated, and interactive visualizations.\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.patches import Rectangle\n",
    "\n",
    "# A module that helps you find files/directories matching a specified pattern.\n",
    "import glob\n",
    "\n",
    "# A library for reading and writing Excel files in Python.\n",
    "import openpyxl\n",
    "\n",
    "# A module that provides a way to interact with the operating system. It allows to perform operations like accessing files, directories, and environment variables.\n",
    "import os\n",
    "\n",
    "# A module for manipulating dates and times.\n",
    "import datetime\n",
    "from datetime import datetime\n",
    "\n",
    "# A module that provides signal processing functions for filtering, spectral analysis, interpolation, and more.\n",
    "from scipy import signal\n",
    "from scipy.signal import savgol_filter, argrelextrema\n",
    "\n",
    "# A library for heart rate variability (HRV) analysis.\n",
    "import heartpy as hp\n",
    "\n",
    "# A module that provides functions for generating random numbers, selecting random elements, shuffling sequences, and more.\n",
    "import random\n",
    "\n",
    "# A class from the sklearn.model_selection module that is used for creating a cross-validation splitting strategy that takes into account grouping or clustering of data samples.\n",
    "from sklearn.model_selection import GroupKFold\n",
    "\n",
    "# A class from the sklearn.impute module that provides strategies for imputing missing values in a dataset.\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# A class from the collections module that helps you count the frequency of elements in a list or an iterable.\n",
    "from collections import Counter\n",
    "\n",
    "# Install several Python packages using the %pip install command. \n",
    "# %pip install heartpy\n",
    "# %pip install matplotlib notebook\n",
    "# %pip install imbalanced-learn\n",
    "# %pip install --upgrade scikit-learn\n",
    "# %pip install --upgrade imbalanced-learn\n",
    "# %pip install scikit-learn==0.24.2\n",
    "# %pip install imbalanced-learn==0.8.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload meta-data \n",
    "The metadata obtained from Nervio contains weight, height and gender data of all patients according to their ID number. This dataset also contains a baseline time used to indicate the onset of anesthesia for each patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the meta data excel file to a data frame\n",
    "directory = \"C:/Users/shaha/Desktop/Final Project-PPG/Patient_details.xlsx\"\n",
    "patients_meta_data = pd.read_excel(directory, engine='openpyxl')\n",
    "\n",
    "# Change the ID column name for convenience\n",
    "patients_meta_data.rename(columns={'IONM': 'patient_ID'}, inplace=True)\n",
    "\n",
    "# Creating a new column of the base time for the indication of the start of anesthesia. \n",
    "# Based on the value of Anesthesia Induction if it exists, or alternatively the value of the first time when propofol/muscle paralyzer is given.\n",
    "patients_meta_data['baseline_time'] = patients_meta_data['Anesthesia Induction'].copy()\n",
    "patients_meta_data['new_column'] = np.nan\n",
    "for i, booli in enumerate(patients_meta_data['Propofol: Diprivan,Diprofol \\ Scholine'].notna()):\n",
    "    if booli:\n",
    "        val = patients_meta_data['Propofol: Diprivan,Diprofol \\ Scholine'][i]\n",
    "        new_val = pd.to_datetime(val).to_pydatetime()\n",
    "        patients_meta_data.at[i, 'new_column'] = new_val\n",
    "patients_meta_data['baseline_time'].fillna(patients_meta_data['new_column'], inplace=True)\n",
    "\n",
    "# Fix a specific value that contains str instead of datetime\n",
    "date_string = patients_meta_data['baseline_time'][59:60].values[0]\n",
    "datetime_obj = datetime.strptime(date_string, \"AM %I:%M %d/%m/%Y\")\n",
    "patients_meta_data.at[59, 'baseline_time'] = datetime_obj\n",
    "\n",
    "# Drop the unneccesery columns of the data frame\n",
    "columns_to_drop = ['new_column', 'Propofol: Diprivan,Diprofol \\ Scholine', 'Anesthesia Induction']\n",
    "patients_meta_data = patients_meta_data.drop(columns_to_drop, axis=1)\n",
    "\n",
    "# Apply SimpleImputer to the Weight column that not have values for all the patients\n",
    "column_to_impute = 'Weight'\n",
    "impute_df = patients_meta_data[[column_to_impute]]\n",
    "imp = SimpleImputer(missing_values=np.nan, strategy='mean')\n",
    "imputed_column = pd.DataFrame(imp.fit_transform(impute_df), columns=[column_to_impute])\n",
    "patients_meta_data[column_to_impute] = imputed_column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload the BPs data \n",
    "The Excel files of the blood pressure labels received from Nervio include two sheets for each patient. One sheet contains the invasive blood pressure measurements taken from the anesthesia system and the other sheet contains the non-invasive blood pressure measurements taken with a measuring cuff. The invasive blood pressure data is more reliable and therefore when this data is available it will be used except in cases where there is no invasive data and therefore the non-invasive data will be taken for labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def import_data_BP(patient):\n",
    "    \"\"\"\n",
    "    This function imports and prepares the blood pressure (BP) files for a specific patient.\n",
    "    \n",
    "    Parameters:\n",
    "    - patient (str): The patient identifier or pattern to match the BP files.\n",
    "    \n",
    "    Returns:\n",
    "    - df_BP_prep (pandas DataFrame): The prepared BP data for the patient.\n",
    "    - type_BP (str): The type of blood pressure (either 'Invasive' or 'Non Invasive').\n",
    "    \"\"\"\n",
    "    \n",
    "    # Directory where the BP files are located\n",
    "    directory = \"C:/Users/shaha/Desktop/Final Project-PPG/DATA_PPG_till 250922/PPG+BP_data/\"\n",
    "    \n",
    "    # Create a pattern to match the BP file for the specified patient\n",
    "    pattern = '*' + patient \n",
    "    \n",
    "    # Search for the BP file matching the pattern\n",
    "    excel_file = glob.glob(directory + pattern + '.xlsx')\n",
    "    \n",
    "    # Specify the columns to import from the BP file\n",
    "    usecols = ['Unnamed: 0', 'Unnamed: 1']\n",
    "    \n",
    "    # Read the Invasive BP sheet from the excel file using pandas\n",
    "    df_IBP = pd.read_excel(excel_file[0], sheet_name='Invasive BP', usecols=usecols, engine='openpyxl')\n",
    "    \n",
    "    # Assume the type of BP is Invasive\n",
    "    type_BP = 'Invasive'\n",
    "    \n",
    "    if len(df_IBP) > 2 : \n",
    "        # If the length of the Invasive BP DataFrame is greater than 2 (indicating it has data),\n",
    "        # perform data preparation on the Invasive BP data using the Data_Preperation_BP function\n",
    "        df_IBP_prep = Data_Preperation_BP(df_IBP)\n",
    "        \n",
    "        # Get the start of the Invasive BP \n",
    "        start_IBP = df_IBP_prep['DataTime'][2]\n",
    "        \n",
    "        # Return the prepared Invasive BP DataFrame and the type of BP\n",
    "        return df_IBP_prep, type_BP\n",
    "   \n",
    "    else: \n",
    "        # If the Invasive BP DataFrame has no data, read the Non Invasive BP sheet from the excel file\n",
    "        df_NIBP = pd.read_excel(excel_file[0], sheet_name='NIBPS D', usecols=usecols, engine='openpyxl')\n",
    "        \n",
    "        # Perform data preparation on the Non Invasive BP data using the Data_Preperation_BP function\n",
    "        df_NIBP_prep = Data_Preperation_BP(df_NIBP)\n",
    "        \n",
    "        # Get the start of the Non Invasive BP using the start_BP function\n",
    "        start_NIBP = df_NIBP_prep['DataTime'][2]\n",
    "        \n",
    "        # Set the type of BP as Non Invasive\n",
    "        type_BP = 'Non Invasive'\n",
    "        \n",
    "        # Return the prepared Non Invasive BP DataFrame and the type of BP\n",
    "        return df_NIBP_prep, type_BP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Data_Preperation_BP(df_BP):\n",
    "    \"\"\"\n",
    "    This function performs data preparation on the blood pressure (BP) data frame.\n",
    "\n",
    "    Parameters:\n",
    "    - df_BP (pandas DataFrame): The BP data frame to be prepared.\n",
    "\n",
    "    Returns:\n",
    "    - df_BP (pandas DataFrame): The prepared BP data frame.\n",
    "    \"\"\"\n",
    "\n",
    "    # Delete the first 2 rows from the BP data frame\n",
    "    df_BP = df_BP.drop(labels=[0, 1], axis=0)\n",
    "\n",
    "    # Fix the indexes to start from 0\n",
    "    df_BP.reset_index(drop=True, inplace=True)\n",
    "\n",
    "    # Rename the titles of the BP data frame\n",
    "    df_BP.rename(columns={'Unnamed: 0': 'DataTime', 'Unnamed: 1': 'Value'}, inplace=True)\n",
    "\n",
    "    # Separate the column 'Value' into two columns: 'SBP' and 'DBP'\n",
    "    df_BP[['SBP', 'DBP']] = df_BP['Value'].str.split('/', expand=True)\n",
    "\n",
    "    # Separate the column 'DataTime' into different columns for each time part\n",
    "    df_BP['DataTime'] = pd.to_datetime(df_BP['DataTime'])\n",
    "\n",
    "    # Check for any NaN values in 'SBP' or 'DBP' columns\n",
    "    c1 = df_BP['SBP'].isna().sum()\n",
    "    c2 = df_BP['DBP'].isna().sum()\n",
    "    if (c1 > 0) or (c2 > 0):\n",
    "        # This BP data includes NaN values. Treat this by dropping them.\n",
    "        df_BP = df_BP.dropna(subset=['SBP'])\n",
    "        df_BP = df_BP.dropna(subset=['DBP'])\n",
    "\n",
    "    # Create a column for mean arterial pressure (MAP) using the formula: 1/3(SBP) + 2/3(DBP)\n",
    "    df_BP['SBP'] = df_BP['SBP'].astype(int)\n",
    "    df_BP['DBP'] = df_BP['DBP'].astype(int)\n",
    "    df_BP['MAP'] = (1/3 * df_BP['SBP']) + (2/3 * df_BP['DBP'])\n",
    "\n",
    "    # Reset the index of the BP data frame\n",
    "    df_BP.reset_index(drop=True, inplace=True)\n",
    "\n",
    "    # Return the prepared BP data frame\n",
    "    return df_BP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload the PPG data\n",
    "The Excel files of the PPG signals received from Nerbio include between a single file and several continuous files for each patient, so the file name includes the patient's ID number. Each file contains many columns extracted from the PPG sensor. The relevant columns we have passed are the PLETH column which describes the PPG wave signal itself, the time column in the jumps in which the sample was taken (every 10 milliseconds) and the Prefusion index column which indicates the correctness of the measurement, so that a measurement in which the value is above 1 is considered correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def import_data_PPG(patient):\n",
    "    \"\"\"\n",
    "    This function finds all the files associated with a specific patient.\n",
    "\n",
    "    Parameters:\n",
    "    - patient (str): The patient identifier or pattern to match the files.\n",
    "\n",
    "    Returns:\n",
    "    - csv_files (list): A list of file paths matching the patient identifier.\n",
    "    \"\"\"\n",
    "\n",
    "    # Folder path where the files are located\n",
    "    folder_path = \"C:/Users/shaha/Desktop/Final Project-PPG/DATA_PPG_till 250922/\"\n",
    "\n",
    "    # Create a pattern to match the files for the specified patient\n",
    "    pattern = '*' + patient\n",
    "\n",
    "    # Search for the files matching the pattern in the folder\n",
    "    csv_files = glob.glob(folder_path + pattern + '.csv')\n",
    "\n",
    "    # Return the list of file paths\n",
    "    return csv_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FileName_and_dfPPG_and_FileNum(patient):\n",
    "    \"\"\"\n",
    "    This function creates data frames of the patient's files and returns the number of files.\n",
    "\n",
    "    Parameters:\n",
    "    - patient (str): The patient identifier or pattern to match the files.\n",
    "\n",
    "    Returns:\n",
    "    - files (list): A list of pandas DataFrames containing the patient's files.\n",
    "    - files_names (list): A list of file names corresponding to each DataFrame.\n",
    "    \"\"\"\n",
    "\n",
    "    # Folder path where the files are located\n",
    "    folder_path = \"C:/Users/shaha/Desktop/Final Project-PPG/DATA_PPG_till 250922/\"\n",
    "\n",
    "    # Initialize empty lists for the files and file names\n",
    "    files = []\n",
    "    files_names = []\n",
    "\n",
    "    # Use the import_data_PPG function to get the list of file paths\n",
    "    csv_files = import_data_PPG(patient)\n",
    "\n",
    "    # Check if any files were found\n",
    "    if csv_files:\n",
    "        # Iterate over each file path\n",
    "        for i, csv in enumerate(csv_files):\n",
    "            # Read the CSV file and create a DataFrame\n",
    "            df = pd.read_csv(csv_files[i])\n",
    "            files.append(df)\n",
    "\n",
    "            # Get the file name by removing the folder path from the file path\n",
    "            file_name = csv.replace(\"\\\\\", \"/\").replace(folder_path, \"\")\n",
    "            files_names.append(file_name)\n",
    "\n",
    "        # Get the number of files\n",
    "        num_of_files = len(files)\n",
    "        print(f'Patient number: {patient}, has {num_of_files} files of PPG.')\n",
    "\n",
    "    # Return the list of DataFrames and file names\n",
    "    return files, files_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def total_time_PPG(df):\n",
    "    \"\"\"\n",
    "    This function calculates the total time of the PPG file based on the 'TIMESTAMP_MS' column.\n",
    "\n",
    "    Parameters:\n",
    "    - df (pandas DataFrame): The PPG data frame containing the 'TIMESTAMP_MS' column.\n",
    "\n",
    "    Returns:\n",
    "    - hours (float): The total time of the PPG file in hours.\n",
    "    \"\"\"\n",
    "\n",
    "    # Get the last value in the 'TIMESTAMP_MS' column (in milliseconds)\n",
    "    last_value = df.iloc[-1]['TIMESTAMP_MS']\n",
    "\n",
    "    # Convert milliseconds to seconds\n",
    "    seconds = last_value / 1000\n",
    "\n",
    "    # Convert seconds to minutes\n",
    "    minutes = seconds / 60\n",
    "\n",
    "    # Convert minutes to hours\n",
    "    hours = minutes / 60\n",
    "\n",
    "    # Return the total time in hours\n",
    "    return hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A function that concatenates all the processes into one function.\n",
    "\n",
    "def patient_all_data(patient, files, files_names, df_BP, type_BP, len_signal, padding, patients_meta_data, all_parts_num=0, rows_parts_num=0, valid_bp_parts_num=0, parts_num=0):\n",
    "    \"\"\"\n",
    "    This function performs various data processing steps and concatenates them into one function.\n",
    "\n",
    "    Parameters:\n",
    "    - patient (str): The patient identifier or pattern.\n",
    "    - files (list): A list of pandas DataFrames representing the patient's files.\n",
    "    - files_names (list): A list of file names corresponding to each DataFrame.\n",
    "    - df_BP (pandas DataFrame): The blood pressure (BP) data frame.\n",
    "    - type_BP (str): The type of blood pressure ('Invasive' or 'Non Invasive').\n",
    "    - len_signal (float): The length of the PPG signal in seconds.\n",
    "    - padding (float): The padding time in seconds.\n",
    "    - patients_meta_data (pandas DataFrame): Meta data of patients.\n",
    "    - all_parts_num (int, optional): The total number of parts encountered in the process.\n",
    "    - rows_parts_num (int, optional): The number of parts that meet the row condition.\n",
    "    - valid_bp_parts_num (int, optional): The number of parts that meet the blood pressure condition.\n",
    "    - parts_num (int, optional): The number of valid segments.\n",
    "\n",
    "    Returns:\n",
    "    - df_list (list): A list of pandas DataFrames representing the merged data for each file.\n",
    "    - concatenated_df (pandas DataFrame): The concatenated data frame with relevant columns.\n",
    "    - total_time_file (float): The total time of the patient's files.\n",
    "    - all_parts_num (int): The updated total number of parts encountered in the process.\n",
    "    - rows_parts_num (int): The updated number of parts that meet the row condition.\n",
    "    - valid_bp_parts_num (int): The updated number of segments that meet the blood pressure condition.\n",
    "    \"\"\"\n",
    "\n",
    "    # Multiply the signal and padding lengths by 100 to account for 10 ms per row\n",
    "    len_signal *= 100\n",
    "    padding *= 100\n",
    "\n",
    "    # Initialize empty lists and variables\n",
    "    df_list = []\n",
    "    total_time_file = 0\n",
    "    all_non_invasive_BP = 0\n",
    "    all_invasive_BP = 0\n",
    "    # Iterate over each file DataFrame\n",
    "    for i, df in enumerate(files):\n",
    "        # Count the total time of the case\n",
    "        total_time_file += total_time_PPG(df)\n",
    "\n",
    "        # Get the start time of the BP measurement from the meta data\n",
    "        start_time = patients_meta_data.loc[patients_meta_data['patient_ID'] == int(patient), 'baseline_time'].values[0]\n",
    "        df.insert(0, 'BaseLine_Time', start_time)\n",
    "\n",
    "        # Insert patient metadata columns into the DataFrame\n",
    "        df = df.assign(Age=patients_meta_data.loc[patients_meta_data['patient_ID'] == int(patient), 'Age'].values[0],\n",
    "                       Weight=patients_meta_data.loc[patients_meta_data['patient_ID'] == int(patient), 'Weight'].values[0],\n",
    "                       Gender=patients_meta_data.loc[patients_meta_data['patient_ID'] == int(patient), 'Gender'].values[0])\n",
    "\n",
    "        # Create a column 'DataTime' filled with the start time of the file\n",
    "        df.insert(0, 'DataTime', start_time)\n",
    "\n",
    "        # Fix the 'DataTime' column with the correct time calculated by 10 ms jumps\n",
    "        df['DataTime'] = pd.to_timedelta(df['TIMESTAMP_MS']*1e6) + df['DataTime']\n",
    "\n",
    "        # Create dummy columns with NaN values according to the BP columns\n",
    "        df = df.assign(DataTime_BP=np.nan, SBP=np.nan, DBP=np.nan, MAP=np.nan, BP_type=np.nan)\n",
    "\n",
    "\n",
    "        # Find the boundaries of the closest datetimes\n",
    "        index0 = abs(df.at[0, 'DataTime'] - df_BP['DataTime']).argmin()\n",
    "        index1 = abs(df.at[df.shape[0]-1, 'DataTime'] - df_BP['DataTime']).argmin()\n",
    "\n",
    "        for row in range(index0, index1+1):\n",
    "            # Find the index of the PPG DataFrame that has the closest time of BP measurement\n",
    "            BP_ind = abs(df['DataTime'] - df_BP.at[row, 'DataTime']).argmin()\n",
    "            \n",
    "            # Insert the BP rows in the appropriate index of the PPG DataFrame\n",
    "            df.at[BP_ind, 'DataTime_BP'] = df_BP.at[row, 'DataTime']\n",
    "            df.at[BP_ind, 'SBP'] = df_BP.at[row, 'SBP']\n",
    "            df.at[BP_ind, 'DBP'] = df_BP.at[row, 'DBP']\n",
    "            df.at[BP_ind, 'MAP'] = df_BP.at[row, 'MAP']\n",
    "            \n",
    "            # Insert the BP type column with int values instead of str \n",
    "            df.loc[BP_ind, 'BP_type'] = {'Invasive': 1, 'Non Invasive': 0}.get(type_BP, -1)\n",
    "                \n",
    "        # Find and insert the value of the BP at the start time of the PPG file\n",
    "        start_BP_ind = abs(start_time - df_BP['DataTime']).argmin()\n",
    "        baseline_BP = df_BP.at[start_BP_ind, 'MAP']\n",
    "        df.insert(0, 'BaseLine_MAP', baseline_BP)\n",
    "\n",
    "        # Create a merged DataFrame with only the signal+padding rows parts of every BP measurement\n",
    "        n = int(len_signal + (padding*2))\n",
    "        has_BP_df = df[~df['DataTime_BP'].isna()]\n",
    "        merged_df = pd.DataFrame()\n",
    "        file_count_parts = 0\n",
    "\n",
    "        for ind in range(len(has_BP_df)):\n",
    "            index = has_BP_df.iloc[ind:ind+1].index[0]\n",
    "            part_df = pd.DataFrame()\n",
    "            part_df = df[:int(index+(padding)+1)][-n:]\n",
    "            all_parts_num += 1\n",
    "            \n",
    "            # Check if every part (n rows) of the merged DataFrame is valid and meets the row condition\n",
    "            cond_1 = (part_df.iloc[-1]['TIMESTAMP_MS'] - part_df.iloc[0]['TIMESTAMP_MS']) == ((n*10)-10)\n",
    "            if cond_1:\n",
    "                rows_parts_num += 1\n",
    "\n",
    "            # Check if every part (n rows) of the merged DataFrame meets the blood pressure condition\n",
    "            cond_2 = ((has_BP_df['SBP'] > 290) & (has_BP_df['SBP'] > 40)).sum() == 0\n",
    "            if cond_1 & cond_2:\n",
    "                valid_bp_parts_num += 1\n",
    "\n",
    "            # Check if every part (n rows) of the merged DataFrame meets the perfusion index condition\n",
    "            cond_3 = (part_df['PERFUSION_INDEX'] >= 1).sum() == len(part_df)\n",
    "            if cond_1 & cond_2 & cond_3:\n",
    "                parts_num += 1\n",
    "                part_df['Part_Number'] = parts_num\n",
    "                merged_df = pd.concat([merged_df, part_df])\n",
    "                file_count_parts += 1\n",
    "\n",
    "        df_list.append(merged_df)\n",
    "        print(f'The {i+1} file of this patient includes {file_count_parts} valid segments of {n/100} seconds.')\n",
    "    \n",
    "    # Concatenate the rows of the data frames\n",
    "    concatenated_df = pd.concat(df_list)\n",
    "\n",
    "    # Drop all non-relevant columns from the concatenated data frame\n",
    "    if len(concatenated_df) > 0:\n",
    "        concatenated_df = concatenated_df.drop(labels=['COUNTER', 'DEVICE_ID', 'SPO2_STATUS', 'BATTERY_PCT', 'PERFUSION_INDEX'], axis=1)\n",
    "    concatenated_df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "    return df_list, concatenated_df, total_time_file, all_parts_num, rows_parts_num, valid_bp_parts_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a full data frame contains all valid PPG segments around valid blood pressure measurements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_ID_list():\n",
    "    \"\"\"\n",
    "    This function retrieves a list of all the unique IDs present in the 'PPG+BP_data' folder.\n",
    "\n",
    "    Returns:\n",
    "    - ID_list_unique (list): A list of unique IDs extracted from the filenames in the folder.\n",
    "    \"\"\"\n",
    "\n",
    "    # Specify the folder path\n",
    "    folder_path = \"C:/Users/shaha/Desktop/Final Project-PPG/DATA_PPG_till 250922/PPG+BP_data/\"\n",
    "\n",
    "    # Get the list of filenames in the folder (excluding the last one)\n",
    "    files = os.listdir(folder_path)[:-1]\n",
    "\n",
    "    # Initialize an empty list to store unique IDs\n",
    "    ID_list_unique = []\n",
    "\n",
    "    # Iterate over each filename\n",
    "    for file in files:\n",
    "        # Extract the ID from the filename\n",
    "        start = file.find(\"_\") + 1\n",
    "        ids = file[start:-5]  # Remove the file extension (.xlsx)\n",
    "        ID_list_unique.append(ids)\n",
    "\n",
    "    return ID_list_unique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Patient number: 22043146, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 6 valid segments of 7.0 seconds.\n",
      "Patient number: 22054597, has 1 files of PPG.\n",
      "The 1 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "Patient number: 22105955, has 2 files of PPG.\n",
      "The 1 file of this patient includes 5 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "Patient number: 22389334, has 5 files of PPG.\n",
      "The 1 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 4 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 5 file of this patient includes 7 valid segments of 7.0 seconds.\n",
      "Patient number: 23772858, has 1 files of PPG.\n",
      "The 1 file of this patient includes 76 valid segments of 7.0 seconds.\n",
      "Patient number: 24806073, has 3 files of PPG.\n",
      "The 1 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 113 valid segments of 7.0 seconds.\n",
      "Patient number: 24893196, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 111 valid segments of 7.0 seconds.\n",
      "Patient number: 24955278, has 4 files of PPG.\n",
      "The 1 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 36 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 46 valid segments of 7.0 seconds.\n",
      "The 4 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 25674542, has 1 files of PPG.\n",
      "The 1 file of this patient includes 5 valid segments of 7.0 seconds.\n",
      "Patient number: 26174344, has 3 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 62 valid segments of 7.0 seconds.\n",
      "Patient number: 26960697, has 3 files of PPG.\n",
      "The 1 file of this patient includes 8 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 28152193, has 4 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 119 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 30 valid segments of 7.0 seconds.\n",
      "The 4 file of this patient includes 108 valid segments of 7.0 seconds.\n",
      "Patient number: 28434190, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 8 valid segments of 7.0 seconds.\n",
      "Patient number: 28781137, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 167 valid segments of 7.0 seconds.\n",
      "Patient number: 29392644, has 2 files of PPG.\n",
      "The 1 file of this patient includes 1 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 22 valid segments of 7.0 seconds.\n",
      "Patient number: 29735122, has 1 files of PPG.\n",
      "The 1 file of this patient includes 45 valid segments of 7.0 seconds.\n",
      "Patient number: 29988449, has 1 files of PPG.\n",
      "The 1 file of this patient includes 22 valid segments of 7.0 seconds.\n",
      "Patient number: 30595899, has 1 files of PPG.\n",
      "The 1 file of this patient includes 151 valid segments of 7.0 seconds.\n",
      "Patient number: 31199642, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 114 valid segments of 7.0 seconds.\n",
      "Patient number: 31523909, has 1 files of PPG.\n",
      "The 1 file of this patient includes 278 valid segments of 7.0 seconds.\n",
      "Patient number: 31865599, has 1 files of PPG.\n",
      "The 1 file of this patient includes 180 valid segments of 7.0 seconds.\n",
      "Patient number: 32385357, has 1 files of PPG.\n",
      "The 1 file of this patient includes 137 valid segments of 7.0 seconds.\n",
      "Patient number: 32401088, has 1 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 33014247, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 139 valid segments of 7.0 seconds.\n",
      "Patient number: 33421934, has 1 files of PPG.\n",
      "The 1 file of this patient includes 201 valid segments of 7.0 seconds.\n",
      "Patient number: 33880222, has 1 files of PPG.\n",
      "The 1 file of this patient includes 111 valid segments of 7.0 seconds.\n",
      "Patient number: 34569765, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 72 valid segments of 7.0 seconds.\n",
      "Patient number: 34804384, has 1 files of PPG.\n",
      "The 1 file of this patient includes 213 valid segments of 7.0 seconds.\n",
      "Patient number: 36385272, has 1 files of PPG.\n",
      "The 1 file of this patient includes 25 valid segments of 7.0 seconds.\n",
      "Patient number: 36448650, has 1 files of PPG.\n",
      "The 1 file of this patient includes 178 valid segments of 7.0 seconds.\n",
      "Patient number: 36558316, has 1 files of PPG.\n",
      "The 1 file of this patient includes 131 valid segments of 7.0 seconds.\n",
      "Patient number: 36624353, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 57 valid segments of 7.0 seconds.\n",
      "Patient number: 36906055, has 1 files of PPG.\n",
      "The 1 file of this patient includes 201 valid segments of 7.0 seconds.\n",
      "Patient number: 36990572, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 37655648, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 105 valid segments of 7.0 seconds.\n",
      "Patient number: 38433448, has 3 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 33 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 5 valid segments of 7.0 seconds.\n",
      "Patient number: 38460441, has 1 files of PPG.\n",
      "The 1 file of this patient includes 3 valid segments of 7.0 seconds.\n",
      "Patient number: 38803393, has 2 files of PPG.\n",
      "The 1 file of this patient includes 2 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 31 valid segments of 7.0 seconds.\n",
      "Patient number: 39041005, has 1 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 39675848, has 1 files of PPG.\n",
      "The 1 file of this patient includes 61 valid segments of 7.0 seconds.\n",
      "Patient number: 40593492, has 1 files of PPG.\n",
      "The 1 file of this patient includes 285 valid segments of 7.0 seconds.\n",
      "Patient number: 41309936, has 1 files of PPG.\n",
      "The 1 file of this patient includes 97 valid segments of 7.0 seconds.\n",
      "Patient number: 42493587, has 1 files of PPG.\n",
      "The 1 file of this patient includes 132 valid segments of 7.0 seconds.\n",
      "Patient number: 42690527, has 1 files of PPG.\n",
      "The 1 file of this patient includes 247 valid segments of 7.0 seconds.\n",
      "Patient number: 43039692, has 2 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 91 valid segments of 7.0 seconds.\n",
      "Patient number: 43296067, has 1 files of PPG.\n",
      "The 1 file of this patient includes 13 valid segments of 7.0 seconds.\n",
      "Patient number: 44419960, has 1 files of PPG.\n",
      "The 1 file of this patient includes 6 valid segments of 7.0 seconds.\n",
      "Patient number: 44851586, has 1 files of PPG.\n",
      "The 1 file of this patient includes 146 valid segments of 7.0 seconds.\n",
      "Patient number: 45603227, has 1 files of PPG.\n",
      "The 1 file of this patient includes 114 valid segments of 7.0 seconds.\n",
      "Patient number: 45620105, has 1 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 46061524, has 1 files of PPG.\n",
      "The 1 file of this patient includes 155 valid segments of 7.0 seconds.\n",
      "Patient number: 46558731, has 1 files of PPG.\n",
      "The 1 file of this patient includes 35 valid segments of 7.0 seconds.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Patient number: 46582101, has 1 files of PPG.\n",
      "The 1 file of this patient includes 35 valid segments of 7.0 seconds.\n",
      "Patient number: 48049854, has 1 files of PPG.\n",
      "The 1 file of this patient includes 201 valid segments of 7.0 seconds.\n",
      "Patient number: 48456249, has 1 files of PPG.\n",
      "The 1 file of this patient includes 126 valid segments of 7.0 seconds.\n",
      "Patient number: 48555243, has 1 files of PPG.\n",
      "The 1 file of this patient includes 26 valid segments of 7.0 seconds.\n",
      "Patient number: 48995255, has 1 files of PPG.\n",
      "The 1 file of this patient includes 97 valid segments of 7.0 seconds.\n",
      "Patient number: 50966344, has 1 files of PPG.\n",
      "The 1 file of this patient includes 268 valid segments of 7.0 seconds.\n",
      "Patient number: 51156276, has 3 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 55 valid segments of 7.0 seconds.\n",
      "Patient number: 51392584, has 1 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 51404164, has 3 files of PPG.\n",
      "The 1 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 2 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "The 3 file of this patient includes 0 valid segments of 7.0 seconds.\n",
      "Patient number: 51579965, has 1 files of PPG.\n",
      "The 1 file of this patient includes 112 valid segments of 7.0 seconds.\n",
      "Patient number: 51653909, has 1 files of PPG.\n",
      "The 1 file of this patient includes 244 valid segments of 7.0 seconds.\n",
      "Patient number: 52109015, has 1 files of PPG.\n",
      "The 1 file of this patient includes 47 valid segments of 7.0 seconds.\n",
      "Patient number: 52615561, has 1 files of PPG.\n",
      "The 1 file of this patient includes 25 valid segments of 7.0 seconds.\n",
      "Patient number: 52949390, has 1 files of PPG.\n",
      "The 1 file of this patient includes 140 valid segments of 7.0 seconds.\n"
     ]
    }
   ],
   "source": [
    "# Processes data for multiple patients and creates a comprehensive DataFrame (full_df) that combines the processed data for all patients. \n",
    "# It also tracks the total time of each patient's case in the case_times list.\n",
    "\n",
    "# Calling the create_ID_list() function to obtain a list of patient IDs (patient_list).\n",
    "patient_list = create_ID_list()\n",
    "\n",
    "# Create an empty DataFrame (full_df) to store the processed data of all patients.\n",
    "full_df = pd.DataFrame()\n",
    "\n",
    "# process the data for the first patient specified as '22043146'.\n",
    "patient = '22043146'\n",
    "\n",
    "# Calls the FileName_and_dfPPG_and_FileNum() function to retrieve the files and file names for the patient.\n",
    "files, files_names = FileName_and_dfPPG_and_FileNum(patient)\n",
    "\n",
    "# Calls the import_data_BP() function to import the patient's BP data and determine the type of BP.\n",
    "df_BP, type_BP = import_data_BP(patient)\n",
    "\n",
    "# Calls the patient_all_data() function to process the patient's data and retrieve relevant information\n",
    "df_list, df, total_time_file, all_parts_num, rows_parts_num, valid_bp_parts_num = patient_all_data(patient, files, files_names, df_BP, type_BP, 5, 1, patients_meta_data)\n",
    "\n",
    "# Adds the patient ID (patient) as a column in the df DataFrame.\n",
    "df['patient_ID'] = patient\n",
    "\n",
    "# Concatenates the df DataFrame to the full_df DataFrame.\n",
    "full_df = pd.concat([full_df, df])\n",
    "\n",
    "# Appends the total time of the patient's case (total_time_file) to the case_times list.\n",
    "case_times = []\n",
    "case_times.append(total_time_file)\n",
    "\n",
    "# Process the data for the remaining patients in the patient_list using a loop. \n",
    "# It performs similar steps as mentioned above, including retrieving files, importing BP data, processing patient data, adding the patient ID as a column, concatenating the processed data to full_df, and appending the total time of each patient's case to case_times.\n",
    "for patient in range(1, len(patient_list)):\n",
    "    files, files_names = FileName_and_dfPPG_and_FileNum(patient_list[patient])\n",
    "    df_BP, type_BP = import_data_BP(patient_list[patient])\n",
    "    parts_num = full_df['Part_Number'][-1:].values[0]\n",
    "    df_list, concatenated_df, total_time_file, all_parts_num, rows_parts_num, valid_bp_parts_num = patient_all_data(patient_list[patient], files, files_names, df_BP, type_BP, 5, 1, patients_meta_data, all_parts_num, rows_parts_num, valid_bp_parts_num, parts_num)\n",
    "    concatenated_df['patient_ID'] = patient_list[patient]\n",
    "    full_df = pd.concat([full_df, concatenated_df])\n",
    "\n",
    "    case_times.append(total_time_file)\n",
    "    \n",
    "# Fix the last patient import data\n",
    "patient = '53403719'\n",
    "directory = \"C:/Users/shaha/Desktop/Final Project-PPG/DATA_PPG_till 250922/\"\n",
    "pattern = '*' + patient\n",
    "excel_file = glob.glob(directory + pattern + '.xlsx')\n",
    "\n",
    "# Read the Excel file for the last patient\n",
    "df = pd.read_excel(excel_file[0], sheet_name='20220524T160016.406+0300_534037', engine='openpyxl')\n",
    "df['patient_ID'] = patient\n",
    "\n",
    "# Prepare the files, files_names, df_BP, and type_BP for the last patient\n",
    "files = [df]\n",
    "file_name = excel_file[0].replace(\"\\\\\", \"/\").replace(directory, \"\")\n",
    "files_names = [file_name]\n",
    "df_BP, type_BP = import_data_BP(patient)\n",
    "\n",
    "# Get the parts_num from full_df\n",
    "parts_num = full_df['Part_Number'][-1:].values[0]\n",
    "\n",
    "# Process the data for the last patient\n",
    "df_list, df, total_time_file, all_parts_num, rows_parts_num, valid_bp_parts_num = patient_all_data(patient, files, files_names, df_BP, type_BP, 5, 1, patients_meta_data, all_parts_num, rows_parts_num, valid_bp_parts_num, parts_num)\n",
    "\n",
    "# Append the total_time_file to case_times\n",
    "case_times.append(total_time_file)\n",
    "\n",
    "# Concatenate the processed data for the last patient to full_df\n",
    "full_df = pd.concat([full_df, df])\n",
    "\n",
    "# Reset the indexes\n",
    "full_df.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculates and prints various statistics related to the processed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total segments before any filtering: 8710\n",
      "Rows segments: 8548, which are 98.14%\n",
      "Valid BP segments: 8548, which are 100.0%\n",
      "Valid segments (after 2 filtering- rows & prefusion index): 6303.0, which are 73.74%\n",
      "Valid segments with invasive BP reference: 5817, and non invasive BP reference: 486\n",
      "Maximum case time: 18.80829, minimum case time: 0.00022\n",
      "Median case time: 2.869\n"
     ]
    }
   ],
   "source": [
    "# Display the total number of parts before any filtering.\n",
    "print(f'Total segments before any filtering: {all_parts_num}')\n",
    "\n",
    "# Display the number of segments that meet the rows filtering condition.\n",
    "print(f'Rows segments: {rows_parts_num}, which are {np.round((rows_parts_num/all_parts_num)*100,2)}%')\n",
    "\n",
    "# Display the number of segments that meet both the rows and perfusion index filtering conditions. the number of segments that meet the valid BP filtering condition.\n",
    "print(f'Valid BP segments: {valid_bp_parts_num}, which are {np.round((valid_bp_parts_num/rows_parts_num)*100,2)}%')\n",
    "\n",
    "# Display the number of segments that meet both the rows and perfusion index filtering conditions.\n",
    "parts_num = full_df['Part_Number'][-1:].values[0]\n",
    "print(f'Valid segments (after 2 filtering- rows & prefusion index): {parts_num}, which are {np.round((parts_num/valid_bp_parts_num)*100,2)}%')\n",
    "\n",
    "# Display the number of valid segments that have invasive & non invasive BP reference\n",
    "non_invasive_BP = full_df.loc[full_df['BP_type'] == 0, 'Part_Number'].nunique()\n",
    "invasive_BP = full_df.loc[full_df['BP_type'] == 1, 'Part_Number'].nunique()\n",
    "print(f'Valid segments with invasive BP reference: {invasive_BP}, and non invasive BP reference: {non_invasive_BP}')\n",
    "\n",
    "# Display the range of the total time of the cases.\n",
    "print(f'Maximum case time: {np.round(np.max(case_times),5)}, minimum case time: {np.round(np.min(case_times),5)}')\n",
    "print(f'Median case time: {np.round(np.median(case_times), 3)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Split the data to train & test\n",
    "The code performs a GroupKFold split on the df_split DataFrame to create train and test sets for a supervised learning task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parts: 6303, 5041 train parts and 1262 test parts.\n"
     ]
    }
   ],
   "source": [
    "# Create a copy of the full_df DataFrame\n",
    "df_split = full_df.copy()\n",
    "\n",
    "# Define the parameters to the GroupKFold split\n",
    "X = df_split.drop(['patient_ID', 'MAP', 'BP_type', 'DBP', 'SBP', 'DataTime_BP', 'BaseLine_MAP'], axis=1)\n",
    "Y = df_split[['MAP', 'BP_type', 'DBP', 'SBP', 'DataTime_BP', 'BaseLine_MAP']]\n",
    "groups = df_split['patient_ID']\n",
    "\n",
    "# Initialize the GroupKFold object with n_splits=5\n",
    "gfk = GroupKFold(n_splits=5)\n",
    "\n",
    "# Generate the train and test indices using the GroupKFold split\n",
    "train_idx, test_idx = next(gfk.split(X, Y, groups=groups))\n",
    "\n",
    "# Create the train and test data frames using the indices\n",
    "\n",
    "# Select the rows from df_split based on the train indices\n",
    "train = df_split.iloc[train_idx]\n",
    "\n",
    "# Select the rows from df_split based on the test indices\n",
    "test = df_split.iloc[test_idx]\n",
    "\n",
    "# Reset the indices of train and test data frames\n",
    "train.reset_index(drop=True, inplace=True)\n",
    "test.reset_index(drop=True, inplace=True)\n",
    "\n",
    "# Create copies of the train and test data frames\n",
    "train_df = train.copy()\n",
    "test_df = test.copy()\n",
    "\n",
    "# Count the parts in the train and test data frames\n",
    "parts_train = len(np.unique(train['Part_Number']))\n",
    "parts_test = len(np.unique(test['Part_Number']))\n",
    "\n",
    "# Print the total number of parts, train parts, and test parts\n",
    "print(f'Total number of parts: {parts_train + parts_test}, {parts_train} train parts and {parts_test} test parts.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_parts(df, kind):\n",
    "    \"\"\"\n",
    "    Count the number of parts and the distribution of BP types in the given DataFrame.\n",
    "    \n",
    "    Parameters:\n",
    "    - df (DataFrame): The DataFrame containing the data.\n",
    "    - kind (str): The kind of data or category to count parts for.\n",
    "    \n",
    "    Returns:\n",
    "    None\n",
    "    \n",
    "    Prints the count of total parts, invasive BP parts, non-invasive BP parts, and parts with no BP recorded.\n",
    "    \"\"\"\n",
    "\n",
    "    # Count the total number of unique segments\n",
    "    total_parts = len(np.unique(df['Part_Number']))\n",
    "    \n",
    "    # Count the invasive & non-invasive parts in the DataFrame\n",
    "    \n",
    "    # Count the number of rows where 'BP_type' is 1.0 (indicating invasive BP)\n",
    "    invasive_BP = (df['BP_type'] == 1.0).sum()\n",
    "    \n",
    "    # Count the number of rows where 'BP_type' is 0 (indicating non-invasive BP)\n",
    "    no_invasive_BP = (df['BP_type'] == 0).sum()\n",
    "\n",
    "    # Print the counts of parts for the given kind\n",
    "    print(f'{kind} - total parts: {total_parts}, invasive BP: {invasive_BP}, non-invasive BP: {no_invasive_BP}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}