h_m_final_term_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "464edd62",
   "metadata": {},
   "source": [
    "## Team Memebers:\n",
    "* ### Prajwal Mani(pbm6)\n",
    "* ### Damian Lopez(jl889)\n",
    "* ### Alexander Schmidt(ats46)\n",
    "* ### Such Khare(sk368)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e2ec768",
   "metadata": {},
   "source": [
    "### Note:\n",
    "##### Tried out many methods but choose the best method in to add in this notebook\n",
    "##### Some of the cells are not excuted in my system  the other teammates trained the model on there system and to combine everything into one file we have pasted the code here.\n",
    "##### So if you want to check out the output of the cells please check the github repo:https://github.com/prajwalmani/H-M-Personalized-Fashion-Recommendations\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "009eb0f5",
   "metadata": {},
   "source": [
    "The notebook is very big so i will be adding a abstract here <br>\n",
    "* For data loading i have used timeseriesgenerator which is work well for time series problem<br>\n",
    "* Initial model which was built is LSTM <br>\n",
    "* Then i tried with LSTM eith hypertunning like learning rate,no of neurons,layers and dropout <br>\n",
    "* For mertrics we used Mean squared error and absoulte mean square error just too take outlier has also consideration\n",
    "* We didn't just stick on to just LSTM or GRU we tired out tensorflow recommder and LGBMRanker\n",
    "* Both of the models have had the same mae and mse "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a939d081",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import preprocessing\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense, LSTM, Dropout,TimeDistributed,GRU,SimpleRNN\n",
    "from keras.preprocessing.sequence import TimeseriesGenerator\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "import tensorflow as tf\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d16a3ff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_cols=[]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8aa14e60",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_df=pd.read_csv(r'articles.csv')\n",
    "customers_df=pd.read_csv(r'customers.csv')\n",
    "transcations_df=pd.read_csv(r'transactions_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9c362bf0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>prod_name</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>product_type_name</th>\n",
       "      <th>product_group_name</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>graphical_appearance_name</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>colour_group_name</th>\n",
       "      <th>...</th>\n",
       "      <th>department_name</th>\n",
       "      <th>index_code</th>\n",
       "      <th>index_name</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>index_group_name</th>\n",
       "      <th>section_no</th>\n",
       "      <th>section_name</th>\n",
       "      <th>garment_group_no</th>\n",
       "      <th>garment_group_name</th>\n",
       "      <th>detail_desc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>108775015</td>\n",
       "      <td>108775</td>\n",
       "      <td>Strap top</td>\n",
       "      <td>253</td>\n",
       "      <td>Vest top</td>\n",
       "      <td>Garment Upper body</td>\n",
       "      <td>1010016</td>\n",
       "      <td>Solid</td>\n",
       "      <td>9</td>\n",
       "      <td>Black</td>\n",
       "      <td>...</td>\n",
       "      <td>Jersey Basic</td>\n",
       "      <td>A</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>1</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>16</td>\n",
       "      <td>Womens Everyday Basics</td>\n",
       "      <td>1002</td>\n",
       "      <td>Jersey Basic</td>\n",
       "      <td>Jersey top with narrow shoulder straps.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>108775044</td>\n",
       "      <td>108775</td>\n",
       "      <td>Strap top</td>\n",
       "      <td>253</td>\n",
       "      <td>Vest top</td>\n",
       "      <td>Garment Upper body</td>\n",
       "      <td>1010016</td>\n",
       "      <td>Solid</td>\n",
       "      <td>10</td>\n",
       "      <td>White</td>\n",
       "      <td>...</td>\n",
       "      <td>Jersey Basic</td>\n",
       "      <td>A</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>1</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>16</td>\n",
       "      <td>Womens Everyday Basics</td>\n",
       "      <td>1002</td>\n",
       "      <td>Jersey Basic</td>\n",
       "      <td>Jersey top with narrow shoulder straps.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>108775051</td>\n",
       "      <td>108775</td>\n",
       "      <td>Strap top (1)</td>\n",
       "      <td>253</td>\n",
       "      <td>Vest top</td>\n",
       "      <td>Garment Upper body</td>\n",
       "      <td>1010017</td>\n",
       "      <td>Stripe</td>\n",
       "      <td>11</td>\n",
       "      <td>Off White</td>\n",
       "      <td>...</td>\n",
       "      <td>Jersey Basic</td>\n",
       "      <td>A</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>1</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>16</td>\n",
       "      <td>Womens Everyday Basics</td>\n",
       "      <td>1002</td>\n",
       "      <td>Jersey Basic</td>\n",
       "      <td>Jersey top with narrow shoulder straps.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>110065001</td>\n",
       "      <td>110065</td>\n",
       "      <td>OP T-shirt (Idro)</td>\n",
       "      <td>306</td>\n",
       "      <td>Bra</td>\n",
       "      <td>Underwear</td>\n",
       "      <td>1010016</td>\n",
       "      <td>Solid</td>\n",
       "      <td>9</td>\n",
       "      <td>Black</td>\n",
       "      <td>...</td>\n",
       "      <td>Clean Lingerie</td>\n",
       "      <td>B</td>\n",
       "      <td>Lingeries/Tights</td>\n",
       "      <td>1</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>61</td>\n",
       "      <td>Womens Lingerie</td>\n",
       "      <td>1017</td>\n",
       "      <td>Under-, Nightwear</td>\n",
       "      <td>Microfibre T-shirt bra with underwired, moulde...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>110065002</td>\n",
       "      <td>110065</td>\n",
       "      <td>OP T-shirt (Idro)</td>\n",
       "      <td>306</td>\n",
       "      <td>Bra</td>\n",
       "      <td>Underwear</td>\n",
       "      <td>1010016</td>\n",
       "      <td>Solid</td>\n",
       "      <td>10</td>\n",
       "      <td>White</td>\n",
       "      <td>...</td>\n",
       "      <td>Clean Lingerie</td>\n",
       "      <td>B</td>\n",
       "      <td>Lingeries/Tights</td>\n",
       "      <td>1</td>\n",
       "      <td>Ladieswear</td>\n",
       "      <td>61</td>\n",
       "      <td>Womens Lingerie</td>\n",
       "      <td>1017</td>\n",
       "      <td>Under-, Nightwear</td>\n",
       "      <td>Microfibre T-shirt bra with underwired, moulde...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   article_id  product_code          prod_name  product_type_no  \\\n",
       "0   108775015        108775          Strap top              253   \n",
       "1   108775044        108775          Strap top              253   \n",
       "2   108775051        108775      Strap top (1)              253   \n",
       "3   110065001        110065  OP T-shirt (Idro)              306   \n",
       "4   110065002        110065  OP T-shirt (Idro)              306   \n",
       "\n",
       "  product_type_name  product_group_name  graphical_appearance_no  \\\n",
       "0          Vest top  Garment Upper body                  1010016   \n",
       "1          Vest top  Garment Upper body                  1010016   \n",
       "2          Vest top  Garment Upper body                  1010017   \n",
       "3               Bra           Underwear                  1010016   \n",
       "4               Bra           Underwear                  1010016   \n",
       "\n",
       "  graphical_appearance_name  colour_group_code colour_group_name  ...  \\\n",
       "0                     Solid                  9             Black  ...   \n",
       "1                     Solid                 10             White  ...   \n",
       "2                    Stripe                 11         Off White  ...   \n",
       "3                     Solid                  9             Black  ...   \n",
       "4                     Solid                 10             White  ...   \n",
       "\n",
       "   department_name index_code        index_name index_group_no  \\\n",
       "0     Jersey Basic          A        Ladieswear              1   \n",
       "1     Jersey Basic          A        Ladieswear              1   \n",
       "2     Jersey Basic          A        Ladieswear              1   \n",
       "3   Clean Lingerie          B  Lingeries/Tights              1   \n",
       "4   Clean Lingerie          B  Lingeries/Tights              1   \n",
       "\n",
       "   index_group_name section_no            section_name garment_group_no  \\\n",
       "0        Ladieswear         16  Womens Everyday Basics             1002   \n",
       "1        Ladieswear         16  Womens Everyday Basics             1002   \n",
       "2        Ladieswear         16  Womens Everyday Basics             1002   \n",
       "3        Ladieswear         61         Womens Lingerie             1017   \n",
       "4        Ladieswear         61         Womens Lingerie             1017   \n",
       "\n",
       "   garment_group_name                                        detail_desc  \n",
       "0        Jersey Basic            Jersey top with narrow shoulder straps.  \n",
       "1        Jersey Basic            Jersey top with narrow shoulder straps.  \n",
       "2        Jersey Basic            Jersey top with narrow shoulder straps.  \n",
       "3   Under-, Nightwear  Microfibre T-shirt bra with underwired, moulde...  \n",
       "4   Under-, Nightwear  Microfibre T-shirt bra with underwired, moulde...  \n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cf9cd77b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 105542 entries, 0 to 105541\n",
      "Data columns (total 25 columns):\n",
      " #   Column                        Non-Null Count   Dtype \n",
      "---  ------                        --------------   ----- \n",
      " 0   article_id                    105542 non-null  int64 \n",
      " 1   product_code                  105542 non-null  int64 \n",
      " 2   prod_name                     105542 non-null  object\n",
      " 3   product_type_no               105542 non-null  int64 \n",
      " 4   product_type_name             105542 non-null  object\n",
      " 5   product_group_name            105542 non-null  object\n",
      " 6   graphical_appearance_no       105542 non-null  int64 \n",
      " 7   graphical_appearance_name     105542 non-null  object\n",
      " 8   colour_group_code             105542 non-null  int64 \n",
      " 9   colour_group_name             105542 non-null  object\n",
      " 10  perceived_colour_value_id     105542 non-null  int64 \n",
      " 11  perceived_colour_value_name   105542 non-null  object\n",
      " 12  perceived_colour_master_id    105542 non-null  int64 \n",
      " 13  perceived_colour_master_name  105542 non-null  object\n",
      " 14  department_no                 105542 non-null  int64 \n",
      " 15  department_name               105542 non-null  object\n",
      " 16  index_code                    105542 non-null  object\n",
      " 17  index_name                    105542 non-null  object\n",
      " 18  index_group_no                105542 non-null  int64 \n",
      " 19  index_group_name              105542 non-null  object\n",
      " 20  section_no                    105542 non-null  int64 \n",
      " 21  section_name                  105542 non-null  object\n",
      " 22  garment_group_no              105542 non-null  int64 \n",
      " 23  garment_group_name            105542 non-null  object\n",
      " 24  detail_desc                   105126 non-null  object\n",
      "dtypes: int64(11), object(14)\n",
      "memory usage: 20.1+ MB\n"
     ]
    }
   ],
   "source": [
    "articles_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e05e704d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>FN</th>\n",
       "      <th>Active</th>\n",
       "      <th>club_member_status</th>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <th>age</th>\n",
       "      <th>postal_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000dbacae5abe5e23885899a1fa44253a17956c6d1c3...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>49.0</td>\n",
       "      <td>52043ee2162cf5aa7ee79974281641c6f11a68d276429a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0000423b00ade91418cceaf3b26c6af3dd342b51fd051e...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2973abc54daa8a5f8ccfe9362140c63247c5eee03f1d93...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>24.0</td>\n",
       "      <td>64f17e6a330a85798e4998f62d0930d14db8db1c054af6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00005ca1c9ed5f5146b52ac8639a40ca9d57aeff4d1bd2...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>54.0</td>\n",
       "      <td>5d36574f52495e81f019b680c843c443bd343d5ca5b1c2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00006413d8573cd20ed7128e53b7b13819fe5cfc2d801f...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>Regularly</td>\n",
       "      <td>52.0</td>\n",
       "      <td>25fa5ddee9aac01b35208d01736e57942317d756b32ddd...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         customer_id   FN  Active  \\\n",
       "0  00000dbacae5abe5e23885899a1fa44253a17956c6d1c3...  NaN     NaN   \n",
       "1  0000423b00ade91418cceaf3b26c6af3dd342b51fd051e...  NaN     NaN   \n",
       "2  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...  NaN     NaN   \n",
       "3  00005ca1c9ed5f5146b52ac8639a40ca9d57aeff4d1bd2...  NaN     NaN   \n",
       "4  00006413d8573cd20ed7128e53b7b13819fe5cfc2d801f...  1.0     1.0   \n",
       "\n",
       "  club_member_status fashion_news_frequency   age  \\\n",
       "0             ACTIVE                   NONE  49.0   \n",
       "1             ACTIVE                   NONE  25.0   \n",
       "2             ACTIVE                   NONE  24.0   \n",
       "3             ACTIVE                   NONE  54.0   \n",
       "4             ACTIVE              Regularly  52.0   \n",
       "\n",
       "                                         postal_code  \n",
       "0  52043ee2162cf5aa7ee79974281641c6f11a68d276429a...  \n",
       "1  2973abc54daa8a5f8ccfe9362140c63247c5eee03f1d93...  \n",
       "2  64f17e6a330a85798e4998f62d0930d14db8db1c054af6...  \n",
       "3  5d36574f52495e81f019b680c843c443bd343d5ca5b1c2...  \n",
       "4  25fa5ddee9aac01b35208d01736e57942317d756b32ddd...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e8d786ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1371980 entries, 0 to 1371979\n",
      "Data columns (total 7 columns):\n",
      " #   Column                  Non-Null Count    Dtype  \n",
      "---  ------                  --------------    -----  \n",
      " 0   customer_id             1371980 non-null  object \n",
      " 1   FN                      476930 non-null   float64\n",
      " 2   Active                  464404 non-null   float64\n",
      " 3   club_member_status      1365918 non-null  object \n",
      " 4   fashion_news_frequency  1355971 non-null  object \n",
      " 5   age                     1356119 non-null  float64\n",
      " 6   postal_code             1371980 non-null  object \n",
      "dtypes: float64(3), object(4)\n",
      "memory usage: 73.3+ MB\n"
     ]
    }
   ],
   "source": [
    "customers_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d0eef394",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>663713001</td>\n",
       "      <td>0.050831</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>541518023</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>00007d2de826758b65a93dd24ce629ed66842531df6699...</td>\n",
       "      <td>505221004</td>\n",
       "      <td>0.015237</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>00007d2de826758b65a93dd24ce629ed66842531df6699...</td>\n",
       "      <td>685687003</td>\n",
       "      <td>0.016932</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>00007d2de826758b65a93dd24ce629ed66842531df6699...</td>\n",
       "      <td>685687004</td>\n",
       "      <td>0.016932</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        t_dat                                        customer_id  article_id  \\\n",
       "0  2018-09-20  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...   663713001   \n",
       "1  2018-09-20  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...   541518023   \n",
       "2  2018-09-20  00007d2de826758b65a93dd24ce629ed66842531df6699...   505221004   \n",
       "3  2018-09-20  00007d2de826758b65a93dd24ce629ed66842531df6699...   685687003   \n",
       "4  2018-09-20  00007d2de826758b65a93dd24ce629ed66842531df6699...   685687004   \n",
       "\n",
       "      price  sales_channel_id  \n",
       "0  0.050831                 2  \n",
       "1  0.030492                 2  \n",
       "2  0.015237                 2  \n",
       "3  0.016932                 2  \n",
       "4  0.016932                 2  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcations_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2a716b67",
   "metadata": {},
   "outputs": [],
   "source": [
    "def missing_data(func_df):\n",
    "    total=func_df.isnull().sum().sort_values(ascending=False)\n",
    "    percent=(func_df.isnull().sum()/func_df.isnull().count()*100).sort_values(ascending=False)\n",
    "    return pd.concat([total,percent],axis=1,keys=['Total','Percent'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "41945da1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>detail_desc</th>\n",
       "      <td>416</td>\n",
       "      <td>0.394156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garment_group_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garment_group_no</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>section_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>section_no</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_group_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_group_no</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_code</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>department_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>department_no</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_code</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_value_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_value_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>colour_group_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>colour_group_code</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>graphical_appearance_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_group_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_type_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_type_no</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prod_name</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Total   Percent\n",
       "detail_desc                     416  0.394156\n",
       "perceived_colour_master_name      0  0.000000\n",
       "garment_group_name                0  0.000000\n",
       "garment_group_no                  0  0.000000\n",
       "section_name                      0  0.000000\n",
       "section_no                        0  0.000000\n",
       "index_group_name                  0  0.000000\n",
       "index_group_no                    0  0.000000\n",
       "index_name                        0  0.000000\n",
       "index_code                        0  0.000000\n",
       "department_name                   0  0.000000\n",
       "department_no                     0  0.000000\n",
       "article_id                        0  0.000000\n",
       "product_code                      0  0.000000\n",
       "perceived_colour_value_name       0  0.000000\n",
       "perceived_colour_value_id         0  0.000000\n",
       "colour_group_name                 0  0.000000\n",
       "colour_group_code                 0  0.000000\n",
       "graphical_appearance_name         0  0.000000\n",
       "graphical_appearance_no           0  0.000000\n",
       "product_group_name                0  0.000000\n",
       "product_type_name                 0  0.000000\n",
       "product_type_no                   0  0.000000\n",
       "prod_name                         0  0.000000\n",
       "perceived_colour_master_id        0  0.000000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_data(articles_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b000a25e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Active</th>\n",
       "      <td>907576</td>\n",
       "      <td>66.150819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FN</th>\n",
       "      <td>895050</td>\n",
       "      <td>65.237831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <td>16009</td>\n",
       "      <td>1.166854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>15861</td>\n",
       "      <td>1.156066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>club_member_status</th>\n",
       "      <td>6062</td>\n",
       "      <td>0.441843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customer_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>postal_code</th>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         Total    Percent\n",
       "Active                  907576  66.150819\n",
       "FN                      895050  65.237831\n",
       "fashion_news_frequency   16009   1.166854\n",
       "age                      15861   1.156066\n",
       "club_member_status        6062   0.441843\n",
       "customer_id                  0   0.000000\n",
       "postal_code                  0   0.000000"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_data(customers_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5c1cb217",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>t_dat</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customer_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>price</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sales_channel_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Total  Percent\n",
       "t_dat                 0      0.0\n",
       "customer_id           0      0.0\n",
       "article_id            0      0.0\n",
       "price                 0      0.0\n",
       "sales_channel_id      0      0.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_data(transcations_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3d4d92ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "def unique_values(func_df):\n",
    "    total=func_df.count()\n",
    "    tt=pd.DataFrame(total)\n",
    "    tt.columns=['Total']\n",
    "    uniques=[]\n",
    "    for col in func_df.columns:\n",
    "        unique=func_df[col].nunique()\n",
    "        uniques.append(unique)\n",
    "    tt['Uniques']=uniques\n",
    "    return tt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c9c65847",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>Uniques</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <td>105542</td>\n",
       "      <td>105542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_code</th>\n",
       "      <td>105542</td>\n",
       "      <td>47224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prod_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>45875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_type_no</th>\n",
       "      <td>105542</td>\n",
       "      <td>132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_type_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product_group_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <td>105542</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>graphical_appearance_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>colour_group_code</th>\n",
       "      <td>105542</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>colour_group_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_value_id</th>\n",
       "      <td>105542</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_value_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <td>105542</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>department_no</th>\n",
       "      <td>105542</td>\n",
       "      <td>299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>department_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_code</th>\n",
       "      <td>105542</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_group_no</th>\n",
       "      <td>105542</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index_group_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>section_no</th>\n",
       "      <td>105542</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>section_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garment_group_no</th>\n",
       "      <td>105542</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garment_group_name</th>\n",
       "      <td>105542</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>detail_desc</th>\n",
       "      <td>105126</td>\n",
       "      <td>43404</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               Total  Uniques\n",
       "article_id                    105542   105542\n",
       "product_code                  105542    47224\n",
       "prod_name                     105542    45875\n",
       "product_type_no               105542      132\n",
       "product_type_name             105542      131\n",
       "product_group_name            105542       19\n",
       "graphical_appearance_no       105542       30\n",
       "graphical_appearance_name     105542       30\n",
       "colour_group_code             105542       50\n",
       "colour_group_name             105542       50\n",
       "perceived_colour_value_id     105542        8\n",
       "perceived_colour_value_name   105542        8\n",
       "perceived_colour_master_id    105542       20\n",
       "perceived_colour_master_name  105542       20\n",
       "department_no                 105542      299\n",
       "department_name               105542      250\n",
       "index_code                    105542       10\n",
       "index_name                    105542       10\n",
       "index_group_no                105542        5\n",
       "index_group_name              105542        5\n",
       "section_no                    105542       57\n",
       "section_name                  105542       56\n",
       "garment_group_no              105542       21\n",
       "garment_group_name            105542       21\n",
       "detail_desc                   105126    43404"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_values(articles_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ce968c68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>Uniques</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>customer_id</th>\n",
       "      <td>1371980</td>\n",
       "      <td>1371980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FN</th>\n",
       "      <td>476930</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Active</th>\n",
       "      <td>464404</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>club_member_status</th>\n",
       "      <td>1365918</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <td>1355971</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>1356119</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>postal_code</th>\n",
       "      <td>1371980</td>\n",
       "      <td>352899</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Total  Uniques\n",
       "customer_id             1371980  1371980\n",
       "FN                       476930        1\n",
       "Active                   464404        1\n",
       "club_member_status      1365918        3\n",
       "fashion_news_frequency  1355971        4\n",
       "age                     1356119       84\n",
       "postal_code             1371980   352899"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_values(customers_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d0b2843e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this block of code is used to map prod_name and product_code\n",
    "index_df = pd.DataFrame(columns = ['product_no','product_name'])\n",
    "for index,row in articles_df.iterrows():\n",
    "    if row['product_code'] not in index_df.product_no.unique():\n",
    "        index_df=index_df.append({'product_no':row['product_code'],'product_name':row['prod_name']},ignore_index = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "979cd5dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_no</th>\n",
       "      <th>product_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>108775</td>\n",
       "      <td>Strap top</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110065</td>\n",
       "      <td>OP T-shirt (Idro)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>111565</td>\n",
       "      <td>20 den 1p Stockings</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>111586</td>\n",
       "      <td>Shape Up 30 den 1p Tights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>111593</td>\n",
       "      <td>Support 40 den 1p Tights</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  product_no               product_name\n",
       "0     108775                  Strap top\n",
       "1     110065          OP T-shirt (Idro)\n",
       "2     111565        20 den 1p Stockings\n",
       "3     111586  Shape Up 30 den 1p Tights\n",
       "4     111593   Support 40 den 1p Tights"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7f9b9b36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(47224, 2)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "048858fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_no</th>\n",
       "      <th>product_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>108775</td>\n",
       "      <td>Strap top</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110065</td>\n",
       "      <td>OP T-shirt (Idro)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>111565</td>\n",
       "      <td>20 den 1p Stockings</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>111586</td>\n",
       "      <td>Shape Up 30 den 1p Tights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>111593</td>\n",
       "      <td>Support 40 den 1p Tights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47219</th>\n",
       "      <td>953450</td>\n",
       "      <td>5pk regular Placement1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47220</th>\n",
       "      <td>953763</td>\n",
       "      <td>SPORT Malaga tank</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47221</th>\n",
       "      <td>956217</td>\n",
       "      <td>Cartwheel dress</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47222</th>\n",
       "      <td>957375</td>\n",
       "      <td>CLAIRE HAIR CLAW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47223</th>\n",
       "      <td>959461</td>\n",
       "      <td>Lounge dress</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>47224 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      product_no               product_name\n",
       "0         108775                  Strap top\n",
       "1         110065          OP T-shirt (Idro)\n",
       "2         111565        20 den 1p Stockings\n",
       "3         111586  Shape Up 30 den 1p Tights\n",
       "4         111593   Support 40 den 1p Tights\n",
       "...          ...                        ...\n",
       "47219     953450     5pk regular Placement1\n",
       "47220     953763          SPORT Malaga tank\n",
       "47221     956217            Cartwheel dress\n",
       "47222     957375           CLAIRE HAIR CLAW\n",
       "47223     959461               Lounge dress\n",
       "\n",
       "[47224 rows x 2 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_df.sort_values('product_no')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "51855091",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_df.to_csv('product_label_dataset',encoding='utf-8',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "40861eab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>Uniques</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>t_dat</th>\n",
       "      <td>31788324</td>\n",
       "      <td>734</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customer_id</th>\n",
       "      <td>31788324</td>\n",
       "      <td>1362281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <td>31788324</td>\n",
       "      <td>104547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>price</th>\n",
       "      <td>31788324</td>\n",
       "      <td>9857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sales_channel_id</th>\n",
       "      <td>31788324</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Total  Uniques\n",
       "t_dat             31788324      734\n",
       "customer_id       31788324  1362281\n",
       "article_id        31788324   104547\n",
       "price             31788324     9857\n",
       "sales_channel_id  31788324        2"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_values(transcations_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6385d1a7",
   "metadata": {},
   "source": [
    "#### Here i am dropping this columns because it has the column which represents in the number format in simple words to reduce multicollinearity\n",
    "#### Eg product_name column has product_name which is same "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10f78614",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols=['prod_name','product_type_name','graphical_appearance_name','colour_group_name','perceived_colour_value_name',\n",
    "      'department_name','index_name','index_group_name','section_name','garment_group_name']\n",
    "articles_df.drop(columns=cols,axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71ef7db8",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25a700dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_encoder = preprocessing.LabelEncoder()\n",
    "cols=['product_group_name','perceived_colour_master_name','index_code']\n",
    "for col in cols:\n",
    "    label_cols.append(col)\n",
    "    articles_df[col]= label_encoder.fit_transform(articles_df[col]) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc4ab85c",
   "metadata": {},
   "source": [
    "### I will be dropping detail_desc column "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "958735b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_df.drop(['detail_desc'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "41f7b6e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>product_group_name</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>perceived_colour_value_id</th>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <th>department_no</th>\n",
       "      <th>index_code</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>section_no</th>\n",
       "      <th>garment_group_no</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>108775015</td>\n",
       "      <td>108775</td>\n",
       "      <td>253</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1676</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>1002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>108775044</td>\n",
       "      <td>108775</td>\n",
       "      <td>253</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>1676</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>1002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>108775051</td>\n",
       "      <td>108775</td>\n",
       "      <td>253</td>\n",
       "      <td>7</td>\n",
       "      <td>1010017</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>1676</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>1002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>110065001</td>\n",
       "      <td>110065</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1339</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>110065002</td>\n",
       "      <td>110065</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>1339</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105537</th>\n",
       "      <td>953450001</td>\n",
       "      <td>953450</td>\n",
       "      <td>302</td>\n",
       "      <td>13</td>\n",
       "      <td>1010014</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>7188</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>26</td>\n",
       "      <td>1021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105538</th>\n",
       "      <td>953763001</td>\n",
       "      <td>953763</td>\n",
       "      <td>253</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1919</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105539</th>\n",
       "      <td>956217002</td>\n",
       "      <td>956217</td>\n",
       "      <td>265</td>\n",
       "      <td>5</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1641</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>1005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105540</th>\n",
       "      <td>957375001</td>\n",
       "      <td>957375</td>\n",
       "      <td>72</td>\n",
       "      <td>0</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>3946</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>52</td>\n",
       "      <td>1019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105541</th>\n",
       "      <td>959461001</td>\n",
       "      <td>959461</td>\n",
       "      <td>265</td>\n",
       "      <td>5</td>\n",
       "      <td>1010016</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>1641</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>1005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>105542 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        article_id  product_code  product_type_no  product_group_name  \\\n",
       "0        108775015        108775              253                   7   \n",
       "1        108775044        108775              253                   7   \n",
       "2        108775051        108775              253                   7   \n",
       "3        110065001        110065              306                  16   \n",
       "4        110065002        110065              306                  16   \n",
       "...            ...           ...              ...                 ...   \n",
       "105537   953450001        953450              302                  13   \n",
       "105538   953763001        953763              253                   7   \n",
       "105539   956217002        956217              265                   5   \n",
       "105540   957375001        957375               72                   0   \n",
       "105541   959461001        959461              265                   5   \n",
       "\n",
       "        graphical_appearance_no  colour_group_code  perceived_colour_value_id  \\\n",
       "0                       1010016                  9                          4   \n",
       "1                       1010016                 10                          3   \n",
       "2                       1010017                 11                          1   \n",
       "3                       1010016                  9                          4   \n",
       "4                       1010016                 10                          3   \n",
       "...                         ...                ...                        ...   \n",
       "105537                  1010014                  9                          4   \n",
       "105538                  1010016                  9                          4   \n",
       "105539                  1010016                  9                          4   \n",
       "105540                  1010016                  9                          4   \n",
       "105541                  1010016                 11                          1   \n",
       "\n",
       "        perceived_colour_master_id  perceived_colour_master_name  \\\n",
       "0                                5                             1   \n",
       "1                                9                            16   \n",
       "2                                9                            16   \n",
       "3                                5                             1   \n",
       "4                                9                            16   \n",
       "...                            ...                           ...   \n",
       "105537                           5                             1   \n",
       "105538                           5                             1   \n",
       "105539                           5                             1   \n",
       "105540                           5                             1   \n",
       "105541                           9                            16   \n",
       "\n",
       "        department_no  index_code  index_group_no  section_no  \\\n",
       "0                1676           0               1          16   \n",
       "1                1676           0               1          16   \n",
       "2                1676           0               1          16   \n",
       "3                1339           1               1          61   \n",
       "4                1339           1               1          61   \n",
       "...               ...         ...             ...         ...   \n",
       "105537           7188           4               3          26   \n",
       "105538           1919           0               1           2   \n",
       "105539           1641           0               1          18   \n",
       "105540           3946           3               2          52   \n",
       "105541           1641           0               1          18   \n",
       "\n",
       "        garment_group_no  \n",
       "0                   1002  \n",
       "1                   1002  \n",
       "2                   1002  \n",
       "3                   1017  \n",
       "4                   1017  \n",
       "...                  ...  \n",
       "105537              1021  \n",
       "105538              1005  \n",
       "105539              1005  \n",
       "105540              1019  \n",
       "105541              1005  \n",
       "\n",
       "[105542 rows x 14 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7c8a7b82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>FN</th>\n",
       "      <th>Active</th>\n",
       "      <th>club_member_status</th>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <th>age</th>\n",
       "      <th>postal_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000dbacae5abe5e23885899a1fa44253a17956c6d1c3...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>49.0</td>\n",
       "      <td>52043ee2162cf5aa7ee79974281641c6f11a68d276429a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0000423b00ade91418cceaf3b26c6af3dd342b51fd051e...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2973abc54daa8a5f8ccfe9362140c63247c5eee03f1d93...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>24.0</td>\n",
       "      <td>64f17e6a330a85798e4998f62d0930d14db8db1c054af6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00005ca1c9ed5f5146b52ac8639a40ca9d57aeff4d1bd2...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>54.0</td>\n",
       "      <td>5d36574f52495e81f019b680c843c443bd343d5ca5b1c2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00006413d8573cd20ed7128e53b7b13819fe5cfc2d801f...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>Regularly</td>\n",
       "      <td>52.0</td>\n",
       "      <td>25fa5ddee9aac01b35208d01736e57942317d756b32ddd...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         customer_id   FN  Active  \\\n",
       "0  00000dbacae5abe5e23885899a1fa44253a17956c6d1c3...  NaN     NaN   \n",
       "1  0000423b00ade91418cceaf3b26c6af3dd342b51fd051e...  NaN     NaN   \n",
       "2  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...  NaN     NaN   \n",
       "3  00005ca1c9ed5f5146b52ac8639a40ca9d57aeff4d1bd2...  NaN     NaN   \n",
       "4  00006413d8573cd20ed7128e53b7b13819fe5cfc2d801f...  1.0     1.0   \n",
       "\n",
       "  club_member_status fashion_news_frequency   age  \\\n",
       "0             ACTIVE                   NONE  49.0   \n",
       "1             ACTIVE                   NONE  25.0   \n",
       "2             ACTIVE                   NONE  24.0   \n",
       "3             ACTIVE                   NONE  54.0   \n",
       "4             ACTIVE              Regularly  52.0   \n",
       "\n",
       "                                         postal_code  \n",
       "0  52043ee2162cf5aa7ee79974281641c6f11a68d276429a...  \n",
       "1  2973abc54daa8a5f8ccfe9362140c63247c5eee03f1d93...  \n",
       "2  64f17e6a330a85798e4998f62d0930d14db8db1c054af6...  \n",
       "3  5d36574f52495e81f019b680c843c443bd343d5ca5b1c2...  \n",
       "4  25fa5ddee9aac01b35208d01736e57942317d756b32ddd...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "60d832fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1371980, 7)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "77937a0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NONE         877711\n",
       "Regularly    477416\n",
       "Monthly         842\n",
       "None              2\n",
       "Name: fashion_news_frequency, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df['fashion_news_frequency'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1e1eda52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The precentage of null value in FN column is 65.24%\n",
      "The precentage of null value in Active column is 66.15%\n"
     ]
    }
   ],
   "source": [
    "print(\"The precentage of null value in FN column is {}%\".format(round(((customers_df['FN'].isna().sum()/customers_df.shape[0])*100),2)))\n",
    "print(\"The precentage of null value in Active column is {}%\".format(round(((customers_df['Active'].isna().sum()/customers_df.shape[0])*100),2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612e47e6",
   "metadata": {},
   "source": [
    "### I am dropping the FN and Active column because it has more than 65 % null values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "d85a7d70",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_df.drop(['FN','Active'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9b478cf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6062"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df['club_member_status'].isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6795a488",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_df['club_member_status'].fillna('ACTIVE',axis=0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6c6692e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df['club_member_status'].isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a55fccde",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_df['fashion_news_frequency'].replace(to_replace='NONE',value='None',inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "3e3d5896",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "None         877713\n",
       "Regularly    477416\n",
       "Monthly         842\n",
       "Name: fashion_news_frequency, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df['fashion_news_frequency'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "4651eca8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "352899"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df['postal_code'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b44fef02",
   "metadata": {},
   "source": [
    "### Since we has so many unique values in postal_code so  i will be dropping this column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "082805df",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_df.drop(['postal_code'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "d7ca5e85",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_df.dropna(axis=0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4bcd5157",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_encoder = preprocessing.LabelEncoder()\n",
    "cols=['club_member_status','fashion_news_frequency']\n",
    "for col in cols:\n",
    "    label_cols.append(col)\n",
    "    customers_df[col]= label_encoder.fit_transform(customers_df[col]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "83717cc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>club_member_status</th>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000dbacae5abe5e23885899a1fa44253a17956c6d1c3...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>49.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0000423b00ade91418cceaf3b26c6af3dd342b51fd051e...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00005ca1c9ed5f5146b52ac8639a40ca9d57aeff4d1bd2...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>54.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00006413d8573cd20ed7128e53b7b13819fe5cfc2d801f...</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>52.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         customer_id  club_member_status  \\\n",
       "0  00000dbacae5abe5e23885899a1fa44253a17956c6d1c3...                   0   \n",
       "1  0000423b00ade91418cceaf3b26c6af3dd342b51fd051e...                   0   \n",
       "2  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...                   0   \n",
       "3  00005ca1c9ed5f5146b52ac8639a40ca9d57aeff4d1bd2...                   0   \n",
       "4  00006413d8573cd20ed7128e53b7b13819fe5cfc2d801f...                   0   \n",
       "\n",
       "   fashion_news_frequency   age  \n",
       "0                       1  49.0  \n",
       "1                       1  25.0  \n",
       "2                       1  24.0  \n",
       "3                       1  54.0  \n",
       "4                       2  52.0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customers_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "0114f0aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>663713001</td>\n",
       "      <td>0.050831</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...</td>\n",
       "      <td>541518023</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>00007d2de826758b65a93dd24ce629ed66842531df6699...</td>\n",
       "      <td>505221004</td>\n",
       "      <td>0.015237</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>00007d2de826758b65a93dd24ce629ed66842531df6699...</td>\n",
       "      <td>685687003</td>\n",
       "      <td>0.016932</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-09-20</td>\n",
       "      <td>00007d2de826758b65a93dd24ce629ed66842531df6699...</td>\n",
       "      <td>685687004</td>\n",
       "      <td>0.016932</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        t_dat                                        customer_id  article_id  \\\n",
       "0  2018-09-20  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...   663713001   \n",
       "1  2018-09-20  000058a12d5b43e67d225668fa1f8d618c13dc232df0ca...   541518023   \n",
       "2  2018-09-20  00007d2de826758b65a93dd24ce629ed66842531df6699...   505221004   \n",
       "3  2018-09-20  00007d2de826758b65a93dd24ce629ed66842531df6699...   685687003   \n",
       "4  2018-09-20  00007d2de826758b65a93dd24ce629ed66842531df6699...   685687004   \n",
       "\n",
       "      price  sales_channel_id  \n",
       "0  0.050831                 2  \n",
       "1  0.030492                 2  \n",
       "2  0.015237                 2  \n",
       "3  0.016932                 2  \n",
       "4  0.016932                 2  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcations_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b013c76",
   "metadata": {},
   "source": [
    "### There is no null values in transcations "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "30d16db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "transcations_df['t_dat']=pd.to_datetime(transcations_df[\"t_dat\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "834d7c98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>31788319</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>fff2282977442e327b45d8c89afde25617d00124d0f999...</td>\n",
       "      <td>929511001</td>\n",
       "      <td>0.059305</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31788320</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>fff2282977442e327b45d8c89afde25617d00124d0f999...</td>\n",
       "      <td>891322004</td>\n",
       "      <td>0.042356</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31788321</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>fff380805474b287b05cb2a7507b9a013482f7dd0bce0e...</td>\n",
       "      <td>918325001</td>\n",
       "      <td>0.043203</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31788322</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>fff4d3a8b1f3b60af93e78c30a7cb4cf75edaf2590d3e5...</td>\n",
       "      <td>833459002</td>\n",
       "      <td>0.006763</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31788323</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>fffef3b6b73545df065b521e19f64bf6fe93bfd450ab20...</td>\n",
       "      <td>898573003</td>\n",
       "      <td>0.033881</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              t_dat                                        customer_id  \\\n",
       "31788319 2020-09-22  fff2282977442e327b45d8c89afde25617d00124d0f999...   \n",
       "31788320 2020-09-22  fff2282977442e327b45d8c89afde25617d00124d0f999...   \n",
       "31788321 2020-09-22  fff380805474b287b05cb2a7507b9a013482f7dd0bce0e...   \n",
       "31788322 2020-09-22  fff4d3a8b1f3b60af93e78c30a7cb4cf75edaf2590d3e5...   \n",
       "31788323 2020-09-22  fffef3b6b73545df065b521e19f64bf6fe93bfd450ab20...   \n",
       "\n",
       "          article_id     price  sales_channel_id  \n",
       "31788319   929511001  0.059305                 2  \n",
       "31788320   891322004  0.042356                 2  \n",
       "31788321   918325001  0.043203                 1  \n",
       "31788322   833459002  0.006763                 1  \n",
       "31788323   898573003  0.033881                 2  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcations_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "46c6452d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(31788324, 5)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcations_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "215d837f",
   "metadata": {},
   "source": [
    "### Here i have selected only the transcations after the year 2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "95506ee3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10967372, 5)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcations_df=transcations_df[transcations_df['t_dat']>'2020-01-01'].copy()\n",
    "transcations_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "31fd034f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20820952</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20820953</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565001</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20820954</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00067622de3151a7219b4ed9922def50b51601fbe41418...</td>\n",
       "      <td>801865004</td>\n",
       "      <td>0.018627</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20820955</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>0010f56acce349e6e82bfef13ee39232a8bc0db0801ca4...</td>\n",
       "      <td>578752001</td>\n",
       "      <td>0.020322</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20820956</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>0010f56acce349e6e82bfef13ee39232a8bc0db0801ca4...</td>\n",
       "      <td>578752001</td>\n",
       "      <td>0.020339</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              t_dat                                        customer_id  \\\n",
       "20820952 2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "20820953 2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "20820954 2020-01-02  00067622de3151a7219b4ed9922def50b51601fbe41418...   \n",
       "20820955 2020-01-02  0010f56acce349e6e82bfef13ee39232a8bc0db0801ca4...   \n",
       "20820956 2020-01-02  0010f56acce349e6e82bfef13ee39232a8bc0db0801ca4...   \n",
       "\n",
       "          article_id     price  sales_channel_id  \n",
       "20820952   797565002  0.030492                 2  \n",
       "20820953   797565001  0.030492                 2  \n",
       "20820954   801865004  0.018627                 1  \n",
       "20820955   578752001  0.020322                 2  \n",
       "20820956   578752001  0.020339                 2  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcations_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7d37db23",
   "metadata": {},
   "outputs": [],
   "source": [
    "inner_merged=pd.merge(transcations_df,articles_df)#inner merge "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "5e856b04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10967372, 18)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inner_merged.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "67d5aa6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>product_group_name</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>perceived_colour_value_id</th>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <th>department_no</th>\n",
       "      <th>index_code</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>section_no</th>\n",
       "      <th>garment_group_no</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>0a472ecdcdf4c940007edc47e9274a525ef2f0350f2a57...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>470dac37792d1e124865d249d1e304f136c11ea46d2ec9...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.029695</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>6593c2dcf880bf65dc68a3c3e6b29646bf95b0e93fae09...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>a4eccd4d68777d7e6076ef26e888ac6dbfa34d371fae6f...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       t_dat                                        customer_id  article_id  \\\n",
       "0 2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   797565002   \n",
       "1 2020-01-02  0a472ecdcdf4c940007edc47e9274a525ef2f0350f2a57...   797565002   \n",
       "2 2020-01-02  470dac37792d1e124865d249d1e304f136c11ea46d2ec9...   797565002   \n",
       "3 2020-01-02  6593c2dcf880bf65dc68a3c3e6b29646bf95b0e93fae09...   797565002   \n",
       "4 2020-01-02  a4eccd4d68777d7e6076ef26e888ac6dbfa34d371fae6f...   797565002   \n",
       "\n",
       "      price  sales_channel_id  product_code  product_type_no  \\\n",
       "0  0.030492                 2        797565              306   \n",
       "1  0.030492                 2        797565              306   \n",
       "2  0.029695                 2        797565              306   \n",
       "3  0.030492                 2        797565              306   \n",
       "4  0.030492                 2        797565              306   \n",
       "\n",
       "   product_group_name  graphical_appearance_no  colour_group_code  \\\n",
       "0                  16                  1010016                 42   \n",
       "1                  16                  1010016                 42   \n",
       "2                  16                  1010016                 42   \n",
       "3                  16                  1010016                 42   \n",
       "4                  16                  1010016                 42   \n",
       "\n",
       "   perceived_colour_value_id  perceived_colour_master_id  \\\n",
       "0                          7                          18   \n",
       "1                          7                          18   \n",
       "2                          7                          18   \n",
       "3                          7                          18   \n",
       "4                          7                          18   \n",
       "\n",
       "   perceived_colour_master_name  department_no  index_code  index_group_no  \\\n",
       "0                            13           1338           1               1   \n",
       "1                            13           1338           1               1   \n",
       "2                            13           1338           1               1   \n",
       "3                            13           1338           1               1   \n",
       "4                            13           1338           1               1   \n",
       "\n",
       "   section_no  garment_group_no  \n",
       "0          61              1017  \n",
       "1          61              1017  \n",
       "2          61              1017  \n",
       "3          61              1017  \n",
       "4          61              1017  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inner_merged.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "b90c0137",
   "metadata": {},
   "outputs": [],
   "source": [
    "final_merged=pd.merge(inner_merged,customers_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "bc6aad54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10906163, 21)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_merged.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "57128369",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>product_group_name</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>...</th>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <th>department_no</th>\n",
       "      <th>index_code</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>section_no</th>\n",
       "      <th>garment_group_no</th>\n",
       "      <th>club_member_status</th>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565001</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-08-14</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>751471001</td>\n",
       "      <td>0.033881</td>\n",
       "      <td>2</td>\n",
       "      <td>751471</td>\n",
       "      <td>272</td>\n",
       "      <td>6</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1722</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1009</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-08-14</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>781613006</td>\n",
       "      <td>0.050831</td>\n",
       "      <td>2</td>\n",
       "      <td>781613</td>\n",
       "      <td>264</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1222</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1008</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-04-24</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>855823003</td>\n",
       "      <td>0.025407</td>\n",
       "      <td>2</td>\n",
       "      <td>855823</td>\n",
       "      <td>265</td>\n",
       "      <td>5</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1344</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>1013</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       t_dat                                        customer_id  article_id  \\\n",
       "0 2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   797565002   \n",
       "1 2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   797565001   \n",
       "2 2020-08-14  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   751471001   \n",
       "3 2020-08-14  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   781613006   \n",
       "4 2020-04-24  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   855823003   \n",
       "\n",
       "      price  sales_channel_id  product_code  product_type_no  \\\n",
       "0  0.030492                 2        797565              306   \n",
       "1  0.030492                 2        797565              306   \n",
       "2  0.033881                 2        751471              272   \n",
       "3  0.050831                 2        781613              264   \n",
       "4  0.025407                 2        855823              265   \n",
       "\n",
       "   product_group_name  graphical_appearance_no  colour_group_code  ...  \\\n",
       "0                  16                  1010016                 42  ...   \n",
       "1                  16                  1010016                  9  ...   \n",
       "2                   6                  1010016                  9  ...   \n",
       "3                   7                  1010016                  9  ...   \n",
       "4                   5                  1010016                  9  ...   \n",
       "\n",
       "   perceived_colour_master_id  perceived_colour_master_name  department_no  \\\n",
       "0                          18                            13           1338   \n",
       "1                           5                             1           1338   \n",
       "2                           5                             1           1722   \n",
       "3                           5                             1           1222   \n",
       "4                           5                             1           1344   \n",
       "\n",
       "   index_code  index_group_no  section_no  garment_group_no  \\\n",
       "0           1               1          61              1017   \n",
       "1           1               1          61              1017   \n",
       "2           0               1          15              1009   \n",
       "3           0               1          15              1008   \n",
       "4           3               2          53              1013   \n",
       "\n",
       "   club_member_status  fashion_news_frequency   age  \n",
       "0                   0                       2  25.0  \n",
       "1                   0                       2  25.0  \n",
       "2                   0                       2  25.0  \n",
       "3                   0                       2  25.0  \n",
       "4                   0                       2  25.0  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_merged.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a61a0e13",
   "metadata": {},
   "source": [
    "### We dont need article_id and customer_id anymore so we will be dropping that too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "665edc1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "int_cols=[]\n",
    "total_columms=final_merged.columns\n",
    "for col in total_columms:\n",
    "    if col not in label_cols:\n",
    "        int_cols.append(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "699e55e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>perceived_colour_value_id</th>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <th>department_no</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>section_no</th>\n",
       "      <th>garment_group_no</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>18</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565001</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-08-14</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>751471001</td>\n",
       "      <td>0.033881</td>\n",
       "      <td>2</td>\n",
       "      <td>751471</td>\n",
       "      <td>272</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1722</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1009</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-08-14</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>781613006</td>\n",
       "      <td>0.050831</td>\n",
       "      <td>2</td>\n",
       "      <td>781613</td>\n",
       "      <td>264</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1222</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1008</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-04-24</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>855823003</td>\n",
       "      <td>0.025407</td>\n",
       "      <td>2</td>\n",
       "      <td>855823</td>\n",
       "      <td>265</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1344</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>1013</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10906158</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>10d455608b1d73268f0a9ff1a392ccd681e665ed471f6d...</td>\n",
       "      <td>932182001</td>\n",
       "      <td>0.033881</td>\n",
       "      <td>2</td>\n",
       "      <td>932182</td>\n",
       "      <td>256</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1636</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1005</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10906159</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>b834a2cb440d2298493d254196abb73e101d8b35b155c6...</td>\n",
       "      <td>870982002</td>\n",
       "      <td>0.061000</td>\n",
       "      <td>2</td>\n",
       "      <td>870982</td>\n",
       "      <td>261</td>\n",
       "      <td>1010016</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1244</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>1007</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10906160</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>83b8a97fd3decf8e21d05a9acaea822eaa569c957ea232...</td>\n",
       "      <td>893427001</td>\n",
       "      <td>0.084729</td>\n",
       "      <td>2</td>\n",
       "      <td>893427</td>\n",
       "      <td>267</td>\n",
       "      <td>1010018</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1722</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1009</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10906161</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>83b8a97fd3decf8e21d05a9acaea822eaa569c957ea232...</td>\n",
       "      <td>893427001</td>\n",
       "      <td>0.084729</td>\n",
       "      <td>2</td>\n",
       "      <td>893427</td>\n",
       "      <td>267</td>\n",
       "      <td>1010018</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1722</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1009</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10906162</th>\n",
       "      <td>2020-09-22</td>\n",
       "      <td>83b8a97fd3decf8e21d05a9acaea822eaa569c957ea232...</td>\n",
       "      <td>903910002</td>\n",
       "      <td>0.084729</td>\n",
       "      <td>2</td>\n",
       "      <td>903910</td>\n",
       "      <td>265</td>\n",
       "      <td>1010016</td>\n",
       "      <td>83</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>1313</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1013</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10906163 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              t_dat                                        customer_id  \\\n",
       "0        2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "1        2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "2        2020-08-14  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "3        2020-08-14  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "4        2020-04-24  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   \n",
       "...             ...                                                ...   \n",
       "10906158 2020-09-22  10d455608b1d73268f0a9ff1a392ccd681e665ed471f6d...   \n",
       "10906159 2020-09-22  b834a2cb440d2298493d254196abb73e101d8b35b155c6...   \n",
       "10906160 2020-09-22  83b8a97fd3decf8e21d05a9acaea822eaa569c957ea232...   \n",
       "10906161 2020-09-22  83b8a97fd3decf8e21d05a9acaea822eaa569c957ea232...   \n",
       "10906162 2020-09-22  83b8a97fd3decf8e21d05a9acaea822eaa569c957ea232...   \n",
       "\n",
       "          article_id     price  sales_channel_id  product_code  \\\n",
       "0          797565002  0.030492                 2        797565   \n",
       "1          797565001  0.030492                 2        797565   \n",
       "2          751471001  0.033881                 2        751471   \n",
       "3          781613006  0.050831                 2        781613   \n",
       "4          855823003  0.025407                 2        855823   \n",
       "...              ...       ...               ...           ...   \n",
       "10906158   932182001  0.033881                 2        932182   \n",
       "10906159   870982002  0.061000                 2        870982   \n",
       "10906160   893427001  0.084729                 2        893427   \n",
       "10906161   893427001  0.084729                 2        893427   \n",
       "10906162   903910002  0.084729                 2        903910   \n",
       "\n",
       "          product_type_no  graphical_appearance_no  colour_group_code  \\\n",
       "0                     306                  1010016                 42   \n",
       "1                     306                  1010016                  9   \n",
       "2                     272                  1010016                  9   \n",
       "3                     264                  1010016                  9   \n",
       "4                     265                  1010016                  9   \n",
       "...                   ...                      ...                ...   \n",
       "10906158              256                  1010016                  9   \n",
       "10906159              261                  1010016                 12   \n",
       "10906160              267                  1010018                  9   \n",
       "10906161              267                  1010018                  9   \n",
       "10906162              265                  1010016                 83   \n",
       "\n",
       "          perceived_colour_value_id  perceived_colour_master_id  \\\n",
       "0                                 7                          18   \n",
       "1                                 4                           5   \n",
       "2                                 4                           5   \n",
       "3                                 4                           5   \n",
       "4                                 4                           5   \n",
       "...                             ...                         ...   \n",
       "10906158                          4                           5   \n",
       "10906159                          1                          11   \n",
       "10906160                          4                           5   \n",
       "10906161                          4                           5   \n",
       "10906162                          4                           7   \n",
       "\n",
       "          department_no  index_group_no  section_no  garment_group_no   age  \n",
       "0                  1338               1          61              1017  25.0  \n",
       "1                  1338               1          61              1017  25.0  \n",
       "2                  1722               1          15              1009  25.0  \n",
       "3                  1222               1          15              1008  25.0  \n",
       "4                  1344               2          53              1013  25.0  \n",
       "...                 ...             ...         ...               ...   ...  \n",
       "10906158           1636               1          15              1005  26.0  \n",
       "10906159           1244               2          53              1007  62.0  \n",
       "10906160           1722               1          15              1009  22.0  \n",
       "10906161           1722               1          15              1009  22.0  \n",
       "10906162           1313               1          11              1013  22.0  \n",
       "\n",
       "[10906163 rows x 16 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_merged[int_cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b3a23a02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['product_group_name', 'perceived_colour_master_name', 'index_code',\n",
       "       'club_member_status', 'fashion_news_frequency'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_merged.columns.intersection(label_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "cfd309eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# final_merged.to_csv('cleaned_h_m_dataset',encoding='utf-8',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3b47e0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "final_merged=pd.read_csv(r'cleaned_h_m_dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e2f1dbf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_lookup=pd.read_csv(r'product_label_dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6d3cbb11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_no</th>\n",
       "      <th>product_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>108775</td>\n",
       "      <td>Strap top</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110065</td>\n",
       "      <td>OP T-shirt (Idro)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>111565</td>\n",
       "      <td>20 den 1p Stockings</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>111586</td>\n",
       "      <td>Shape Up 30 den 1p Tights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>111593</td>\n",
       "      <td>Support 40 den 1p Tights</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   product_no               product_name\n",
       "0      108775                  Strap top\n",
       "1      110065          OP T-shirt (Idro)\n",
       "2      111565        20 den 1p Stockings\n",
       "3      111586  Shape Up 30 den 1p Tights\n",
       "4      111593   Support 40 den 1p Tights"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "product_lookup.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1a48c990",
   "metadata": {},
   "outputs": [],
   "source": [
    "product_lookup['product_no']=product_lookup['product_no'].astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "daace972",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t_dat</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>product_group_name</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>...</th>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <th>department_no</th>\n",
       "      <th>index_code</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>section_no</th>\n",
       "      <th>garment_group_no</th>\n",
       "      <th>club_member_status</th>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565002</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>18</td>\n",
       "      <td>13</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-01-02</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>797565001</td>\n",
       "      <td>0.030492</td>\n",
       "      <td>2</td>\n",
       "      <td>797565</td>\n",
       "      <td>306</td>\n",
       "      <td>16</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1338</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>1017</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-08-14</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>751471001</td>\n",
       "      <td>0.033881</td>\n",
       "      <td>2</td>\n",
       "      <td>751471</td>\n",
       "      <td>272</td>\n",
       "      <td>6</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1722</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1009</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-08-14</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>781613006</td>\n",
       "      <td>0.050831</td>\n",
       "      <td>2</td>\n",
       "      <td>781613</td>\n",
       "      <td>264</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1222</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1008</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-04-24</td>\n",
       "      <td>00025f8226be50dcab09402a2cacd520a99e112fe01fdd...</td>\n",
       "      <td>855823003</td>\n",
       "      <td>0.025407</td>\n",
       "      <td>2</td>\n",
       "      <td>855823</td>\n",
       "      <td>265</td>\n",
       "      <td>5</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1344</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>1013</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        t_dat                                        customer_id  article_id  \\\n",
       "0  2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   797565002   \n",
       "1  2020-01-02  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   797565001   \n",
       "2  2020-08-14  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   751471001   \n",
       "3  2020-08-14  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   781613006   \n",
       "4  2020-04-24  00025f8226be50dcab09402a2cacd520a99e112fe01fdd...   855823003   \n",
       "\n",
       "      price  sales_channel_id  product_code  product_type_no  \\\n",
       "0  0.030492                 2        797565              306   \n",
       "1  0.030492                 2        797565              306   \n",
       "2  0.033881                 2        751471              272   \n",
       "3  0.050831                 2        781613              264   \n",
       "4  0.025407                 2        855823              265   \n",
       "\n",
       "   product_group_name  graphical_appearance_no  colour_group_code  ...  \\\n",
       "0                  16                  1010016                 42  ...   \n",
       "1                  16                  1010016                  9  ...   \n",
       "2                   6                  1010016                  9  ...   \n",
       "3                   7                  1010016                  9  ...   \n",
       "4                   5                  1010016                  9  ...   \n",
       "\n",
       "   perceived_colour_master_id  perceived_colour_master_name  department_no  \\\n",
       "0                          18                            13           1338   \n",
       "1                           5                             1           1338   \n",
       "2                           5                             1           1722   \n",
       "3                           5                             1           1222   \n",
       "4                           5                             1           1344   \n",
       "\n",
       "   index_code  index_group_no  section_no  garment_group_no  \\\n",
       "0           1               1          61              1017   \n",
       "1           1               1          61              1017   \n",
       "2           0               1          15              1009   \n",
       "3           0               1          15              1008   \n",
       "4           3               2          53              1013   \n",
       "\n",
       "   club_member_status  fashion_news_frequency   age  \n",
       "0                   0                       2  25.0  \n",
       "1                   0                       2  25.0  \n",
       "2                   0                       2  25.0  \n",
       "3                   0                       2  25.0  \n",
       "4                   0                       2  25.0  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_merged.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b59c26f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2020-09-22'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_merged.t_dat.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0642d2b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train=final_merged[(final_merged['t_dat']>'2020-08-28') & (final_merged['t_dat']<'2020-09-20')].copy()\n",
    "df_test=final_merged[final_merged['t_dat']>='2020-09-20'].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8fda8a63",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2020-09-19'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.t_dat.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bfa20c46",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.drop(['t_dat'],axis=1,inplace=True)\n",
    "df_test.drop(['t_dat'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9f456139",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols=df_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f7396053",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>price</th>\n",
       "      <th>sales_channel_id</th>\n",
       "      <th>product_code</th>\n",
       "      <th>product_type_no</th>\n",
       "      <th>product_group_name</th>\n",
       "      <th>graphical_appearance_no</th>\n",
       "      <th>colour_group_code</th>\n",
       "      <th>perceived_colour_value_id</th>\n",
       "      <th>perceived_colour_master_id</th>\n",
       "      <th>perceived_colour_master_name</th>\n",
       "      <th>department_no</th>\n",
       "      <th>index_code</th>\n",
       "      <th>index_group_no</th>\n",
       "      <th>section_no</th>\n",
       "      <th>garment_group_no</th>\n",
       "      <th>club_member_status</th>\n",
       "      <th>fashion_news_frequency</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>470dac37792d1e124865d249d1e304f136c11ea46d2ec9...</td>\n",
       "      <td>794575002</td>\n",
       "      <td>0.033881</td>\n",
       "      <td>2</td>\n",
       "      <td>794575</td>\n",
       "      <td>308</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>1643</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>51</td>\n",
       "      <td>1002</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>470dac37792d1e124865d249d1e304f136c11ea46d2ec9...</td>\n",
       "      <td>877711001</td>\n",
       "      <td>0.059305</td>\n",
       "      <td>2</td>\n",
       "      <td>877711</td>\n",
       "      <td>92</td>\n",
       "      <td>12</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>3528</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>64</td>\n",
       "      <td>1020</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>470dac37792d1e124865d249d1e304f136c11ea46d2ec9...</td>\n",
       "      <td>878190004</td>\n",
       "      <td>0.067780</td>\n",
       "      <td>2</td>\n",
       "      <td>878190</td>\n",
       "      <td>262</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1201</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>1007</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>470dac37792d1e124865d249d1e304f136c11ea46d2ec9...</td>\n",
       "      <td>909014001</td>\n",
       "      <td>0.118627</td>\n",
       "      <td>2</td>\n",
       "      <td>909014</td>\n",
       "      <td>262</td>\n",
       "      <td>7</td>\n",
       "      <td>1010016</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1201</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>1007</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>470dac37792d1e124865d249d1e304f136c11ea46d2ec9...</td>\n",
       "      <td>898694001</td>\n",
       "      <td>0.050831</td>\n",
       "      <td>2</td>\n",
       "      <td>898694</td>\n",
       "      <td>264</td>\n",
       "      <td>7</td>\n",
       "      <td>1010010</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>1222</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1008</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          customer_id  article_id     price  \\\n",
       "28  470dac37792d1e124865d249d1e304f136c11ea46d2ec9...   794575002  0.033881   \n",
       "65  470dac37792d1e124865d249d1e304f136c11ea46d2ec9...   877711001  0.059305   \n",
       "72  470dac37792d1e124865d249d1e304f136c11ea46d2ec9...   878190004  0.067780   \n",
       "75  470dac37792d1e124865d249d1e304f136c11ea46d2ec9...   909014001  0.118627   \n",
       "76  470dac37792d1e124865d249d1e304f136c11ea46d2ec9...   898694001  0.050831   \n",
       "\n",
       "    sales_channel_id  product_code  product_type_no  product_group_name  \\\n",
       "28                 2        794575              308                   7   \n",
       "65                 2        877711               92                  12   \n",
       "72                 2        878190              262                   7   \n",
       "75                 2        909014              262                   7   \n",
       "76                 2        898694              264                   7   \n",
       "\n",
       "    graphical_appearance_no  colour_group_code  perceived_colour_value_id  \\\n",
       "28                  1010016                 11                          1   \n",
       "65                  1010016                  9                          4   \n",
       "72                  1010016                  9                          4   \n",
       "75                  1010016                  9                          4   \n",
       "76                  1010010                 12                          1   \n",
       "\n",
       "    perceived_colour_master_id  perceived_colour_master_name  department_no  \\\n",
       "28                           9                            16           1643   \n",
       "65                           5                             1           3528   \n",
       "72                           5                             1           1201   \n",
       "75                           5                             1           1201   \n",
       "76                          11                             0           1222   \n",
       "\n",
       "    index_code  index_group_no  section_no  garment_group_no  \\\n",
       "28           3               2          51              1002   \n",
       "65           2               1          64              1020   \n",
       "72           0               1          19              1007   \n",
       "75           0               1          19              1007   \n",
       "76           0               1          15              1008   \n",
       "\n",
       "    club_member_status  fashion_news_frequency   age  \n",
       "28                   0                       2  26.0  \n",
       "65                   0                       2  26.0  \n",
       "72                   0                       2  26.0  \n",
       "75                   0                       2  26.0  \n",
       "76                   0                       2  26.0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ac1fc465",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_series=df_train['product_code'].values.reshape((-1, 1))\n",
    "train_series_scaled=df_train['product_code'].values.reshape((-1, 1))\n",
    "test_series=df_test['product_code'].values.reshape((-1, 1))\n",
    "scaler=MinMaxScaler()\n",
    "train_series=scaler.fit_transform(train_series)\n",
    "test_series=scaler.transform(test_series)\n",
    "scaler_pred=MinMaxScaler()\n",
    "scaler_pred.fit_transform(train_series_scaled)\n",
    "train_generator = TimeseriesGenerator(data=train_series, targets=train_series,length=10,batch_size=512)\n",
    "test_generator = TimeseriesGenerator(data=test_series, targets=test_series,length=10,batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8a48f832",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "lstm_model = keras.models.Sequential([\n",
    "    keras.layers.LSTM(256, input_shape = (10, 1), return_sequences=True),\n",
    "    keras.layers.LSTM(256, return_sequences=True),\n",
    "    keras.layers.LSTM(256, return_sequences=True),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f684958a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1586/1587 [============================>.] - ETA: 0s - loss: 0.0303 - mae: 0.1225\n",
      "Epoch 1: saving model to lstm_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 88s 52ms/step - loss: 0.0303 - mae: 0.1225 - val_loss: 0.0290 - val_mae: 0.1039\n",
      "Epoch 2/5\n",
      "1586/1587 [============================>.] - ETA: 0s - loss: 0.0251 - mae: 0.1148\n",
      "Epoch 2: saving model to lstm_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 83s 52ms/step - loss: 0.0251 - mae: 0.1147 - val_loss: 0.0281 - val_mae: 0.1026\n",
      "Epoch 3/5\n",
      "1586/1587 [============================>.] - ETA: 0s - loss: 0.0249 - mae: 0.1143\n",
      "Epoch 3: saving model to lstm_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 85s 53ms/step - loss: 0.0249 - mae: 0.1143 - val_loss: 0.0283 - val_mae: 0.1029\n",
      "Epoch 4/5\n",
      "1586/1587 [============================>.] - ETA: 0s - loss: 0.0248 - mae: 0.1141\n",
      "Epoch 4: saving model to lstm_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 86s 54ms/step - loss: 0.0248 - mae: 0.1141 - val_loss: 0.0271 - val_mae: 0.1026\n",
      "Epoch 5/5\n",
      "1586/1587 [============================>.] - ETA: 0s - loss: 0.0246 - mae: 0.1138\n",
      "Epoch 5: saving model to lstm_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 86s 54ms/step - loss: 0.0246 - mae: 0.1137 - val_loss: 0.0272 - val_mae: 0.1033\n"
     ]
    }
   ],
   "source": [
    "checkpoint_path = \"lstm_model/cp.ckpt\"\n",
    "\n",
    "# Create a callback that saves the model's weights\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 verbose=1)\n",
    "es_callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)\n",
    "adam_optimizer = tf.keras.optimizers.Adam()\n",
    "lstm_model.compile(optimizer=adam_optimizer, loss = 'mean_squared_error',metrics=['mae'])\n",
    "history = lstm_model.fit(train_generator, \n",
    "                         validation_data = test_generator, \n",
    "                         epochs = 5, \n",
    "                         batch_size=512,\n",
    "                         shuffle=False,\n",
    "                         callbacks=[cp_callback,es_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e21144e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x263cf6c8b20>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_model.load_weights(checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4d4cd16d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "188/188 [==============================] - 4s 21ms/step - loss: 0.0272 - mae: 0.1033\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0271553173661232, 0.10326212644577026]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_model.evaluate(test_generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "82623968",
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_model.save(\"lstm_model.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7289ba89",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = lstm_model.predict(test_generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ec4d77d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions=predictions.reshape((-1, 1))\n",
    "prod_no_unscaled = scaler_pred.inverse_transform(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "82eaa4c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[863802.25],\n",
       "       [864562.94],\n",
       "       [868274.56],\n",
       "       ...,\n",
       "       [903815.75],\n",
       "       [904078.  ],\n",
       "       [903119.9 ]], dtype=float32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prod_no_unscaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "63f6d929",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[707269],\n",
       "       [812364],\n",
       "       [814230],\n",
       "       ...,\n",
       "       [893427],\n",
       "       [893427],\n",
       "       [903910]], dtype=int64)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test['product_code'].values.reshape((-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "007ff1ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "listoflist=prod_no_unscaled.tolist()\n",
    "flatList = [ item for elem in listoflist for item in elem]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c6c00592",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The next 5 products customer will be buying based on the recommendation system is:\n",
      "['PQ SAZ TENCEL MIX DRESS']\n",
      "['Sussie Flared Leggings']\n",
      "['MILES price s/s tee TP']\n",
      "['Coachella brazilian 2p']\n",
      "['ELLA dancesuit']\n"
     ]
    }
   ],
   "source": [
    "count=0\n",
    "product_count=5\n",
    "print('The next {} products customer will be buying based on the recommendation system is:'.format(product_count))\n",
    "for item in flatList:\n",
    "    if count<product_count:\n",
    "        product_id=round(item)\n",
    "        temp_df=product_lookup.iloc[(product_lookup['product_no']-product_id).abs().argsort()[:1]]\n",
    "        print(temp_df['product_name'].values)\n",
    "        temp_df.drop(temp_df.index, inplace=True)\n",
    "        count+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ec666b1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "lstm_tunned_model = keras.models.Sequential([\n",
    "    keras.layers.LSTM(512, input_shape = (10, 1), return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.LSTM(512, return_sequences=True,dropout=0.2),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82f1e47b",
   "metadata": {},
   "source": [
    "When used SGD with decay learning rate the loss went to infinty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f903fab2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0389 - mae: 0.1346\n",
      "Epoch 1: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 714s 438ms/step - loss: 0.0389 - mae: 0.1346 - val_loss: 0.0334 - val_mae: 0.1108\n",
      "Epoch 2/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0258 - mae: 0.1164\n",
      "Epoch 2: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 725s 457ms/step - loss: 0.0258 - mae: 0.1164 - val_loss: 0.0326 - val_mae: 0.1091\n",
      "Epoch 3/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0257 - mae: 0.1163\n",
      "Epoch 3: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 731s 461ms/step - loss: 0.0257 - mae: 0.1163 - val_loss: 0.0337 - val_mae: 0.1113\n",
      "Epoch 4/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0256 - mae: 0.1158\n",
      "Epoch 4: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 733s 462ms/step - loss: 0.0256 - mae: 0.1158 - val_loss: 0.0324 - val_mae: 0.1086\n",
      "Epoch 5/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0254 - mae: 0.1153\n",
      "Epoch 5: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 752s 474ms/step - loss: 0.0254 - mae: 0.1153 - val_loss: 0.0302 - val_mae: 0.1048\n",
      "Epoch 6/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0253 - mae: 0.1152\n",
      "Epoch 6: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 734s 463ms/step - loss: 0.0253 - mae: 0.1152 - val_loss: 0.0296 - val_mae: 0.1038\n",
      "Epoch 7/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0259 - mae: 0.1156\n",
      "Epoch 7: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 732s 461ms/step - loss: 0.0259 - mae: 0.1156 - val_loss: 0.0324 - val_mae: 0.1084\n",
      "Epoch 8/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0253 - mae: 0.1150\n",
      "Epoch 8: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 732s 461ms/step - loss: 0.0253 - mae: 0.1150 - val_loss: 0.0301 - val_mae: 0.1045\n",
      "Epoch 9/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0252 - mae: 0.1148\n",
      "Epoch 9: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 729s 459ms/step - loss: 0.0252 - mae: 0.1148 - val_loss: 0.0302 - val_mae: 0.1046\n",
      "Epoch 10/10\n",
      "1587/1587 [==============================] - ETA: 0s - loss: 0.0252 - mae: 0.1148\n",
      "Epoch 10: saving model to lstm_tunned_model\\cp.ckpt\n",
      "1587/1587 [==============================] - 731s 461ms/step - loss: 0.0252 - mae: 0.1148 - val_loss: 0.0302 - val_mae: 0.1046\n"
     ]
    }
   ],
   "source": [
    "checkpoint_path = \"lstm_tunned_model/cp.ckpt\"\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 verbose=1)\n",
    "es_callback = tf.keras.callbacks.EarlyStopping(monitor='loss',patience=3)\n",
    "adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "lstm_tunned_model.compile(optimizer=adam_optimizer, loss = 'mean_squared_error',metrics=['mae'])\n",
    "history = lstm_tunned_model.fit(train_generator, \n",
    "                         validation_data = test_generator, \n",
    "                         epochs = 10, \n",
    "                         batch_size=512,\n",
    "                         shuffle=False,\n",
    "                         callbacks=[cp_callback,es_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "405b2f1f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x23c3e63a280>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_tunned_model.load_weights(checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "befe0d2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_tunned_model.save(\"lstm_tunned_model.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "60715754",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "188/188 [==============================] - 30s 161ms/step - loss: 0.0302 - mae: 0.1046\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.03017345629632473, 0.10464475303888321]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_tunned_model.evaluate(test_generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4e7c180",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = lstm_tunned_model.predict(test_generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "36aafbae",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions=predictions.reshape((-1, 1))\n",
    "prod_no_unscaled = scaler_pred.inverse_transform(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "a71cf902",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[862389.25],\n",
       "       [865587.25],\n",
       "       [882061.44],\n",
       "       ...,\n",
       "       [884698.9 ],\n",
       "       [884698.25],\n",
       "       [884697.6 ]], dtype=float32)"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prod_no_unscaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "004f7b7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "listoflist=prod_no_unscaled.tolist()\n",
    "flatList = [ item for elem in listoflist for item in elem]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "aa52e13c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The next 5 products customer will be buying based on the recommendation system is:\n",
      "['CE Biaritz dress']\n",
      "['Simba Top']\n",
      "['Stavanger fleece set']\n",
      "['Dana double chain']\n",
      "['ISABELL BOW']\n"
     ]
    }
   ],
   "source": [
    "count=0\n",
    "product_count=5\n",
    "print('The next {} products customer will be buying based on the recommendation system is:'.format(product_count))\n",
    "for item in flatList:\n",
    "    if count<product_count:\n",
    "        product_id=round(item)\n",
    "        temp_df=product_lookup.iloc[(product_lookup['product_no']-product_id).abs().argsort()[:1]]\n",
    "        print(temp_df['product_name'].values)\n",
    "        temp_df.drop(temp_df.index, inplace=True)\n",
    "        count+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e122042a",
   "metadata": {},
   "source": [
    "#### Tensorflow recommonder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5bd15c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip3 install pyspark\n",
    "#!pip3 install tensorflow-recommenders\n",
    "#!pip3 install -q scann"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18562265",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd \n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "#import spark\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a618766",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql import SQLContext\n",
    "from pyspark.sql import functions as F\n",
    "from pyspark.sql import Window\n",
    "from pyspark.sql.types import StructType,StructField, StringType, IntegerType, ArrayType, DoubleType, BooleanType"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "828f428a",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None, \"display.max_columns\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "906ed9d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Pandas\n",
    "articlesDF = pd.read_csv('articles.csv')\n",
    "customersDF = pd.read_csv('customers.csv')\n",
    "transactions_trainDF = pd.read_csv('transactions_train.csv')\n",
    "'''\n",
    "PySpark\n",
    "articles = spark.read.option('header', True).csv('articles.csv')\n",
    "customer = spark.read.option('header', True).csv('customers.csv')\n",
    "transactions_train = spark.read.option('header', True).csv('transactions_train.csv')\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2690f8b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF.shape , customersDF.shape, transactions_trainDF.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d92e5cb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b614201b",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84b8c1b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF = articlesDF.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "766a22aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b8929ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1698db03",
   "metadata": {},
   "outputs": [],
   "source": [
    "articlesDF = articlesDF[['article_id', 'product_code', 'prod_name', 'product_type_no','department_no', 'department_name', 'detail_desc']]\n",
    "articlesDF.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c2706c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_trainDF['t_dat'].max() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4d50f45",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_trainDF = transactions_trainDF[transactions_trainDF['t_dat'] >= '2020-01-01']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef448b7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_trainDF.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "791220d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_trainDF.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7535cf39",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_trainDF.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11cca113",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = transactions_trainDF.groupby(['customer_id','article_id'])['t_dat'].agg('count').reset_index()\n",
    "tmp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d80c778",
   "metadata": {},
   "source": [
    "### Merge Customer and Transactions to then clean the data based on Customer's transactions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c014a02",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF = pd.merge(transactions_trainDF, customersDF, on='customer_id', how='left', suffixes=('', '_remove'))\n",
    "customerTransactionsDF.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a293e168",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ff4c6bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81240132",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF['FN'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea9b6e86",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF = customerTransactionsDF.drop(['FN', 'club_member_status', 'sales_channel_id','fashion_news_frequency', 'postal_code'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a50331ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0e14562",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba9cb366",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF['Active'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcdd57f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF['Active'].fillna(value=0.0, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6a59ffe",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c2ee75d",
   "metadata": {},
   "outputs": [],
   "source": [
    "customerTransactionsDF.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d8e3a13",
   "metadata": {},
   "outputs": [],
   "source": [
    "HMRecommendationDF = pd.merge(customerTransactionsDF, articlesDF, on='article_id', how='left', suffixes=('','_remove'))\n",
    "HMRecommendationDF.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87e86989",
   "metadata": {},
   "outputs": [],
   "source": [
    "HMRecommendationDF.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12e25691",
   "metadata": {},
   "outputs": [],
   "source": [
    "HMRecommendationDF['t_dat'] = pd.to_datetime(HMRecommendationDF['t_dat'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d7e83b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For improvments refer to https://www.tensorflow.org/recommenders\n",
    "import tensorflow_recommenders as tfrs\n",
    "\n",
    "from pathlib import Path\n",
    "from typing import Dict, Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf5c2dba",
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_customer_ids = HMRecommendationDF['customer_id'].unique()\n",
    "unique_article_ids = HMRecommendationDF['article_id'].unique() \n",
    "\n",
    "article_ds = tf.data.Dataset.from_tensor_slices(dict(HMRecommendationDF[['article_id']]))\n",
    "articles = article_ds.map(lambda x: x['article_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c30cd39e",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_dimension = 64\n",
    "\n",
    "# Customer ID\n",
    "customer_model = tf.keras.Sequential([\n",
    "  tf.keras.layers.StringLookup(\n",
    "      vocabulary=unique_customer_ids, mask_token=None),  \n",
    "  tf.keras.layers.Embedding(len(unique_customer_ids) + 1, embedding_dimension)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27b5edc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "HMRecommendationDF['article_id'] = HMRecommendationDF['article_id'].astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "750a9d17",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Article ID\n",
    "article_model = tf.keras.Sequential([\n",
    "  tf.keras.layers.StringLookup(\n",
    "      vocabulary=unique_article_ids, mask_token=None),\n",
    "  tf.keras.layers.Embedding(len(unique_article_ids) + 1, embedding_dimension)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f71203d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class HMRModel(tfrs.Model):\n",
    "    \n",
    "    def __init__(self, customer_model, article_model):\n",
    "        super().__init__()\n",
    "        self.article_model: tf.keras.Model = article_model\n",
    "        self.customer_model: tf.keras.Model = customer_model\n",
    "        self.task = tfrs.tasks.Retrieval(\n",
    "        metrics=tfrs.metrics.FactorizedTopK(\n",
    "            candidates=articles.batch(128).map(self.article_model),            \n",
    "            ),\n",
    "        )        \n",
    "\n",
    "    def compute_loss(self, features: Dict[str, tf.Tensor], training=False) -> tf.Tensor:\n",
    "    \n",
    "        customer_embeddings = self.customer_model(features[\"customer_id\"])    \n",
    "        article_embeddings = self.article_model(features[\"article_id\"])\n",
    "\n",
    "        # Returns loss and the metrics of the model \n",
    "        return self.task(customer_embeddings, article_embeddings,compute_metrics=not training)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ee68d87",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = HMRModel(customer_model, article_model)\n",
    "model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efe698bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splittng the data\n",
    "HMRecommendationDF['t_dat'].max() , HMRecommendationDF['t_dat'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39a367a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b96621ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 70% training and 30% Testing\n",
    "train_temp=HMRecommendationDF.sample(frac=0.7,random_state=200) \n",
    "test_temp=HMRecommendationDF.drop(train_temp.index)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c61f07ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = tf.data.Dataset.from_tensor_slices(dict(train_temp[['customer_id','article_id']])).shuffle(100_000).batch(256).cache()\n",
    "test_data = tf.data.Dataset.from_tensor_slices(dict(test_temp[['customer_id','article_id']])).batch(256).cache()\n",
    "\n",
    "num_epochs = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e91250fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train_data, epochs=num_epochs, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ec3cd6",
   "metadata": {},
   "source": [
    "#### RankItem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d44b90c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "from tqdm import tqdm\n",
    "from abc import ABC, abstractmethod\n",
    "from typing import List, Dict, Any, Union"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07b44fc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_train = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/transactions_train.csv')\n",
    "transactions_train['t_dat'] = pd.to_datetime(transactions_train['t_dat'])\n",
    "customers_df = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/customers.csv')\n",
    "articles_df = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/articles.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c39b282",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ItemFeatures(ABC):\n",
    "    @abstractmethod\n",
    "    def get(self, *args, **kwargs) -> pd.DataFrame:\n",
    "        \"\"\"\n",
    "        article_id -> features\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fd15d98",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CategoryTransform(ItemFeatures):\n",
    "    \"\"\"\n",
    "    factorize all articles columns\n",
    "    \"\"\"\n",
    "    def __init__(self, articles_df: pd.DataFrame):\n",
    "        self.articles_df = articles_df\n",
    "\n",
    "    def get(self):\n",
    "        self.__feature_columns = list(filter(lambda x: 'name' in x, self.articles_df.columns))[1:]\n",
    "        filtered_articles = self.articles_df[self.__feature_columns]\n",
    "        filtered_articles = filtered_articles.apply(lambda x: pd.factorize(x)[0])\n",
    "        filtered_articles['article_id'] = self.articles_df['article_id']\n",
    "\n",
    "        features = filtered_articles.set_index('article_id').astype('int8')\n",
    "        return features\n",
    "\n",
    "    def get_columns(self):\n",
    "        return self.__feature_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7268f793",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AggrTransform(ItemFeatures):\n",
    "    \"\"\"\n",
    "    aggregation transactions features : mean, max and etc...\n",
    "    \"\"\"\n",
    "    def __init__(self, articles_df: pd.DataFrame, transactions_df: pd.DataFrame):\n",
    "        self.articles_df = articles_df\n",
    "        self.transactions_df = transactions_df\n",
    "\n",
    "    def get(self):\n",
    "        stats = self._get_stats()\n",
    "        return stats\n",
    "\n",
    "    def _get_stats(self):\n",
    "        transactions_more = self.transactions_df.merge(self.articles_df, on = ('article_id'))\n",
    "        grouped = (\n",
    "            transactions_more.\n",
    "            groupby('article_id')\n",
    "        )\n",
    "\n",
    "        counts = (\n",
    "            grouped['article_id']\n",
    "            .count()\n",
    "            .to_frame()\n",
    "            .rename(columns = {'article_id': 'count'})\n",
    "            .astype('int16')\n",
    "            .reset_index()\n",
    "            .set_index('article_id')\n",
    "        )\n",
    "        sums = (\n",
    "            grouped['price']\n",
    "            .sum()\n",
    "            .to_frame()\n",
    "            .astype('float32')\n",
    "            .rename(columns = {\n",
    "                'price': 'sum_price'\n",
    "            })\n",
    "        )\n",
    "        means = (\n",
    "            grouped['price']\n",
    "            .mean()\n",
    "            .to_frame()\n",
    "            .astype('float32')\n",
    "            .rename(columns = {\n",
    "                'price': 'mean_price'\n",
    "            })\n",
    "        )\n",
    "        mins = (\n",
    "            grouped['price']\n",
    "            .min()\n",
    "            .to_frame()\n",
    "            .astype('float32')\n",
    "            .rename(columns = {\n",
    "               'price': 'min_price' \n",
    "            })\n",
    "        )\n",
    "        maxs = (\n",
    "            grouped['price']\n",
    "            .max()\n",
    "            .to_frame()\n",
    "            .astype('float32')\n",
    "            .rename(columns = {\n",
    "                'price': 'max_price'\n",
    "            })\n",
    "        )\n",
    "        \n",
    "        output_df = (\n",
    "            counts\n",
    "            .merge(sums, on = ('article_id'))\n",
    "            .merge(means, on = ('article_id'))\n",
    "            .merge(mins, on = ('article_id'))\n",
    "            .merge(maxs, on = ('article_id'))\n",
    "        )\n",
    "        return output_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0430bcc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TopTransforms(ItemFeatures):\n",
    "    \"\"\"\n",
    "    whether category appears in top categories\n",
    "    \"\"\"\n",
    "    def __init__(self, articles_df: pd.DataFrame, topk = 3):\n",
    "        self.articles_df = articles_df\n",
    "        self.topk = topk\n",
    "    \n",
    "    def get(self):\n",
    "        name_cols = list(filter(lambda x: 'name' in x, self.articles_df.columns))  \n",
    "        \n",
    "        value_counts = self._get_value_counts(name_cols)\n",
    "        value_counts = {\n",
    "            f'{k}_{self.topk}': self.articles_df[k].isin(v).astype('int8') for k, v in value_counts.items()\n",
    "        }\n",
    "        \n",
    "        output_df = self.articles_df.assign(**value_counts)\n",
    "        output_df = output_df[['article_id'] + list(value_counts.keys())].set_index('article_id')\n",
    "        return output_df\n",
    "        \n",
    "    def _get_value_counts(self, name_cols: List[str]):\n",
    "        value_counts = self.articles_df[name_cols].apply(pd.Series.value_counts)\n",
    "        get_index = lambda x: value_counts.sort_values(x, ascending = False)[x][:self.topk].index  \n",
    "        value_counts = dict(zip(name_cols, map(lambda x: get_index(x), name_cols)))\n",
    "        return value_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a038dae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ItemFeaturesCollector:\n",
    "    @staticmethod\n",
    "    def collect(features: Union[List[ItemFeatures], List[str]], **kwargs) -> pd.DataFrame:\n",
    "        output_df = None\n",
    "\n",
    "        for feature in tqdm(features):\n",
    "            if isinstance(feature, ItemFeatures):\n",
    "                feature_out = feature.get(**kwargs)\n",
    "            if isinstance(feature, str):\n",
    "                try:\n",
    "                    feature_out = pd.read_csv(feature)\n",
    "                except:\n",
    "                    feature_out = pd.read_parquet(feature)\n",
    "\n",
    "            if output_df is None:\n",
    "                output_df = feature_out\n",
    "            else:\n",
    "                output_df = output_df.merge(feature_out, on = ('article_id'))\n",
    "        return output_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aab19450",
   "metadata": {},
   "outputs": [],
   "source": [
    "item_features = ItemFeaturesCollector.collect([\n",
    "    CategoryTransform(articles_df),\n",
    "    AggrTransform(articles_df, transactions_train.iloc[:100_000]),\n",
    "    TopTransforms(articles_df)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f8e0ff0",
   "metadata": {},
   "outputs": [],
   "source": [
    "item_features.to_parquet('item_features.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc2600a",
   "metadata": {},
   "source": [
    "### RankUser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fc11f19",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from abc import ABC, abstractmethod\n",
    "from pathlib import Path\n",
    "from tqdm import tqdm\n",
    "import pickle\n",
    "from collections import defaultdict\n",
    "from typing import List, Dict, Any, Union"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6dbb9d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_train = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/transactions_train.csv')\n",
    "transactions_train['t_dat'] = pd.to_datetime(transactions_train['t_dat'])\n",
    "customers_df = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/customers.csv')\n",
    "articles_df = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/articles.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bf9468c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class UserFeatures(ABC):\n",
    "    @abstractmethod\n",
    "    def get(self) -> pd.DataFrame:\n",
    "        \"\"\"\n",
    "        customer_id -> features\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd4eae4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AggrFeatures(UserFeatures):\n",
    "    \"\"\"\n",
    "    basic aggregation features(min, max, mean and etc...)\n",
    "    \"\"\"\n",
    "    def __init__(self, transactions_df):\n",
    "        self.groupby_df = transactions_df.groupby('customer_id', as_index = False)\n",
    "\n",
    "    def get(self):\n",
    "        output_df = (\n",
    "            self.groupby_df['price']\n",
    "            .agg({\n",
    "                'mean_transactions': 'mean',\n",
    "                'max_transactions': 'max',\n",
    "                'min_transactions': 'min',\n",
    "                'median_transactions': 'median',\n",
    "                'sum_transactions': 'sum',\n",
    "                'max_minus_min_transactions': lambda x: x.max()-x.min()\n",
    "            })\n",
    "            .set_index('customer_id')\n",
    "            .astype('float32')\n",
    "        )\n",
    "        return output_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98a184a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CountFeatures(UserFeatures):\n",
    "    \"\"\"\n",
    "    basic features connected with transactions\n",
    "    \"\"\"\n",
    "    def __init__(self, transactions_df, topk = 10):\n",
    "        self.transactions_df = transactions_df\n",
    "        self.topk = topk\n",
    "\n",
    "    def get(self):\n",
    "        grouped = self.transactions_df.groupby('customer_id', as_index = False)\n",
    "        #number of transactions, number of online articles,\n",
    "        #number of transactions bigger than mean price of transactions\n",
    "        a = (\n",
    "            grouped\n",
    "            .agg({\n",
    "                'article_id': 'count',\n",
    "                'price': lambda x: sum(np.array(x) > x.mean()),\n",
    "                'sales_channel_id': lambda x: sum(x == 2),\n",
    "            })\n",
    "            .rename(columns = {\n",
    "                'article_id': 'n_transactions',\n",
    "                'price': 'n_transactions_bigger_mean',\n",
    "                'sales_channel_id': 'n_online_articles'\n",
    "            })\n",
    "            .set_index('customer_id')\n",
    "            .astype('int8')\n",
    "        )\n",
    "        #number of unique articles, number of store articles\n",
    "        b = (\n",
    "            grouped\n",
    "            .agg({\n",
    "                'article_id': 'nunique',\n",
    "                'sales_channel_id': lambda x: sum(x == 1),\n",
    "            })\n",
    "            .rename(columns = {\n",
    "                'article_id': 'n_unique_articles',\n",
    "                'sales_channel_id': 'n_store_articles',\n",
    "            })\n",
    "            .set_index('customer_id')\n",
    "            .astype('int8')\n",
    "        )\n",
    "        #number of transactions that are in top\n",
    "        topk_articles = self.transactions_df['article_id'].value_counts()[:self.topk].index\n",
    "        c = (\n",
    "            grouped['article_id']\n",
    "            .agg({\n",
    "               f'top_article_{i}':  lambda x: sum(x == k) for i, k in enumerate(topk_articles)\n",
    "            }\n",
    "            )\n",
    "            .set_index('customer_id')\n",
    "            .astype('int8')\n",
    "        )\n",
    "        \n",
    "        output_df = a.merge(b, on = ('customer_id')).merge(c, on = ('customer_id'))\n",
    "        return output_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba876da6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomerFeatures(UserFeatures):\n",
    "    \"\"\"\n",
    "    All columns from customers dataframe\n",
    "    \"\"\"\n",
    "    def __init__(self, customers_df):\n",
    "        self.customers_df = self._prepare_customers(customers_df)\n",
    "    \n",
    "    def _prepare_customers(self, customers_df):\n",
    "        customers_df['FN'] = customers_df['FN'].fillna(0).astype('int8')\n",
    "        customers_df['Active'] = customers_df['Active'].fillna(0).astype('int8')\n",
    "        customers_df['club_member_status'] = customers_df['club_member_status'].fillna('UNKNOWN')\n",
    "        customers_df['age'] = customers_df['age'].fillna(customers_df['age'].mean()).astype('int8')\n",
    "        customers_df['fashion_news_frequency'] = (\n",
    "            customers_df['fashion_news_frequency']\n",
    "            .replace('None', 'NONE')\n",
    "            .replace(np.nan, 'NONE')\n",
    "        )\n",
    "        return customers_df\n",
    "\n",
    "    def get(self):\n",
    "        output = (\n",
    "            self.customers_df[filter(lambda x: x != 'postal_code', customers_df.columns)]\n",
    "            .set_index('customer_id')\n",
    "        )\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d5ccbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ArticlesFeatures(UserFeatures):\n",
    "    \"\"\"\n",
    "    returns article features: whether category appears in top categories\n",
    "    \"\"\"\n",
    "    def __init__(self, transactions_df, articles, topk = 10):\n",
    "        self.merged_df = transactions_df.merge(articles, on = ('article_id'))\n",
    "        self.articles = articles\n",
    "        self.topk = topk\n",
    "    \n",
    "    def get(self):\n",
    "        output_df = None\n",
    "\n",
    "        for col in tqdm(self.articles.columns, desc = 'extracting features'):\n",
    "            if 'name' in col:\n",
    "                if output_df is None:\n",
    "                    output_df = self.aggregate_topk(self.merged_df, col, self.topk)\n",
    "                else:\n",
    "                    intermediate_out = self.aggregate_topk(self.merged_df, col, self.topk)\n",
    "                    output_df = output_df.merge(intermediate_out, on = ('customer_id'))\n",
    "        return output_df\n",
    "\n",
    "    def return_value_counts(self, df, column_name, k):\n",
    "        value_counts = df[column_name].value_counts()[:k].index\n",
    "        value_counts = list(map(lambda x: x[1], value_counts))\n",
    "        return value_counts\n",
    "\n",
    "    def aggregate_topk(self, merged_df, column_name, k):\n",
    "        grouped_df_indx = merged_df.groupby('customer_id')\n",
    "        grouped_df = merged_df.groupby('customer_id', as_index = False)\n",
    "        \n",
    "        topk_values = self.return_value_counts(grouped_df_indx, column_name, k)\n",
    "        #how many transactions appears in top category(column)\n",
    "        n_top_k = (\n",
    "            grouped_df[column_name]\n",
    "            .agg({\n",
    "                f'top_{column_name}_{i}': lambda x: sum(x == k) for i, k in enumerate(topk_values)\n",
    "            })\n",
    "            .set_index('customer_id')\n",
    "            .astype('int16')\n",
    "        )\n",
    "        return n_top_k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0adf01b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class UserFeaturesCollector:\n",
    "    \"\"\"\n",
    "    collect all features and aggregate them\n",
    "    \"\"\"\n",
    "    @staticmethod\n",
    "    def collect(features: Union[List[UserFeatures], List[str]], **kwargs) -> pd.DataFrame:\n",
    "        output_df = None\n",
    "\n",
    "        for feature in tqdm(features):\n",
    "            if isinstance(feature, UserFeatures):\n",
    "                feature_out = feature.get(**kwargs)\n",
    "            if isinstance(feature, str):\n",
    "                try:\n",
    "                    feature_out = pd.read_csv(feature)\n",
    "                except:\n",
    "                    feature_out = pd.read_parquet(feature)\n",
    "\n",
    "            if output_df is None:\n",
    "                output_df = feature_out\n",
    "            else:\n",
    "                output_df = output_df.merge(feature_out, on = ('customer_id'))\n",
    "        return output_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36b2d8dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_features = UserFeaturesCollector.collect([\n",
    "    AggrFeatures(transactions_train.iloc[:10_000]),\n",
    "    CountFeatures(transactions_train.iloc[:10_000], 3),\n",
    "    CustomerFeatures(customers_df),\n",
    "    ArticlesFeatures(transactions_train.iloc[:10_000], articles_df, 3),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2d8c5c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_features.to_parquet('user_features.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "048ead3d",
   "metadata": {},
   "source": [
    "### LGBM Ranker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a61dd0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lightgbm.sklearn import LGBMRanker\n",
    "from datetime import timedelta\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2146fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_features = pd.read_parquet('C:/Users/smexy/Documents/Datasets/CS677/user_features.parquet')\n",
    "item_features = pd.read_parquet('C:/Users/smexy/Documents/Datasets/CS677/item_features.parquet')\n",
    "transactions_df = pd.read_csv('C:/Users/smexy/Documents/Datasets/CS677/transactions_train.csv')\n",
    "transactions_df.t_dat = pd.to_datetime( transactions_df.t_dat )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a66106c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_4w = transactions_df[transactions_df['t_dat'] >= pd.to_datetime('2020-08-24')].copy()\n",
    "df_3w = transactions_df[transactions_df['t_dat'] >= pd.to_datetime('2020-08-31')].copy()\n",
    "df_2w = transactions_df[transactions_df['t_dat'] >= pd.to_datetime('2020-09-07')].copy()\n",
    "df_1w = transactions_df[transactions_df['t_dat'] >= pd.to_datetime('2020-09-15')].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf6eca82",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_features[['club_member_status', 'fashion_news_frequency']] = (\n",
    "                   user_features[['club_member_status', 'fashion_news_frequency']]\n",
    "                   .apply(lambda x: pd.factorize(x)[0])\n",
    ").astype('int8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdbca760",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions_df = (\n",
    "    transactions_df\n",
    "    .merge(user_features, on = ('customer_id'))\n",
    "    .merge(item_features, on = ('article_id'))\n",
    ")\n",
    "transactions_df.sort_values(['t_dat', 'customer_id'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d154ac8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#for simplicity let's take only 1M rows\n",
    "N_ROWS = 1_000_000\n",
    "\n",
    "train = transactions_df.loc[ transactions_df.t_dat <= pd.to_datetime('2020-09-15') ].iloc[:N_ROWS]\n",
    "valid = transactions_df.loc[ transactions_df.t_dat >= pd.to_datetime('2020-09-16') ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35090724",
   "metadata": {},
   "outputs": [],
   "source": [
    "del transactions_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2333e10",
   "metadata": {},
   "outputs": [],
   "source": [
    "purchase_dict_4w = {}\n",
    "\n",
    "for i,x in enumerate(zip(df_4w['customer_id'], df_4w['article_id'])):\n",
    "    cust_id, art_id = x\n",
    "    if cust_id not in purchase_dict_4w:\n",
    "        purchase_dict_4w[cust_id] = {}\n",
    "    \n",
    "    if art_id not in purchase_dict_4w[cust_id]:\n",
    "        purchase_dict_4w[cust_id][art_id] = 0\n",
    "    \n",
    "    purchase_dict_4w[cust_id][art_id] += 1\n",
    "\n",
    "dummy_list_4w = list((df_4w['article_id'].value_counts()).index)[:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a13f622",
   "metadata": {},
   "outputs": [],
   "source": [
    "purchase_dict_3w = {}\n",
    "\n",
    "for i,x in enumerate(zip(df_3w['customer_id'], df_3w['article_id'])):\n",
    "    cust_id, art_id = x\n",
    "    if cust_id not in purchase_dict_3w:\n",
    "        purchase_dict_3w[cust_id] = {}\n",
    "    \n",
    "    if art_id not in purchase_dict_3w[cust_id]:\n",
    "        purchase_dict_3w[cust_id][art_id] = 0\n",
    "    \n",
    "    purchase_dict_3w[cust_id][art_id] += 1\n",
    "\n",
    "dummy_list_3w = list((df_3w['article_id'].value_counts()).index)[:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8ecb391",
   "metadata": {},
   "outputs": [],
   "source": [
    "purchase_dict_2w = {}\n",
    "\n",
    "for i,x in enumerate(zip(df_2w['customer_id'], df_2w['article_id'])):\n",
    "    cust_id, art_id = x\n",
    "    if cust_id not in purchase_dict_2w:\n",
    "        purchase_dict_2w[cust_id] = {}\n",
    "    \n",
    "    if art_id not in purchase_dict_2w[cust_id]:\n",
    "        purchase_dict_2w[cust_id][art_id] = 0\n",
    "    \n",
    "    purchase_dict_2w[cust_id][art_id] += 1\n",
    "\n",
    "dummy_list_2w = list((df_2w['article_id'].value_counts()).index)[:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53af972f",
   "metadata": {},
   "outputs": [],
   "source": [
    "purchase_dict_1w = {}\n",
    "\n",
    "for i,x in enumerate(zip(df_1w['customer_id'], df_1w['article_id'])):\n",
    "    cust_id, art_id = x\n",
    "    if cust_id not in purchase_dict_1w:\n",
    "        purchase_dict_1w[cust_id] = {}\n",
    "    \n",
    "    if art_id not in purchase_dict_1w[cust_id]:\n",
    "        purchase_dict_1w[cust_id][art_id] = 0\n",
    "    \n",
    "    purchase_dict_1w[cust_id][art_id] += 1\n",
    "\n",
    "dummy_list_1w = list((df_1w['article_id'].value_counts()).index)[:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17720b47",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_candidates(customers_id, n_candidates = 12):\n",
    "  \"\"\"\n",
    "  df - basically, dataframe with customers(customers should be unique)\n",
    "  \"\"\"\n",
    "  prediction_dict = {}\n",
    "  dummy_list = list((df_2w['article_id'].value_counts()).index)[:n_candidates]\n",
    "\n",
    "  for i, cust_id in tqdm(enumerate(customers_id)):\n",
    "    # comment this for validation\n",
    "    if cust_id in purchase_dict_1w:\n",
    "        l = sorted((purchase_dict_1w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n",
    "        l = [y[0] for y in l]\n",
    "        if len(l)>n_candidates:\n",
    "            s = l[:n_candidates]\n",
    "        else:\n",
    "            s = l+dummy_list_1w[:(n_candidates-len(l))]\n",
    "    elif cust_id in purchase_dict_2w:\n",
    "        l = sorted((purchase_dict_2w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n",
    "        l = [y[0] for y in l]\n",
    "        if len(l)>n_candidates:\n",
    "            s = l[:n_candidates]\n",
    "        else:\n",
    "            s = l+dummy_list_2w[:(n_candidates-len(l))]\n",
    "    elif cust_id in purchase_dict_3w:\n",
    "        l = sorted((purchase_dict_3w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n",
    "        l = [y[0] for y in l]\n",
    "        if len(l)>n_candidates:\n",
    "            s = l[:n_candidates]\n",
    "        else:\n",
    "            s = l+dummy_list_3w[:(n_candidates-len(l))]\n",
    "    elif cust_id in purchase_dict_4w:\n",
    "        l = sorted((purchase_dict_4w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n",
    "        l = [y[0] for y in l]\n",
    "        if len(l)>n_candidates:\n",
    "            s = l[:n_candidates]\n",
    "        else:\n",
    "            s = l+dummy_list_4w[:(n_candidates-len(l))]\n",
    "    else:\n",
    "        s = dummy_list\n",
    "    prediction_dict[cust_id] = s\n",
    "\n",
    "  k = list(map(lambda x: x[0], prediction_dict.items()))\n",
    "  v = list(map(lambda x: x[1], prediction_dict.items()))\n",
    "  negatives_df = pd.DataFrame({'customer_id': k, 'negatives': v})\n",
    "  negatives_df = (\n",
    "      negatives_df\n",
    "      .explode('negatives')\n",
    "      .rename(columns = {'negatives': 'article_id'})\n",
    "  )\n",
    "  return negatives_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8df13ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "#take only last 15 transactions\n",
    "train['rank'] = range(len(train))\n",
    "train = (\n",
    "    train\n",
    "    .assign(\n",
    "        rn = train.groupby(['customer_id'])['rank']\n",
    "                  .rank(method='first', ascending=False))\n",
    "    .query(\"rn <= 15\")\n",
    "    .drop(columns = ['price', 'sales_channel_id'])\n",
    "    .sort_values(['t_dat', 'customer_id'])\n",
    ")\n",
    "train['label'] = 1\n",
    "\n",
    "del train['rank']\n",
    "del train['rn']\n",
    "\n",
    "valid.sort_values(['t_dat', 'customer_id'], inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19114f25",
   "metadata": {},
   "outputs": [],
   "source": [
    "last_dates = (\n",
    "    train\n",
    "    .groupby('customer_id')['t_dat']\n",
    "    .max()\n",
    "    .to_dict()\n",
    ")\n",
    "\n",
    "negatives = prepare_candidates(train['customer_id'].unique(), 15)\n",
    "negatives['t_dat'] = negatives['customer_id'].map(last_dates)\n",
    "\n",
    "negatives = (\n",
    "    negatives\n",
    "    .merge(user_features, on = ('customer_id'))\n",
    "    .merge(item_features, on = ('article_id'))\n",
    ")\n",
    "negatives['label'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d30a938d",
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.concat([train, negatives])\n",
    "train.sort_values(['customer_id', 't_dat'], inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05e05378",
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_baskets = train.groupby(['customer_id'])['article_id'].count().values\n",
    "# valid_baskets = valid.groupby(['customer_id'])['article_id'].count().values\n",
    "train_baskets = train.groupby(['customer_id'])['article_id'].count().values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bda9c77b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ranker = LGBMRanker(\n",
    "    objective=\"lambdarank\",\n",
    "    metric=\"ndcg\",\n",
    "    boosting_type=\"dart\",\n",
    "    max_depth=7,\n",
    "    n_estimators=300,\n",
    "    importance_type='gain',\n",
    "    verbose=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bb37cf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "ranker = ranker.fit(\n",
    "    train.drop(columns = ['t_dat', 'customer_id', 'article_id', 'label']),\n",
    "    train.pop('label'),\n",
    "    group=train_baskets,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1db47b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "candidates = prepare_candidates(sample_sub.customer_id.unique(), 12)\n",
    "candidates = (\n",
    "    candidates\n",
    "    .merge(user_features, on = ('customer_id'))\n",
    "    .merge(item_features, on = ('article_id'))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "183428d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = []\n",
    "batch_size = 1_000_000\n",
    "for bucket in tqdm(range(0, len(candidates), batch_size)):\n",
    "  outputs = ranker.predict(\n",
    "      candidates.iloc[bucket: bucket+batch_size]\n",
    "      .drop(columns = ['customer_id', 'article_id'])\n",
    "      )\n",
    "  preds.append(outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85e8cef3",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = np.concatenate(preds)\n",
    "candidates['preds'] = preds\n",
    "preds = candidates[['customer_id', 'article_id', 'preds']]\n",
    "preds.sort_values(['customer_id', 'preds'], ascending=False, inplace = True)\n",
    "preds = (\n",
    "    preds\n",
    "    .groupby('customer_id')[['article_id']]\n",
    "    .aggregate(lambda x: x.tolist())\n",
    ")\n",
    "preds['article_id'] = preds['article_id'].apply(lambda x: ' '.join(['0'+str(k) for k in x]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81f6f323",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = sample_sub[['customer_id']].merge(\n",
    "    preds\n",
    "    .reset_index()\n",
    "    .rename(columns = {'article_id': 'prediction'}), how = 'left')\n",
    "preds['prediction'].fillna(' '.join(['0'+str(art) for art in dummy_list_2w]), inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20cfe44e",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds.to_csv('predictionsheet.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09975b1f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pacmanenv",
   "language": "python",
   "name": "pacmanenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}