InitialAnalysisTemplate_JobAid.ipynb

{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":5,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install numpy xlrd pandas matplotlib seaborn sklearn","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"0201256b-c665-4a80-bb0c-e2885a22780d"},{"cell_type":"code","source":"#Import your Libraries\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport sklearn.metrics as metrics\n%matplotlib inline","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"2dac3d66-b2d4-49df-8540-82d05eeff307"},{"cell_type":"code","source":"# %%timeit -n 1\n# Load your data  -- start with CreditScoring.csv... then online retail\ndf = pd.read_csv('./data/REPLACE_WITH_YOUR_FILE')\n#  you can also pull from urls like this:   \n# df = pd.read_csv('https://raw.githubusercontent.com/fenago/MLEssentials/main/datasets/Life%20Expectancy%20Data.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"c21f8a1a-a9d5-4277-868f-bef4e51785a9"},{"cell_type":"markdown","source":"## Notes\n\nThis session covers data collection and some procedures of data preparation. \n\n**Commands, functions, and methods:** \n\n* `!wget` - Linux shell command for downloading data \n* `pd.read.csv()` - read csv files \n* `df.head()` - take a look of the dataframe \n* `df.head().T` - take a look of the transposed dataframe \n* `df.columns` - retrieve column names of a dataframe \n* `df.columns.str.lower()` - lowercase all the letters \n* `df.columns.str.replace(' ', '_')` - replace the space separator \n* `df.dtypes` - retrieve data types of all series \n* `df.index` - retrive indices of a dataframe\n* `pd.to_numeric()` - convert a series values to numerical values. The `errors=coerce` argument allows making the transformation despite some encountered errors. \n* `df.fillna()` - replace NAs with some value \n* `(df.x == \"yes\").astype(int)` - convert x series of yes-no values to numerical values.\n* `df['Weight'] = df['Weight'].astype(int)` - this takes a single column of data and converts the data type","metadata":{},"id":"ecdea6a4-8fb0-428d-bf36-ac708509cc9c"},{"cell_type":"code","source":"len(df)","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"823b6b3a-c279-4b3e-9b75-75ab01f8ed0b"},{"cell_type":"code","source":"df.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"dfdc07fd-82d6-411c-bf4e-4012315538e1"},{"cell_type":"code","source":"df.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"4f5bc2f4-a159-4a24-b02d-cf234b7daefa"},{"cell_type":"code","source":"df.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"b8057edf-cf08-4bfe-aa36-f56b649f6500"},{"cell_type":"code","source":"df.nunique()","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"960afc07-cb45-4851-aa9b-6a564d943273"},{"cell_type":"code","source":"df.corr()","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"17fa6add-bdac-4d33-8b96-e102f3d13bc5"},{"cell_type":"code","source":"# Basic Data Cleaning\ndf.columns = df.columns.str.lower().str.replace(' ', '_') # A\n \nstring_columns = list(df.dtypes[df.dtypes == 'object'].index) # B\n \nfor col in string_columns:\n    df[col] = df[col].str.lower().str.replace(' ', '_') # C","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"03c95958-f524-486f-84dd-62eccffe2985"},{"cell_type":"code","source":"# MAKE SURE THAT YOU WRANGLE YOUR DATA.  THIS IS AN EXAMPLE OF THE TYPES OF THINGS THAT ARE NEEDED\n# SKIP THIS CEL - IT IS ONLY TO REITERATE THE NEED TO CLEAN \n# For instance - in the CreditScoring dataset - there are numerous 99999999 that need to be replaced\n# Obviously don't run this with your dataset\n# for c in ['income', 'assets', 'debt']:\n#    df[c] = df[c].replace(to_replace=99999999, value=np.nan)\n#df = df[df.status != 'unk']   # Also make sure to treat the target variable","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"2cbc6fd3-13ac-4947-b727-b15f7cd74520"},{"cell_type":"code","source":"df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"5694febc-4570-4e82-a40a-4e347e468eb6"},{"cell_type":"code","source":"df.head().T","metadata":{"trusted":true},"execution_count":null,"outputs":[],"id":"cfd18cea-dbe5-4ac0-864c-922704a8f46a"},{"cell_type":"markdown","source":"### Create Visuals so you can gain a business understanding of your data","metadata":{},"id":"4d169ba1-107e-422f-902d-d99750428755"},{"cell_type":"code","source":"# Replace with your target variable --- df.YOUR_TARGET_VARIABLE  \n# Look for major data imbalances\n# Also replace your X label\n# REPLACE YOUR TARGET VARIABLE\nplt.figure(figsize=(6, 4))\n\nsns.histplot(df.<replace with your target variable>, bins=40, color='black', alpha=1)\nplt.ylabel('Frequency')\nplt.xlabel('<replace with your target variable>')\nplt.title('PUT A LABEL ON IT')\n\nplt.show()","metadata":{},"execution_count":null,"outputs":[],"id":"227d8604-3160-4070-a1b8-b7c2e71ce771"},{"cell_type":"markdown","source":"## Notes\n* (1) Check for NaN under a single DataFrame column:\n\n* `df['your column name'].isnull().values.any()`\n\n* (2) Count the NaN under a single DataFrame column:\n\n`df['your column name'].isnull().sum()`\n\n* (3) Check for NaN under an entire DataFrame:\n\n`df.isnull().values.any()`\n\n* (4) Count the NaN under an entire DataFrame:\n\n`df.isnull().sum().sum()`","metadata":{},"id":"dd27ed74-32d7-4220-afb6-1e7f9288aa73"},{"cell_type":"code","source":"# Check for nulls --- you do NOT want nulls when you train\ndf.isnull().sum()","metadata":{},"execution_count":null,"outputs":[],"id":"4ad70d6b-e458-4bb7-be22-f550a07aed9f"},{"cell_type":"code","source":"# Check for the percentage of missing values\ndf.isnull().sum() / df.shape[0] * 100","metadata":{},"execution_count":null,"outputs":[],"id":"63f7fa39-21a7-44f5-9814-de07d05eb627"},{"cell_type":"code","source":"#check Value Counts\n# df.\"REPLACE WITH FIELD NAME\".value_counts()\ndf.value_counts()","metadata":{},"execution_count":null,"outputs":[],"id":"2719e6c0-f89e-48fe-adb8-d53be71f6c4e"},{"cell_type":"code","source":"df.head()","metadata":{},"execution_count":null,"outputs":[],"id":"a8a3347d-5141-4b92-bb07-3dc2dbc1bf3a"},{"cell_type":"code","source":"#delete columns --- this may or may NOT be needed.  As before - skip if you don't need it\n# You will encounter times where you will want to delete columns.  This is how you do that.\n# df = df.drop(['x5_latitude', 'x6_longitude', 'x1_transaction_date'], axis=1)\n# df","metadata":{},"execution_count":null,"outputs":[],"id":"2871e2ab-8677-43b4-bba1-36fe991438ab"},{"cell_type":"code","source":"# Split Data\n# i.e.:  address = London, UK\n# df[['city', 'country']] = df['address'].str.split(',', expand=True)","metadata":{},"execution_count":null,"outputs":[],"id":"303e98c0-40ed-42c3-9c74-df96f0747223"},{"cell_type":"code","source":"# Change any Data Types\n#Replace Data Types to Integer\n# df[\"Customer Number\"] = df['Customer Number'].astype('int')\n#Replace Data Types to String\n# df[\"Customer Number\"] = df['Customer Number'].astype('str')\n#Replace Data Types to Boolean\n# df[\"IsPurchased\"] = df['IsPurchased'].astype('bool')\n#Replace Data Types to Float\n# df[\"Total Spend\"] = df['Total Spend'].astype('float')\n#Replace Data Types to Datetime with format= '%Y%m%d'\n# df['Dates'] = pd.to_datetime(df['Dates'], format='%Y%m%d')","metadata":{},"execution_count":null,"outputs":[],"id":"7b792d06-355a-4924-b5ad-d7949dc42941"},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[],"id":"d0f1773c-07cd-424d-b068-a5fac3003633"},{"cell_type":"markdown","source":"## Univariate Analysis","metadata":{},"id":"0d761ee1-1821-4a01-840e-5e3d526eefac"},{"cell_type":"code","source":"## STICK TO CATEGORICAL COLUMNS INITIALLY\n#plot the histogram to see the distribution of the point data.\nsns.displot(data, x=\"YOUR_VARIABLE\")","metadata":{},"execution_count":null,"outputs":[],"id":"91bae25c-a110-474a-ab44-40a45af228ca"},{"cell_type":"code","source":"sns.countplot(x=\"YOUR_VARIABLE\", data=df)","metadata":{},"execution_count":null,"outputs":[],"id":"b7fc96db-35c9-44ce-aaf9-36ca4daf959f"},{"cell_type":"code","source":"df['YOUR_VARIABLE'].value_counts()","metadata":{},"execution_count":null,"outputs":[],"id":"1c4c87eb-2400-4843-8431-5da18a3f0c01"},{"cell_type":"code","source":"#measure its skewness and kurtosis\ndata['YOUR_VARIABLE'].agg(['skew', 'kurtosis']).transpose()","metadata":{},"execution_count":null,"outputs":[],"id":"251bbdf9-d9fe-49ff-98ba-7a6e85f6ef52"},{"cell_type":"code","source":"#check for outliers\nax = sns.boxplot(x=data[\"YOUR_VARIABLE\"])","metadata":{},"execution_count":null,"outputs":[],"id":"bb7cc890-ef3b-4c2b-859f-b2632d609670"},{"cell_type":"markdown","source":"![image info](https://miro.medium.com/max/1400/1*_aN1iaiVUTdoyPbyj-kVjA.jpeg)","metadata":{},"id":"d2c0ddf9-3a3d-41aa-9c86-01a97867186c"},{"cell_type":"markdown","source":"## Bivariate Analysis","metadata":{},"id":"1cd057ac-5753-47b3-b683-67542803ad7b"},{"cell_type":"code","source":"# Pick 2 variables to compare and replace SEX and DEFAULT WITH THE TWO VARIABLES\n# Stick with Categorical variables for now\nsns.set(rc={'figure.figsize':(15,10)})\nedu = sns.countplot(x='SEX', hue='DEFAULT', data=df)\nedu.set_xticklabels(['Male','Female'])\nplt.show()","metadata":{},"execution_count":null,"outputs":[],"id":"8185f87b-a16b-41b0-bffc-115c58c6928a"},{"cell_type":"code","source":"# Evaluate the Cross Tab\npd.crosstab(df.SEX,df.DEFAULT,normalize='index',margins=True)","metadata":{},"execution_count":null,"outputs":[],"id":"a1a09cac-b15d-40db-990c-c68cfcd522c5"},{"cell_type":"markdown","source":"## Correlation","metadata":{},"id":"b7cf74c6-1fce-4466-9b46-6c7f6029da2d"},{"cell_type":"code","source":"# Pearson Correlation\ncorrMatrix = df.corr()\nsns.heatmap(corrMatrix, annot = True, cmap= 'coolwarm')","metadata":{},"execution_count":null,"outputs":[],"id":"6a9a8101-96e2-44a0-98ed-7db58386e4c2"},{"cell_type":"code","source":"# Spearman Correlation\nsns.set(rc={'figure.figsize':(30,10)})\nsns.set_context(\"talk\", font_scale=0.7)","metadata":{},"execution_count":null,"outputs":[],"id":"50406a92-d2c6-4bbb-a3fa-2c3829359bb7"},{"cell_type":"code","source":"sns.heatmap(df.iloc[:,1:].corr(method='spearman'), cmap='rainbow_r', annot=True)","metadata":{},"execution_count":null,"outputs":[],"id":"63bcb85a-ec1e-458a-b505-a94e9cc719a7"},{"cell_type":"code","source":"# To get the Correlation between your variable of interest and the rest of the variables\n# - replace \"DEFAULT\" with your variable of interest.\ndf.drop(\"DEFAULT\", axis=1).apply(lambda x: x.corr(df.DEFAULT,method='spearman'))","metadata":{},"execution_count":null,"outputs":[],"id":"bdba25f0-c19f-4ac5-afb3-97d632d8023f"}]}