初赛/最终程序.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __说明:__\n",
    "\n",
    "本程序是 阿里云天池 的比赛 津南数字制造算法挑战赛(赛场一) B 榜20名提交结果所使用的最终程序\n",
    "\n",
    "队伍名称 Drop, 作者: 陶亚凡 陕西科技大学\n",
    "\n",
    "我的 github 和 博客:(虽然没东西把, 但是还是求波关注, follow, stars /捂脸)\n",
    "\n",
    "github: https://github.com/taoyafan\n",
    "\n",
    "博客: https://me.csdn.net/taoyafan\n",
    "\n",
    "队友（Blue 电子科技大学的） github 和 博客：(也求波关注, follow, stars /捂脸)\n",
    "\n",
    "github：https://github.com/BluesChang\n",
    "\n",
    "博客：https://blueschang.github.io\n",
    "\n",
    "程序各个部分很大程度的参考了鱼佬的 baseline\n",
    "\n",
    "感谢队友的很多贡献, 感谢鱼佬和他的的 baseline, 感谢林有夕大佬让我在群里不停的学到新知识.\n",
    "\n",
    "因为我这是第一了 ML 的比赛, 看着鱼佬 baseline 开始学习 pandas, sklearn 还有相关知识, 所以水平实在有限. 希望各位大佬给点意见或建议, 有什么问题或者可以改进的地方请告知我, 灰常感谢. \n",
    "\n",
    "一直想开源, 但是成绩太差, 趁着还在首页, 赶快开源了, 我也是抱着学习的心态, 也没想着拿奖, 所以这个程序也没啥保留~\n",
    "\n",
    "+ __程序思路:__ \n",
    "\n",
    "读取数 -> 手动处理训练集明显异常数据 -> 数据清洗 -> 特征工程 -> 训练\n",
    "\n",
    "__数据清洗:__\n",
    "\n",
    "删除缺失率过高的数据 -> 处理字符时间(段) -> 计算时间差 -> 处理异常值 -> 删除单一类别占比过大的特征\n",
    "\n",
    "__特征工程:__\n",
    "\n",
    "构建新特征 -> 利用特征之间的相关性预测 nan 值 -> 后向特征选择\n",
    "\n",
    "__训练:__\n",
    "\n",
    "使用 lgb 和 xgb 自动选择最优参数, 然后融合\n",
    "\n",
    "+ __数据通路:__\n",
    "\n",
    "开始学鱼佬的 baseline 自己写着写着变量名太多了,前后运行总是出现各种小错误, 所以对整体结构改了好多次, 最终使用了 pipe line, 包括了整个数据清洗和特征工程部分, 所以变量名少了, 各个部分也不存在耦合了, 所以我觉得有必要介绍下数据通路:\n",
    "\n",
    "数据读取得到 train, test ----> 合并得到 full ---> 经过 pipe line 得到 pipe_data ---> 训练集测试集分割得到 X_train 和 X_test ---> 训练得到结果 oof 和 predictions\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入包，读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/deap/tools/_hypervolume/pyhv.py:33: ImportWarning:\n",
      "\n",
      "Falling back to the python version of hypervolume module. Expect this to be very slow.\n",
      "\n",
      "/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning:\n",
      "\n",
      "can't resolve package from __spec__ or __package__, falling back on __name__ and __path__\n",
      "\n",
      "/usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning:\n",
      "\n",
      "Not importing directory /usr/local/lib/python3.6/dist-packages/mpl_toolkits: missing __init__\n",
      "\n",
      "/usr/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning:\n",
      "\n",
      "Not importing directory /usr/local/lib/python3.6/dist-packages/google: missing __init__\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from scipy import sparse\n",
    "import warnings\n",
    "import time\n",
    "import sys\n",
    "import os\n",
    "import re\n",
    "import datetime\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import plotly.offline as py\n",
    "import plotly.graph_objs as go \n",
    "import plotly.tools as tls\n",
    "from xgboost import XGBRegressor\n",
    "from tpot import TPOTRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning:\n",
      "\n",
      "can't resolve package from __spec__ or __package__, falling back on __name__ and __path__\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score, GridSearchCV, KFold, RepeatedKFold, ShuffleSplit\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, ExtraTreesRegressor\n",
    "from sklearn.svm import SVR, LinearSVR\n",
    "from sklearn.linear_model import ElasticNet, SGDRegressor, BayesianRidge\n",
    "from sklearn.kernel_ridge import KernelRidge\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
    "from sklearn.metrics import log_loss\n",
    "from sklearn.preprocessing import Imputer\n",
    "from scipy.stats import skew\n",
    "from sklearn.preprocessing import RobustScaler, StandardScaler, MinMaxScaler, Normalizer\n",
    "from sklearn.decomposition import PCA, KernelPCA\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import ParameterGrid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script><script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window._Plotly) {require(['plotly'],function(plotly) {window._Plotly=plotly;});}</script>"
      ]
     },
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py.init_notebook_mode(connected=True)\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "pd.set_option('display.max_columns',None)\n",
    "pd.set_option('max_colwidth',100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 设定文件名, 读取文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file_name = 'data/jinnan_round1_train_20181227.csv'\n",
    "test_file_name = 'data/jinnan_round1_testB_20190121.csv'\n",
    "test_name = 'testB'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 读取数据， 改名\n",
    "train = pd.read_csv(train_file_name, encoding = 'gb18030')\n",
    "test  = pd.read_csv(test_file_name, encoding = 'gb18030')\n",
    "train.rename(columns={'样本id':'id', '收率':'target'}, inplace = True)\n",
    "test.rename(columns={'样本id':'id', '收率':'target'}, inplace = True)\n",
    "target_name = 'target'\n",
    "\n",
    "# 存在异常数据，改为 nan\n",
    "train.loc[1304, 'A25'] = np.nan\n",
    "train['A25'] = train['A25'].astype(float)\n",
    "\n",
    "# 去掉 id 前缀\n",
    "train['id'] = train['id'].apply(lambda x: int(x.split('_')[1]))\n",
    "test['id'] = test['id'].apply(lambda x: int(x.split('_')[1]))\n",
    "\n",
    "train.drop(train[train[target_name] < 0.87].index, inplace=True)\n",
    "full=pd.concat([train, test], ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据清洗"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 删除缺失率高的特征\n",
    "\n",
    "+ __删除缺失值大于 th_high 的值__\n",
    "+ __缺失值在 th_low 和 th_high 之间的特征根据是否缺失增加新特征__\n",
    "  \n",
    "  如 B10 缺失较高，增加新特征 B10_null，如果缺失为1，否则为0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class del_nan_feature(BaseEstimator, TransformerMixin):\n",
    "    \n",
    "    def __init__(self, th_high=0.85, th_low=0.02):\n",
    "        self.th_high = th_high\n",
    "        self.th_low = th_low\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'del_nan_feature', ' '*5, '-'*30, '\\n')\n",
    "        print(\"shape before process = {}\".format(X.shape))\n",
    "\n",
    "        # 删除高缺失率特征\n",
    "        X.dropna(axis=1, thresh=(1-self.th_high)*X.shape[0], inplace=True)\n",
    "        \n",
    "        \n",
    "        # 缺失率较高，增加新特征\n",
    "        for col in X.columns:\n",
    "            if col == 'target':\n",
    "                continue\n",
    "            \n",
    "            miss_rate = X[col].isnull().sum()/ X.shape[0]\n",
    "            if miss_rate > self.th_low:\n",
    "                print(\"Missing rate of {} is {:.3f} exceed {}, adding new feature {}\".\n",
    "                     format(col, miss_rate, self.th_low, col+'_null'))\n",
    "                X[col+'_null'] = 0\n",
    "                X.loc[X[pd.isnull(X[col])].index, [col+'_null']] = 1\n",
    "        print(\"shape = {}\".format(X.shape))\n",
    "\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理字符时间（段）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 处理时间\n",
    "def timeTranSecond(t):\n",
    "    try:\n",
    "        h,m,s=t.split(\":\")\n",
    "    except:\n",
    "\n",
    "        if t=='1900/1/9 7:00':\n",
    "            return 7*3600/3600\n",
    "        elif t=='1900/1/1 2:30':\n",
    "            return (2*3600+30*60)/3600\n",
    "        elif pd.isnull(t):\n",
    "            return np.nan\n",
    "        else:\n",
    "            return 0\n",
    "\n",
    "    try:\n",
    "        tm = (int(h)*3600+int(m)*60+int(s))/3600\n",
    "    except:\n",
    "        return (30*60)/3600\n",
    "\n",
    "    return tm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 处理时间差\n",
    "def getDuration(se):\n",
    "    try:\n",
    "        sh,sm,eh,em=re.findall(r\"\\d+\",se)\n",
    "#         print(\"sh, sm, eh, em = {}, {}, {}, {}\".format(sh, em, eh, em))\n",
    "    except:\n",
    "        if pd.isnull(se):\n",
    "            return np.nan, np.nan, np.nan\n",
    "\n",
    "    try:\n",
    "        t_start = (int(sh)*3600 + int(sm)*60)/3600\n",
    "        t_end = (int(eh)*3600 + int(em)*60)/3600\n",
    "        \n",
    "        if t_start > t_end:\n",
    "            tm = t_end - t_start + 24\n",
    "        else:\n",
    "            tm = t_end - t_start\n",
    "    except:\n",
    "        if se=='19:-20:05':\n",
    "            return 19, 20, 1\n",
    "        elif se=='15:00-1600':\n",
    "            return 15, 16, 1\n",
    "        else:\n",
    "            print(\"se = {}\".format(se))\n",
    "\n",
    "\n",
    "    return t_start, t_end, tm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class handle_time_str(BaseEstimator, TransformerMixin):\n",
    "    \n",
    "    def __init__(self):\n",
    "        pass\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'handle_time_str', ' '*5, '-'*30, '\\n')\n",
    "\n",
    "        for f in ['A5','A7','A9','A11','A14','A16','A24','A26','B5','B7']:\n",
    "            try:\n",
    "                X[f] = X[f].apply(timeTranSecond)\n",
    "            except:\n",
    "                print(f,'应该在前面被删除了！')\n",
    "\n",
    "\n",
    "        for f in ['A20','A28','B4','B9','B10','B11']:\n",
    "            try:\n",
    "                start_end_diff = X[f].apply(getDuration)\n",
    "                \n",
    "                X[f+'_start'] = start_end_diff.apply(lambda x: x[0])\n",
    "                X[f+'_end'] = start_end_diff.apply(lambda x: x[1])\n",
    "                X[f] = start_end_diff.apply(lambda x: x[2])\n",
    "\n",
    "            except:\n",
    "                print(f,'应该在前面被删除了！')\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算时间差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def t_start_t_end(t):\n",
    "    if pd.isnull(t[0]) or pd.isnull(t[1]):\n",
    "#         print(\"t_start = {}, t_end = {}, id = {}\".format(t[0], t[1], t[2]))\n",
    "        return np.nan\n",
    "        \n",
    "    if t[1] < t[0]:\n",
    "        t[1] += 24\n",
    "        \n",
    "    dt = t[1] - t[0]\n",
    "\n",
    "    if(dt > 24 or dt < 0):\n",
    "#         print(\"dt error, t_start = {}, t_end = {}, id = {}\".format(t[0], t[1], t[2]))\n",
    "        return np.nan\n",
    "    \n",
    "    return dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class calc_time_diff(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'calc_time_diff', ' '*5, '-'*30, '\\n')\n",
    "\n",
    "        # t_start 为时间的开始， tn 为中间的时间，减去 t_start 得到时间差\n",
    "        t_start = ['A9', 'A24', 'B5']\n",
    "        tn = {'A9':['A11', 'A14', 'A16'], 'A24':['A26'], 'B5':['B7']}\n",
    "        \n",
    "        # 计算时间差\n",
    "        for t_s in t_start:\n",
    "            for t_e in tn[t_s]:\n",
    "                X[t_e+'-'+t_s] = X[[t_s,t_e, target_name]].apply(t_start_t_end, axis=1)\n",
    "                \n",
    "        # 所有结果保留 3 位小数\n",
    "        X = X.apply(lambda x:round(x, 3))\n",
    "        \n",
    "        print(\"shape = {}\".format(X.shape))\n",
    "        \n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __单一类别个数小于 threshold 的值视为异常值, 改为 nan__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class handle_outliers(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def __init__(self, threshold=2):\n",
    "        self.th = threshold\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'handle_outliers', ' '*5, '-'*30, '\\n')\n",
    "        category_col = [col for col in X if col not in ['id', 'target']]\n",
    "        for col in category_col:\n",
    "            label = X[col].value_counts(dropna=False).index.tolist()\n",
    "            for i, num in enumerate(X[col].value_counts(dropna=False).values):\n",
    "                if num <= self.th:\n",
    "#                     print(\"Number of label {} in feature {} is {}\".format(label[i], col, num))\n",
    "                    X.loc[X[col]==label[i], [col]] = np.nan\n",
    "        \n",
    "        print(\"shape = {}\".format(X.shape))\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 删除单一类别占比过大特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class del_single_feature(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def __init__(self, threshold=0.98):\n",
    "        # 删除单一类别占比大于 threshold 的特征\n",
    "        self.th = threshold\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'del_single_feature', ' '*5, '-'*30, '\\n')\n",
    "        category_col = [col for col in X if col not in ['target']]\n",
    "        \n",
    "        for col in category_col:\n",
    "            rate = X[col].value_counts(normalize=True, dropna=False).values[0]\n",
    "            \n",
    "            if rate > self.th:\n",
    "                print(\"{} 的最大类别占比是 {}, drop it\".format(col, rate))\n",
    "                X.drop(col, axis=1, inplace=True)\n",
    "\n",
    "        print(\"shape = {}\".format(X.shape))\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 特征工程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获得训练集与测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_data(pipe_data, target_name='target'):\n",
    "   \n",
    "    # 特征列名\n",
    "    category_col = [col for col in pipe_data if col not in ['target',target_name]]\n",
    "    \n",
    "    # 训练、测试行索引\n",
    "    train_idx = pipe_data[np.logical_not(pd.isnull(pipe_data[target_name]))].index\n",
    "    test_idx = pipe_data[pd.isnull(pipe_data[target_name])].index\n",
    "    \n",
    "    # 获得 train、test 数据\n",
    "    X_train = pipe_data.loc[train_idx, category_col].values.astype(np.float64)\n",
    "    y_train = np.squeeze(pipe_data.loc[train_idx, [target_name]].values.astype(np.float64))\n",
    "    X_test = pipe_data.loc[test_idx, category_col].values.astype(np.float64)\n",
    "    \n",
    "    return X_train, y_train, X_test, test_idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## xgb(用于特征 nan 值预测)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### xgb\n",
    "def xgb_predict(X_train, y_train, X_test, params=None, verbose_eval=100):\n",
    "    \n",
    "    if params == None:\n",
    "        xgb_params = {'eta': 0.05, 'max_depth': 10, 'subsample': 0.8, 'colsample_bytree': 0.8, \n",
    "                  'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True, 'nthread': 4}\n",
    "    else:\n",
    "        xgb_params = params\n",
    "\n",
    "    folds = KFold(n_splits=10, shuffle=True, random_state=2018)\n",
    "    oof_xgb = np.zeros(len(X_train))\n",
    "    predictions_xgb = np.zeros(len(X_test))\n",
    "\n",
    "    for fold_, (trn_idx, val_idx) in enumerate(folds.split(X_train, y_train)):\n",
    "        if(verbose_eval):\n",
    "            print(\"fold n°{}\".format(fold_+1))\n",
    "            print(\"len trn_idx  {}\".format(len(trn_idx)))\n",
    "            \n",
    "        trn_data = xgb.DMatrix(X_train[trn_idx], y_train[trn_idx])\n",
    "        val_data = xgb.DMatrix(X_train[val_idx], y_train[val_idx])\n",
    "\n",
    "        watchlist = [(trn_data, 'train'), (val_data, 'valid_data')]\n",
    "        clf = xgb.train(dtrain=trn_data,\n",
    "                        num_boost_round=20000,\n",
    "                        evals=watchlist,\n",
    "                        early_stopping_rounds=200,\n",
    "                        verbose_eval=verbose_eval,\n",
    "                        params=xgb_params)\n",
    "        \n",
    "        \n",
    "        oof_xgb[val_idx] = clf.predict(xgb.DMatrix(X_train[val_idx]), ntree_limit=clf.best_ntree_limit)\n",
    "        predictions_xgb += clf.predict(xgb.DMatrix(X_test), ntree_limit=clf.best_ntree_limit) / folds.n_splits\n",
    "\n",
    "    if(verbose_eval):\n",
    "        print(\"CV score: {:<8.8f}\".format(mean_squared_error(oof_xgb, y_train)))\n",
    "    return oof_xgb, predictions_xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 根据 B14 构建新特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class add_new_features(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def __init__(self):\n",
    "        pass\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "\n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'add_new_features', ' '*5, '-'*30, '\\n')\n",
    "\n",
    "        # 经过测试,只有 B14 / B12 有用\n",
    "        \n",
    "#         X['B14/A1'] = X['B14'] / X['A1']\n",
    "#         X['B14/A3'] = X['B14'] / X['A3']\n",
    "#         X['B14/A4'] = X['B14'] / X['A4']\n",
    "#         X['B14/A19'] = X['B14'] / X['A19']\n",
    "#         X['B14/B1'] = X['B14'] / X['B1']\n",
    "#         X['B14/B9'] = X['B14'] / X['B9']\n",
    "\n",
    "        X['B14/B12'] = X['B14'] / X['B12']\n",
    "        \n",
    "        print(\"shape = {}\".format(X.shape))\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 选择特征, nan 值填充\n",
    "\n",
    "+ __选择可能有效的特征__   (只是为了加快选择时间)\n",
    "\n",
    "+ __利用其他特征预测 nan，取最近值填充__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_closest(indexes, predicts):\n",
    "    print(\"From {}\".format(predicts))\n",
    "\n",
    "    for i, one in enumerate(predicts):\n",
    "        predicts[i] = indexes[np.argsort(abs(indexes - one))[0]]\n",
    "\n",
    "    print(\"To {}\".format(predicts))\n",
    "    return predicts\n",
    "    \n",
    "\n",
    "def value_select_eval(pipe_data, selected_features):\n",
    "    \n",
    "    # 经过多次测试, 只选择可能是有用的特征\n",
    "    cols_with_nan = [col for col in pipe_data.columns \n",
    "                     if pipe_data[col].isnull().sum()>0 and col in selected_features]\n",
    "\n",
    "    for col in cols_with_nan:\n",
    "        X_train, y_train, X_test, test_idx = split_data(pipe_data, target_name=col)\n",
    "        oof_xgb, predictions_xgb = xgb_predict(X_train, y_train, X_test, verbose_eval=False)\n",
    "        \n",
    "        print(\"-\"*100, end=\"\\n\\n\")\n",
    "        print(\"CV normal MAE scores of predicting {} is {}\".\n",
    "              format(col, mean_absolute_error(oof_xgb, y_train)/np.mean(y_train)))\n",
    "        \n",
    "        pipe_data.loc[test_idx, [col]] = get_closest(pipe_data[col].value_counts().index,\n",
    "                                              predictions_xgb)\n",
    "\n",
    "    pipe_data = pipe_data[selected_features+['target']]\n",
    "\n",
    "    return pipe_data\n",
    "\n",
    "# pipe_data = value_eval(pipe_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class selsected_fill_nans(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def __init__(self, selected_features = ['A3_null', 'A6', 'A16', 'A25', 'A28', 'A28_end',\n",
    "                                           'B5', 'B10_null', 'B11_null', 'B14', 'B14/B12', 'id']):\n",
    "        self.selected_fearutes = selected_features\n",
    "        pass\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'selsected_fill_nans', ' '*5, '-'*30, '\\n')\n",
    "\n",
    "        X = value_select_eval(X, self.selected_fearutes)\n",
    "\n",
    "        print(\"shape = {}\".format(X.shape))\n",
    "        return X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def modeling_cross_validation(data):\n",
    "    X_train, y_train, X_test, test_idx = split_data(data,\n",
    "                                                    target_name='target')\n",
    "    oof_xgb, _ = xgb_predict(X_train, y_train, X_test, verbose_eval=False)\n",
    "    print('-'*100, end='\\n\\n')\n",
    "    return mean_squared_error(oof_xgb, y_train)\n",
    "\n",
    "\n",
    "def featureSelect(data):\n",
    "\n",
    "    init_cols = [f for f in data.columns if f not in ['target']]\n",
    "    best_cols = init_cols.copy()\n",
    "    best_score = modeling_cross_validation(data[best_cols+['target']])\n",
    "    print(\"初始 CV score: {:<8.8f}\".format(best_score))\n",
    "\n",
    "    for col in init_cols:\n",
    "        best_cols.remove(col)\n",
    "        score = modeling_cross_validation(data[best_cols+['target']])\n",
    "        print(\"当前选择特征: {}, CV score: {:<8.8f}, 最佳cv score: {:<8.8f}\".\n",
    "              format(col, score, best_score), end=\" \")\n",
    "        \n",
    "        if best_score - score > 0.0000004:\n",
    "            best_score = score\n",
    "            print(\"有效果,删除！！！！\")\n",
    "        else:\n",
    "            best_cols.append(col)\n",
    "            print(\"保留\")\n",
    "\n",
    "    print('-'*100)\n",
    "    print(\"优化后 CV score: {:<8.8f}\".format(best_score))\n",
    "    return best_cols, best_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 后向选择特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class select_feature(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def __init__(self, init_features = None):\n",
    "        self.init_features = init_features\n",
    "        pass\n",
    "        \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        print('-'*30, ' '*5, 'select_feature', ' '*5, '-'*30, '\\n')\n",
    "        \n",
    "        if self.init_features:\n",
    "            X = X[self.init_features + ['target']]\n",
    "            best_features = self.init_features\n",
    "        else:\n",
    "            best_features = [col for col in X.columns]\n",
    "        \n",
    "        last_feartues = []\n",
    "        iteration = 0\n",
    "        equal_time = 0\n",
    "        \n",
    "        best_CV = 1\n",
    "        best_CV_feature = []\n",
    "        \n",
    "        # 打乱顺序,但是使用相同种子,保证每次运行结果相同\n",
    "        np.random.seed(2018)\n",
    "        while True:\n",
    "            print(\"Iteration = {}\\n\".format(iteration))\n",
    "            best_features, score = featureSelect(X[best_features + ['target']])\n",
    "            \n",
    "            # 保存最优 CV 的参数\n",
    "            if score < best_CV:\n",
    "                best_CV = score\n",
    "                best_CV_feature = best_features\n",
    "                print(\"Found best score :{}, with features :{}\".format(best_CV, best_features))\n",
    "                \n",
    "            np.random.shuffle(best_features)\n",
    "            print(\"\\nCurrent fearure length = {}\".format(len(best_features)))\n",
    "            \n",
    "            # 最终 3 次迭代相同，则终止迭代\n",
    "            if len(best_features) == len(last_feartues):\n",
    "                equal_time += 1\n",
    "                if equal_time == 3:\n",
    "                    break\n",
    "            else:\n",
    "                equal_time = 0\n",
    "            \n",
    "            last_feartues = best_features\n",
    "            iteration = iteration + 1\n",
    "\n",
    "            print(\"\\n\\n\\n\")\n",
    "            \n",
    "        return X[best_features + ['target']]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建 pipeline, 处理数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------       del_nan_feature       ------------------------------ \n",
      "\n",
      "shape before process = (1532, 44)\n",
      "Missing rate of A3 is 0.029 exceed 0.02, adding new feature A3_null\n",
      "Missing rate of B10 is 0.172 exceed 0.02, adding new feature B10_null\n",
      "Missing rate of B11 is 0.597 exceed 0.02, adding new feature B11_null\n",
      "shape = (1532, 44)\n",
      "------------------------------       handle_time_str       ------------------------------ \n",
      "\n",
      "A7 应该在前面被删除了！\n",
      "------------------------------       calc_time_diff       ------------------------------ \n",
      "\n",
      "shape = (1532, 61)\n",
      "------------------------------       handle_outliers       ------------------------------ \n",
      "\n",
      "shape = (1532, 61)\n",
      "------------------------------       del_single_feature       ------------------------------ \n",
      "\n",
      "shape = (1532, 61)\n",
      "------------------------------       add_new_features       ------------------------------ \n",
      "\n",
      "shape = (1532, 62)\n",
      "------------------------------       selsected_fill_nans       ------------------------------ \n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting A16 is 0.006573812182966658\n",
      "From [2.82036011 2.0116301  2.78110905 2.24891324 2.64147919 2.61220436\n",
      " 2.5752665  3.00300634 2.9279013  2.84588192 2.99439096 3.15101939\n",
      " 1.30144072 3.0146347  2.44444267 2.60455203 2.70574424 2.75994805\n",
      " 2.58867866 3.00614175 2.78697994 2.03778946 2.69046123 2.72509097\n",
      " 2.03607538 2.52129808 2.99479207 2.92738628 2.41858149 2.70892806\n",
      " 2.80188948 2.75916436 2.00558983 2.99666125 3.02267092 2.11280097\n",
      " 2.88487023 2.52905945 3.2504842  2.92606165 2.52358037 2.57779263\n",
      " 2.58069354 2.91890304 2.9953025  2.49374625 2.68844172 2.45054981\n",
      " 3.02282879 2.01016228]\n",
      "To [3.  2.  3.  2.  2.5 2.5 2.5 3.  3.  3.  3.  3.  1.5 3.  2.5 2.5 2.5 3.\n",
      " 2.5 3.  3.  2.  2.5 2.5 2.  2.5 3.  3.  2.5 2.5 3.  3.  2.  3.  3.  2.\n",
      " 3.  2.5 3.5 3.  2.5 2.5 2.5 3.  3.  2.5 2.5 2.5 3.  2. ]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting A25 is 0.006985261873501027\n",
      "From [74.69595861 76.41018534 79.13030624 80.63013458 83.78137398 69.97030067\n",
      " 79.99184084 81.28120136]\n",
      "To [75. 76. 79. 80. 80. 70. 80. 80.]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting A28 is 0.037333278007722834\n",
      "From [1.16017157 0.60625793 1.06248447 0.79223084 0.98663072 0.9867671\n",
      " 0.93546665 0.80992338 0.5366797  1.07250487 0.89877572]\n",
      "To [1.167 0.667 1.    0.667 1.    1.    1.    0.667 0.5   1.    1.   ]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting A6 is 0.07248546276204286\n",
      "From [38.14284706 37.4052155  28.88562822 32.06546164 32.16541529 36.24718952\n",
      " 36.2260437  22.94354749 39.25126076 24.79291415 32.7412734  35.56525469\n",
      " 33.02388406 25.72406816]\n",
      "To [38. 37. 29. 32. 32. 36. 36. 23. 39. 25. 33. 36. 33. 26.]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting B14 is 0.001297790416243159\n",
      "From [400.24114609 402.01158142 400.45770264 401.50468063 402.03158188\n",
      " 337.98728943 402.02846909 401.99261093 402.0224762  341.76803589\n",
      " 400.6063652 ]\n",
      "To [400. 400. 400. 400. 400. 340. 400. 400. 400. 340. 400.]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting B5 is 0.016972058774115534\n",
      "From [15.12159047 15.72788435 14.3109533  19.78048182 14.65058997 14.53606975\n",
      " 15.33118927 14.2788609  13.99365219 14.93855688 14.00136399 14.85706538\n",
      " 16.71887732 21.97685862 13.37160268 15.54346603 14.7121506 ]\n",
      "To [15.  15.5 14.5 20.  14.5 14.5 15.5 14.5 14.  15.  14.  15.  16.5 22.\n",
      " 13.5 15.5 14.5]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting A28_end is 0.012402601510432685\n",
      "From [13.98934126  9.29064429  0.80129172 13.6963979  10.73074675 15.34772384\n",
      "  8.8364796  16.79893827 15.48608887 15.33091271 19.34231484 12.45444262\n",
      " 15.49709904 14.68686521 15.09787071 15.34401202 17.98878217 10.97875953\n",
      " 10.45144868 10.7770443  10.52596968 12.5426327  14.00931859 16.79731178\n",
      " 14.87435162 20.49663401 14.73405266 17.8244971   5.21450764 14.84811437]\n",
      "To [14.   9.   1.  13.5 10.5 15.5  9.  17.  15.5 15.5 19.5 12.5 15.5 14.5\n",
      " 15.  15.5 18.  11.  10.5 11.  10.5 12.5 14.  17.  15.  20.5 14.5 18.\n",
      "  5.  15. ]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "CV normal MAE scores of predicting B14/B12 is 0.002980983626315855\n",
      "From [0.44779329 0.49992409 0.50056992 0.3333493  0.49980065 0.81825209\n",
      " 0.49989815 0.49929611 0.76591279 0.50158779 0.80235612 0.4996887 ]\n",
      "To [0.44444444 0.5        0.5        0.33333333 0.5        0.85\n",
      " 0.5        0.5        0.7        0.5        0.85       0.5       ]\n",
      "shape = (1532, 13)\n",
      "------------------------------       select_feature       ------------------------------ \n",
      "\n",
      "Iteration = 0\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011896\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011909, 最佳cv score: 0.00011896 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012150, 最佳cv score: 0.00011896 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00011866, 最佳cv score: 0.00011896 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00011865, 最佳cv score: 0.00011896 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A28, CV score: 0.00011749, 最佳cv score: 0.00011896 有效果,删除！！！！\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A28_end, CV score: 0.00011743, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B5, CV score: 0.00011818, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011870, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B11_null, CV score: 0.00011940, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012126, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012029, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018189, 最佳cv score: 0.00011749 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011749\n",
      "Found best score :0.00011748851254246741, with features :['A3_null', 'A6', 'A16', 'A25', 'A28_end', 'B5', 'B10_null', 'B11_null', 'B14', 'B14/B12', 'id']\n",
      "\n",
      "Current fearure length = 11\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 1\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011888\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011882, 最佳cv score: 0.00011888 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00011935, 最佳cv score: 0.00011888 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B11_null, CV score: 0.00011808, 最佳cv score: 0.00011888 有效果,删除！！！！\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012059, 最佳cv score: 0.00011808 保留\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012291, 最佳cv score: 0.00011808 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A28_end, CV score: 0.00011717, 最佳cv score: 0.00011808 有效果,删除！！！！\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B5, CV score: 0.00011785, 最佳cv score: 0.00011717 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012231, 最佳cv score: 0.00011717 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00011826, 最佳cv score: 0.00011717 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011966, 最佳cv score: 0.00011717 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018469, 最佳cv score: 0.00011717 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011717\n",
      "Found best score :0.00011716922906431079, with features :['A3_null', 'A25', 'B14', 'B14/B12', 'B5', 'A6', 'A16', 'B10_null', 'id']\n",
      "\n",
      "Current fearure length = 9\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 2\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011730\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012321, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011892, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012037, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00011723, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B5, CV score: 0.00011902, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00011825, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018102, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012256, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011870, 最佳cv score: 0.00011730 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011730\n",
      "\n",
      "Current fearure length = 9\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 3\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011678\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00011825, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011825, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B5, CV score: 0.00011854, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00011864, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018284, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012127, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012289, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012132, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011959, 最佳cv score: 0.00011678 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011678\n",
      "Found best score :0.00011677842465073222, with features :['A25', 'B10_null', 'B5', 'A16', 'id', 'B14', 'A6', 'B14/B12', 'A3_null']\n",
      "\n",
      "Current fearure length = 9\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 4\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011833\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012009, 最佳cv score: 0.00011833 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012129, 最佳cv score: 0.00011833 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011953, 最佳cv score: 0.00011833 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012132, 最佳cv score: 0.00011833 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B5, CV score: 0.00011791, 最佳cv score: 0.00011833 有效果,删除！！！！\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00012081, 最佳cv score: 0.00011791 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011889, 最佳cv score: 0.00011791 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00012040, 最佳cv score: 0.00011791 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018179, 最佳cv score: 0.00011791 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011791\n",
      "\n",
      "Current fearure length = 8\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 5\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011747\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018279, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00012058, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012409, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012101, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011944, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00012004, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011840, 最佳cv score: 0.00011747 保留\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012226, 最佳cv score: 0.00011747 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011747\n",
      "\n",
      "Current fearure length = 8\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 6\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011715\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018398, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00011984, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00012132, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012198, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011808, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00011993, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012184, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012037, 最佳cv score: 0.00011715 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011715\n",
      "\n",
      "Current fearure length = 8\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Iteration = 7\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "初始 CV score: 0.00011709\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A25, CV score: 0.00012090, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: id, CV score: 0.00018248, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14/B12, CV score: 0.00012002, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B14, CV score: 0.00012325, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A3_null, CV score: 0.00012002, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A16, CV score: 0.00011987, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: A6, CV score: 0.00012154, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "当前选择特征: B10_null, CV score: 0.00011942, 最佳cv score: 0.00011709 保留\n",
      "----------------------------------------------------------------------------------------------------\n",
      "优化后 CV score: 0.00011709\n",
      "\n",
      "Current fearure length = 8\n",
      "(1532, 9)\n"
     ]
    }
   ],
   "source": [
    "selected_features = ['A3_null', 'A6', 'A16', 'A25', 'A28', 'A28_end', \n",
    "                     'B5', 'B10_null', 'B11_null', 'B14', 'B14/B12', 'id']\n",
    "\n",
    "pipe = Pipeline([\n",
    "                ('del_nan_feature', del_nan_feature()),\n",
    "                ('handle_time_str', handle_time_str()),\n",
    "                ('calc_time_diff', calc_time_diff()),\n",
    "                ('Handle_outliers', handle_outliers(2)),\n",
    "                ('del_single_feature', del_single_feature(1)),\n",
    "                ('add_new_features', add_new_features()),\n",
    "                ('selsected_fill_nans', selsected_fill_nans(selected_features)),\n",
    "                ('select_feature', select_feature(selected_features)),\n",
    "                ])\n",
    "\n",
    "pipe_data = pipe.fit_transform(full.copy())\n",
    "print(pipe_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自动调参"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_best_params(pipe_data, predict_fun, param_grid):\n",
    "    \n",
    "    # 获得 train 和 test, 归一化\n",
    "    X_train, y_train, X_test, test_idx = split_data(pipe_data, target_name='target')\n",
    "    min_max_scaler = MinMaxScaler()\n",
    "    X_train = min_max_scaler.fit_transform(X_train)\n",
    "    X_test = min_max_scaler.transform(X_test)\n",
    "    best_score = 1\n",
    "\n",
    "    # 遍历所有参数,寻找最优\n",
    "    for params in ParameterGrid(param_grid):\n",
    "        print('-'*100, \"\\nparams = \\n{}\\n\".format(params))\n",
    "\n",
    "        oof, predictions = predict_fun(X_train, y_train, X_test, params=params, verbose_eval=False)\n",
    "        score = mean_squared_error(oof, y_train)\n",
    "        print(\"CV score: {}, current best score: {}\".format(score, best_score))\n",
    "\n",
    "        if best_score > score:\n",
    "            print(\"Found new best score: {}\".format(score))\n",
    "            best_score = score\n",
    "            best_params = params\n",
    "\n",
    "\n",
    "    print('\\n\\nbest params: {}'.format(best_params))\n",
    "    print('best score: {}'.format(best_score))\n",
    "    \n",
    "    return best_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## lgb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lgb_predict(X_train, y_train, X_test, params=None, verbose_eval=100):\n",
    "    \n",
    "    if params == None:\n",
    "        lgb_param = {'num_leaves': 20, 'min_data_in_leaf': 2, 'objective':'regression', 'max_depth': 4,\n",
    "         'learning_rate': 0.06, \"min_child_samples\": 3, \"boosting\": \"gbdt\", \"feature_fraction\": 1,\n",
    "         \"bagging_freq\": 0.7, \"bagging_fraction\": 1, \"bagging_seed\": 11, \"metric\": 'mse', \"lambda_l2\": 0.003,\n",
    "         \"verbosity\": -1}\n",
    "    else :\n",
    "        lgb_param = params\n",
    "        \n",
    "    folds = KFold(n_splits=10, shuffle=True, random_state=2018)\n",
    "    oof_lgb = np.zeros(len(X_train))\n",
    "    predictions_lgb = np.zeros(len(X_test))\n",
    "\n",
    "    for fold_, (trn_idx, val_idx) in enumerate(folds.split(X_train, y_train)):\n",
    "        if verbose_eval:\n",
    "            print(\"fold n°{}\".format(fold_+1))\n",
    "            \n",
    "        trn_data = lgb.Dataset(X_train[trn_idx], y_train[trn_idx])\n",
    "        val_data = lgb.Dataset(X_train[val_idx], y_train[val_idx])\n",
    "\n",
    "        num_round = 10000\n",
    "        clf = lgb.train(lgb_param, trn_data, num_round, valid_sets = [trn_data, val_data],\n",
    "                        verbose_eval=verbose_eval, early_stopping_rounds = 100)\n",
    "        \n",
    "        oof_lgb[val_idx] = clf.predict(X_train[val_idx], num_iteration=clf.best_iteration)\n",
    "\n",
    "        predictions_lgb += clf.predict(X_test, num_iteration=clf.best_iteration) / folds.n_splits\n",
    "\n",
    "        if verbose_eval:\n",
    "            print(\"CV score: {:<8.8f}\".format(mean_squared_error(oof_lgb, y_train)))\n",
    "    \n",
    "    return oof_lgb, predictions_lgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __选择最优参数__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.06, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011489364207207567, current best score: 1\n",
      "Found new best score: 0.00011489364207207567\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.06, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012002848815037457, current best score: 0.00011489364207207567\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.06, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011402092342458887, current best score: 0.00011489364207207567\n",
      "Found new best score: 0.00011402092342458887\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.06, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011882313706702633, current best score: 0.00011402092342458887\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.06, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011427665390150208, current best score: 0.00011402092342458887\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.06, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.000118469789122594, current best score: 0.00011402092342458887\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.12, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011547690407588457, current best score: 0.00011402092342458887\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.12, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011985943852356268, current best score: 0.00011402092342458887\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011231611613708764, current best score: 0.00011402092342458887\n",
      "Found new best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011748828797017007, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.12, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011554903372234801, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.12, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012001078341271754, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.24, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.0001135845354614368, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.24, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011950413736010373, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.24, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011449170101534524, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.24, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012067409892140623, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.24, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV score: 0.00011435307236140136, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.0003, 'learning_rate': 0.24, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012009711500307733, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.06, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011480005750940053, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.06, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012025520678151364, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.06, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011420826273292763, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.06, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011871676666472398, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.06, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011447430938895559, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.06, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011845637169175561, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011541082150277204, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011943383299618423, current best score: 0.00011231611613708764\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011212722352757958, current best score: 0.00011231611613708764\n",
      "Found new best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.0001181836666297253, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.0001158514519987164, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012000426127843114, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.24, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011360641302273318, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.24, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011965654294675753, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.24, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011443935391500729, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.24, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV score: 0.00012091648099518892, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.24, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011431927180539175, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.24, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012002843916496843, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.06, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011451755742912798, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.06, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011991034882396216, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.06, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011421574014520517, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.06, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011862220062373018, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.06, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011490378784720141, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.06, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011862098330828783, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.12, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011474358647366374, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.12, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011966504510458172, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011221151541733946, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011777070012250079, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.12, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.0001153000539253855, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.12, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00012101952260099292, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.24, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011434513539248205, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.24, 'max_depth': 3, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011965789595988069, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.24, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV score: 0.000114353078699699, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.24, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.0001212472410367223, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.24, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.00011433021385951927, current best score: 0.00011212722352757958\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.003, 'learning_rate': 0.24, 'max_depth': 5, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 3, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "\n",
      "CV score: 0.0001202623231156629, current best score: 0.00011212722352757958\n",
      "\n",
      "\n",
      "best params: {'bagging_fraction': 1, 'bagging_freq': 1, 'bagging_seed': 11, 'boosting': 'gbdt', 'feature_fraction': 0.7, 'lambda_l2': 0.001, 'learning_rate': 0.12, 'max_depth': 4, 'metric': 'mse', 'min_child_samples': 3, 'min_data_in_leaf': 2, 'num_leaves': 20, 'objective': 'regression', 'verbosity': -1}\n",
      "best score: 0.00011212722352757958\n"
     ]
    }
   ],
   "source": [
    "param_grid = [\n",
    "        {'num_leaves': [20], 'min_data_in_leaf': [2, 3], 'objective':['regression'],\n",
    "         'max_depth': [3, 4, 5], 'learning_rate': [0.06, 0.12, 0.24], \"min_child_samples\": [3],\n",
    "         \"boosting\": [\"gbdt\"], \"feature_fraction\": [0.7], \"bagging_freq\": [1],\n",
    "         \"bagging_fraction\": [1], \"bagging_seed\": [11], \"metric\": ['mse'],\n",
    "         \"lambda_l2\": [0.0003, 0.001, 0.003], \"verbosity\": [-1]}\n",
    "              ]\n",
    "\n",
    "lgb_best_params = find_best_params(pipe_data, lgb_predict, param_grid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __lgb 训练__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold n°1\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttraining's l2: 7.64635e-05\tvalid_1's l2: 0.00013957\n",
      "CV score: 0.76865147\n",
      "fold n°2\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "Early stopping, best iteration is:\n",
      "[82]\ttraining's l2: 8.03721e-05\tvalid_1's l2: 0.000110882\n",
      "CV score: 0.68291002\n",
      "fold n°3\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.30193e-05\tvalid_1's l2: 0.000139552\n",
      "Early stopping, best iteration is:\n",
      "[173]\ttraining's l2: 5.68838e-05\tvalid_1's l2: 0.000138392\n",
      "CV score: 0.59707720\n",
      "fold n°4\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.73108e-05\tvalid_1's l2: 0.000104267\n",
      "Early stopping, best iteration is:\n",
      "[104]\ttraining's l2: 7.56046e-05\tvalid_1's l2: 9.52206e-05\n",
      "CV score: 0.51161256\n",
      "fold n°5\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.36518e-05\tvalid_1's l2: 0.000110701\n",
      "Early stopping, best iteration is:\n",
      "[111]\ttraining's l2: 6.91289e-05\tvalid_1's l2: 0.0001088\n",
      "CV score: 0.42667732\n",
      "fold n°6\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.63032e-05\tvalid_1's l2: 0.000123434\n",
      "Early stopping, best iteration is:\n",
      "[110]\ttraining's l2: 7.12232e-05\tvalid_1's l2: 0.000120677\n",
      "CV score: 0.34177334\n",
      "fold n°7\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "Early stopping, best iteration is:\n",
      "[88]\ttraining's l2: 7.56938e-05\tvalid_1's l2: 0.000111444\n",
      "CV score: 0.25611485\n",
      "fold n°8\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.83909e-05\tvalid_1's l2: 8.13893e-05\n",
      "Early stopping, best iteration is:\n",
      "[177]\ttraining's l2: 6.21352e-05\tvalid_1's l2: 7.96722e-05\n",
      "CV score: 0.17049800\n",
      "fold n°9\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.74223e-05\tvalid_1's l2: 0.000108159\n",
      "Early stopping, best iteration is:\n",
      "[183]\ttraining's l2: 5.99078e-05\tvalid_1's l2: 0.000107585\n",
      "CV score: 0.08544523\n",
      "fold n°10\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's l2: 5.66965e-05\tvalid_1's l2: 0.000109477\n",
      "Early stopping, best iteration is:\n",
      "[176]\ttraining's l2: 5.98262e-05\tvalid_1's l2: 0.000108649\n",
      "CV score: 0.00011213\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train, X_test, test_idx = split_data(pipe_data, target_name='target')\n",
    "min_max_scaler = MinMaxScaler()\n",
    "X_train = min_max_scaler.fit_transform(X_train)\n",
    "X_test = min_max_scaler.transform(X_test)\n",
    "oof_lgb, predictions_lgb = lgb_predict(X_train, y_train, X_test, params=lgb_best_params, verbose_eval=200) #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __选择最优参数__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011466648461334117, current best score: 1\n",
      "Found new best score: 0.00011466648461334117\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011456634369353389, current best score: 0.00011466648461334117\n",
      "Found new best score: 0.00011456634369353389\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011522095556659337, current best score: 0.00011456634369353389\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.00011575362802403785, current best score: 0.00011456634369353389\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.0001153688729909817, current best score: 0.00011456634369353389\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011393958598424273, current best score: 0.00011456634369353389\n",
      "Found new best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011519961810983215, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.0001159343681149051, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011533954423435673, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011435484501228615, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011550346442947287, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.7, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.00011732503571073892, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011507860801671209, current best score: 0.00011393958598424273\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011373673653760243, current best score: 0.00011393958598424273\n",
      "Found new best score: 0.00011373673653760243\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011455317620732011, current best score: 0.00011373673653760243\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.0001157106534400235, current best score: 0.00011373673653760243\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011484817588831511, current best score: 0.00011373673653760243\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011350220852811235, current best score: 0.00011373673653760243\n",
      "Found new best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011417123679618333, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.0001139991054633333, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011455587677046536, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011509936251941176, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV score: 0.00011510995861245286, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.00011452153482808672, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011658284174959551, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011423913370431543, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011457178138601278, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 4, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.00011543189691404133, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011610932754520534, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.00011469866498564288, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.00011495607118486482, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.00011659799844084755, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.4}\n",
      "\n",
      "CV score: 0.00011684188710827838, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "\n",
      "CV score: 0.0001155643748809089, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.8}\n",
      "\n",
      "CV score: 0.0001150891914023831, current best score: 0.00011350220852811235\n",
      "---------------------------------------------------------------------------------------------------- \n",
      "params = \n",
      "{'colsample_bytree': 1, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 6, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 1}\n",
      "\n",
      "CV score: 0.0001174593981966976, current best score: 0.00011350220852811235\n",
      "\n",
      "\n",
      "best params: {'colsample_bytree': 0.9, 'eta': 0.03, 'eval_metric': 'rmse', 'max_depth': 5, 'nthread': 4, 'num_round': 1000, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.6}\n",
      "best score: 0.00011350220852811235\n"
     ]
    }
   ],
   "source": [
    "param_grid = [\n",
    "              {'silent': [1],\n",
    "               'nthread': [4],\n",
    "               'eval_metric': ['rmse'],\n",
    "               'eta': [0.03],\n",
    "               'objective': ['reg:linear'],\n",
    "               'max_depth': [4, 5, 6],\n",
    "               'num_round': [1000],\n",
    "               'subsample': [0.4, 0.6, 0.8, 1],\n",
    "               'colsample_bytree': [0.7, 0.9, 1],\n",
    "               }\n",
    "              ]\n",
    "\n",
    "xgb_best_params = find_best_params(pipe_data, xgb_predict, param_grid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __xgb 训练__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold n°1\n",
      "len trn_idx  1244\n",
      "[0]\ttrain-rmse:0.41223\tvalid_data-rmse:0.414495\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.00937\tvalid_data-rmse:0.012154\n",
      "[400]\ttrain-rmse:0.00738\tvalid_data-rmse:0.012149\n",
      "Stopping. Best iteration:\n",
      "[249]\ttrain-rmse:0.008639\tvalid_data-rmse:0.011963\n",
      "\n",
      "fold n°2\n",
      "len trn_idx  1244\n",
      "[0]\ttrain-rmse:0.412554\tvalid_data-rmse:0.411489\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009572\tvalid_data-rmse:0.010408\n",
      "[400]\ttrain-rmse:0.007492\tvalid_data-rmse:0.010257\n",
      "Stopping. Best iteration:\n",
      "[321]\ttrain-rmse:0.008054\tvalid_data-rmse:0.010154\n",
      "\n",
      "fold n°3\n",
      "len trn_idx  1244\n",
      "[0]\ttrain-rmse:0.412511\tvalid_data-rmse:0.412077\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009498\tvalid_data-rmse:0.012291\n",
      "[400]\ttrain-rmse:0.007413\tvalid_data-rmse:0.011734\n",
      "[600]\ttrain-rmse:0.00631\tvalid_data-rmse:0.011799\n",
      "Stopping. Best iteration:\n",
      "[481]\ttrain-rmse:0.006915\tvalid_data-rmse:0.011714\n",
      "\n",
      "fold n°4\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.412355\tvalid_data-rmse:0.413351\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009632\tvalid_data-rmse:0.010239\n",
      "[400]\ttrain-rmse:0.007535\tvalid_data-rmse:0.009976\n",
      "Stopping. Best iteration:\n",
      "[361]\ttrain-rmse:0.00781\tvalid_data-rmse:0.009915\n",
      "\n",
      "fold n°5\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.412679\tvalid_data-rmse:0.410576\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009628\tvalid_data-rmse:0.010779\n",
      "[400]\ttrain-rmse:0.007522\tvalid_data-rmse:0.010314\n",
      "[600]\ttrain-rmse:0.006444\tvalid_data-rmse:0.010495\n",
      "Stopping. Best iteration:\n",
      "[406]\ttrain-rmse:0.007485\tvalid_data-rmse:0.010309\n",
      "\n",
      "fold n°6\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.412697\tvalid_data-rmse:0.410264\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009618\tvalid_data-rmse:0.01148\n",
      "[400]\ttrain-rmse:0.007507\tvalid_data-rmse:0.011228\n",
      "Stopping. Best iteration:\n",
      "[298]\ttrain-rmse:0.008307\tvalid_data-rmse:0.011154\n",
      "\n",
      "fold n°7\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.412244\tvalid_data-rmse:0.414249\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009547\tvalid_data-rmse:0.010669\n",
      "[400]\ttrain-rmse:0.007386\tvalid_data-rmse:0.010475\n",
      "Stopping. Best iteration:\n",
      "[276]\ttrain-rmse:0.008424\tvalid_data-rmse:0.010363\n",
      "\n",
      "fold n°8\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.41229\tvalid_data-rmse:0.414101\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009677\tvalid_data-rmse:0.009724\n",
      "[400]\ttrain-rmse:0.007648\tvalid_data-rmse:0.00927\n",
      "[600]\ttrain-rmse:0.006596\tvalid_data-rmse:0.009145\n",
      "[800]\ttrain-rmse:0.005798\tvalid_data-rmse:0.009118\n",
      "[1000]\ttrain-rmse:0.005144\tvalid_data-rmse:0.009119\n",
      "Stopping. Best iteration:\n",
      "[934]\ttrain-rmse:0.00535\tvalid_data-rmse:0.009081\n",
      "\n",
      "fold n°9\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.412592\tvalid_data-rmse:0.411176\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009536\tvalid_data-rmse:0.01135\n",
      "[400]\ttrain-rmse:0.007511\tvalid_data-rmse:0.010768\n",
      "[600]\ttrain-rmse:0.006424\tvalid_data-rmse:0.010755\n",
      "Stopping. Best iteration:\n",
      "[534]\ttrain-rmse:0.006738\tvalid_data-rmse:0.010726\n",
      "\n",
      "fold n°10\n",
      "len trn_idx  1245\n",
      "[0]\ttrain-rmse:0.412429\tvalid_data-rmse:0.412775\n",
      "Multiple eval metrics have been passed: 'valid_data-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid_data-rmse hasn't improved in 200 rounds.\n",
      "[200]\ttrain-rmse:0.009541\tvalid_data-rmse:0.011636\n",
      "[400]\ttrain-rmse:0.007462\tvalid_data-rmse:0.010853\n",
      "[600]\ttrain-rmse:0.006416\tvalid_data-rmse:0.010884\n",
      "Stopping. Best iteration:\n",
      "[409]\ttrain-rmse:0.007404\tvalid_data-rmse:0.010834\n",
      "\n",
      "CV score: 0.00011350\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train, X_test, test_idx = split_data(pipe_data, target_name='target')\n",
    "min_max_scaler = MinMaxScaler()\n",
    "X_train = min_max_scaler.fit_transform(X_train)\n",
    "X_test = min_max_scaler.transform(X_test)\n",
    "oof_xgb, predictions_xgb = xgb_predict(X_train, y_train, X_test, params=xgb_best_params, verbose_eval=200) #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型融合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold 0\n",
      "fold 1\n",
      "fold 2\n",
      "fold 3\n",
      "fold 4\n",
      "fold 5\n",
      "fold 6\n",
      "fold 7\n",
      "fold 8\n",
      "fold 9\n",
      "0.00011094962448042872\n"
     ]
    }
   ],
   "source": [
    "# stacking\n",
    "train_stack = np.vstack([oof_lgb, oof_xgb]).transpose()\n",
    "test_stack = np.vstack([predictions_lgb, predictions_xgb]).transpose()\n",
    "\n",
    "folds_stack = RepeatedKFold(n_splits=5, n_repeats=2, random_state=4590)\n",
    "oof_stack = np.zeros(train_stack.shape[0])\n",
    "predictions = np.zeros(test_stack.shape[0])\n",
    "\n",
    "for fold_, (trn_idx, val_idx) in enumerate(folds_stack.split(train_stack, y_train)):\n",
    "    print(\"fold {}\".format(fold_))\n",
    "    trn_data, trn_y = train_stack[trn_idx], y_train[trn_idx]\n",
    "    val_data, val_y = train_stack[val_idx], y_train[val_idx]\n",
    "    \n",
    "    clf_3 = BayesianRidge()\n",
    "    clf_3.fit(trn_data, trn_y)\n",
    "    \n",
    "    oof_stack[val_idx] = clf_3.predict(val_data)\n",
    "    predictions += clf_3.predict(test_stack) / 10\n",
    "    \n",
    "final_score = mean_squared_error(y_train, oof_stack)\n",
    "print(final_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 生成提交结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __生成文件名__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result/testB_lgb_xgb_11095_8features_20190122_10:36:07.csv\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "model_name = \"lgb_xgb\"\n",
    "file_name = 'result/{}_{}_{:5.0f}_{}features_{}.csv'.format(\n",
    "                                        test_name, \n",
    "                                        model_name,\n",
    "                                        final_score*1e8, X_train.shape[1],\n",
    "                                        time.strftime('%Y%m%d_%H:%M:%S',time.localtime(time.time())))\n",
    "                                        \n",
    "print(file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ __写入文件__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_df = pd.read_csv(test_file_name, encoding = 'gb18030')\n",
    "sub_df = sub_df[['样本id', 'A1']]\n",
    "sub_df['A1'] = predictions\n",
    "sub_df['A1'] = sub_df['A1'].apply(lambda x:round(x, 3))\n",
    "sub_df.to_csv(file_name, header=0, index=0) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "toc": {
   "base_numbering": 1.0,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}