2021/04/EnsembleLearning/Task10 前向分布算法与梯度提升决策树.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**写在前面：**本节内容是 [Datawhale四月组队学习 - 集成学习（中）- CH4-集成学习之boosting -【Task10 前向分布算法与梯度提升决策树】](https://github.com/datawhalechina/team-learning-data-mining/tree/master/EnsembleLearning) 的学习笔记，对应CH4的第4、5节，学习周期3天"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-23T15:21:46.018234Z",
     "start_time": "2021-04-23T15:21:46.014209Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.datasets import make_friedman1,make_regression\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sklaern参数\n",
    "**参考资料：**\n",
    "- [scikit-learn 梯度提升树(GBDT)调参小结](https://www.cnblogs.com/pinard/p/6143927.html)\n",
    "- [机器学习系列(11)_Python中Gradient Boosting Machine(GBM）调参方法详解](https://blog.csdn.net/han_xiaoyang/article/details/52663170)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**GradientBoostingClassifier重要参数**：\n",
    "- `loss`:{'deviance','exponential'}, default='deviance'  \n",
    "    deviance:对数似然损失函数，对具有概率输出的分类的偏差；exponential:指数损失函数，退化为AdaBoost算法，只能用于二分类问题。\n",
    "- `learning_rate`:float, default=0.1  \n",
    "    学习率，即每个弱学习器的权重缩减系数α。\n",
    "- `n_estimators`:int, default=100  \n",
    "    弱学习器的个数，或者弱学习器的最大迭代次数。梯度提升对于过度拟合具有相当强的鲁棒性，因此大量提升通常会带来更好的性能。\n",
    "- `subsample`:float, default=1.0  \n",
    "    对每个学习器的子采样比例，可防止过拟合，选择<1的值会使方差减小偏差增大。\n",
    "- `criterion`:{'friedman_mse', 'mse', 'mae'}, default='friedman_mse'  \n",
    "    衡量划分质量的指标。'friedman_mse'通常是最好的，'mae'不推荐使用。\n",
    "- `min_samples_split`:int or float, default=2  \n",
    "    内部节点再划分所需最小样本数。可避免过拟合，但需要综合考虑样本数进行调整，避免出现欠拟合。\n",
    "- `min_samples_leaf`:int or float, default=1  \n",
    "    叶子节点最少样本数。可避免过拟合，但需要综合考虑样本数进行调整，避免出现欠拟合。需要注意的是，在不均等分类问题中(imbalanced class problems)，一般这个参数需要被设定为较小的值，因为大部分少数类别（minority class）含有的样本都比较小。\n",
    "- `min_weight_fraction_leaf`:float, default=0.0  \n",
    "    和`min_samples_leaf`相似，这里设置的是样本比例，和`min_samples_leaf`只需设置一个。\n",
    "- `max_depth`:int, default=3  \n",
    "    树的最大深度。 \n",
    "- `min_impurity_decrease`:float, default=0.0  \n",
    "    限制信息增益的大小。\n",
    "- `min_impurity_split`:float, default=None  \n",
    "    节点划分最小不纯度。(sklearn:不推荐使用，使用`min_impurity_decrease`代替)\n",
    "- `init`:estimator or 'zero', default=None  \n",
    "    一个estimator，用于计算初始预测。init必须提供fit和predict_proba\n",
    "- `random_state`:int, RandomState instance or None, default=None  \n",
    "    随机种子。\n",
    "- `max_features`:{'auto', 'sqrt', 'log2'}, int or float, default=None  \n",
    "    划分时考虑的最大特征数。'None'代表考虑所有特征；'log2'代表最多考虑$\\log_2N$个特征；'sqrt'和'auto'代表最多考虑$\\sqrt{N}$个特征；int代表考虑max_features个特征；float代表考虑int(max_features * n_features)个特征。会使方差减小和偏差增大。  \n",
    "    - 注意：在找到至少一个有效的节点样本分区之前，分割的搜索不会停止，即使它需要有效检查多个max_features要素也是如此。\n",
    "- `max_leaf_nodes`:int, default=None  \n",
    "    最大叶子节点数。\n",
    "- `warm_start`:bool, default=False  \n",
    "    是否热启动。\n",
    "- `validation_fraction`:float, default=0.1  \n",
    "    判断早停时划分为验证集的样本比例。\n",
    "- `n_iter_no_change`:int, default=None  \n",
    "    迭代`n_iter_no_change`次中`tol`没有得到改善时，将会早停。\n",
    "- `tol`:float, default=1e-4  \n",
    "    早停的容忍度。当`n_iter_no_change`次迭代中不能得到`tol`的改善时，将会早停。\n",
    "- `ccp_alpha`:non-negative float, default=0.0  \n",
    "    Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than `ccp_alpha` will be chosen. By default, no pruning is performed. See Minimal Cost-Complexity Pruning for details.  \n",
    "    \n",
    "    \n",
    "**GradientBoostingRegressor重要参数**：  \n",
    "- `loss`:{'ls', 'lad', 'huber', 'quantile'}, default='ls'  \n",
    "    'ls'指最小二乘回归，'lad'指最小绝对偏差，'huber'是二者结合，'quantile'允许分位数回归（用于`alpha`指定分位数）。\n",
    "- `learning_rate`:float, default=0.1  \n",
    "    学习率。\n",
    "- `n_estimators`:int, default=100  \n",
    "    弱学习器的个数，或者弱学习器的最大迭代次数。梯度提升对于过度拟合具有相当强的鲁棒性，因此大量提升通常会带来更好的性能。\n",
    "- `subsample`:float, default=1.0  \n",
    "    对每个学习器的子采样比例，可防止过拟合，选择<1的值会使方差减小偏差增大。\n",
    "- `criterion`:{'friedman_mse', 'mse', 'mae'}, default='friedman_mse'  \n",
    "    衡量划分质量的指标。'friedman_mse'通常是最好的，'mae'不推荐使用。\n",
    "- `min_samples_split`:int or float, default=2  \n",
    "    内部节点再划分所需最小样本数。可避免过拟合，但需要综合考虑样本数进行调整，避免出现欠拟合。\n",
    "- `min_samples_leaf`:int or float, default=1  \n",
    "    叶子节点最少样本数。可避免过拟合，但需要综合考虑样本数进行调整，避免出现欠拟合。\n",
    "- `min_weight_fraction_leaf`:float, default=0.0  \n",
    "- `max_depth`:int, default=3  \n",
    "    树的最大深度。 \n",
    "- `min_impurity_decrease`:float, default=0.0  \n",
    "    限制信息增益的大小。\n",
    "- `init`:estimator or 'zero', default=None  \n",
    "    一个estimator，用于计算初始预测。init必须提供fit和predict_proba\n",
    "- `random_state`:int, RandomState instance or None, default=None  \n",
    "    随机种子。\n",
    "- `max_features`:{'auto', 'sqrt', 'log2'}, int or float, default=None  \n",
    "    划分时考虑的最大特征数。'None'代表考虑所有特征；'log2'代表最多考虑$\\log_2N$个特征；'sqrt'和'auto'代表最多考虑$\\sqrt{N}$个特征；int代表考虑max_features个特征；float代表考虑int(max_features * n_features)个特征。会使方差减小和偏差增大。  \n",
    "    - 注意：在找到至少一个有效的节点样本分区之前，分割的搜索不会停止，即使它需要有效检查多个max_features要素也是如此。\n",
    "- `alpha`:float, default=0.9  \n",
    "    Huber损失函数和分位数损失函数的alpha分位数。仅当loss='huber'或loss='quantile'时。\n",
    "- `max_leaf_nodes`:int, default=None  \n",
    "    最大叶子节点数。\n",
    "- `warm_start`:bool, default=False  \n",
    "    是否热启动。\n",
    "- `validation_fraction`:float, default=0.1  \n",
    "    判断早停时划分为验证集的样本比例。\n",
    "- `n_iter_no_change`:int, default=None  \n",
    "    迭代`n_iter_no_change`次中`tol`没有得到改善时，将会早停。\n",
    "- `tol`:float, default=1e-4  \n",
    "    早停的容忍度。当`n_iter_no_change`次迭代中不能得到`tol`的改善时，将会早停。\n",
    "- `ccp_alpha`:non-negative float, default=0.0  \n",
    "    Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than `ccp_alpha` will be chosen. By default, no pruning is performed. See Minimal Cost-Complexity Pruning for details.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一些尝试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-23T15:21:04.961969Z",
     "start_time": "2021-04-23T15:21:04.909725Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.702628537100702"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = make_friedman1(n_samples=1200, random_state=0, noise=1.0)\n",
    "X_train, X_test = X[:200], X[200:]\n",
    "y_train, y_test = y[:200], y[200:]\n",
    "est = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1,\n",
    "    max_depth=2, random_state=0, loss='ls').fit(X_train, y_train)\n",
    "mean_squared_error(y_test, est.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-23T15:31:13.013529Z",
     "start_time": "2021-04-23T15:31:12.884492Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.440031029624667"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = make_regression(random_state=0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "reg = GradientBoostingRegressor(random_state=0)\n",
    "reg.fit(X_train, y_train)\n",
    "reg.score(X_test, y_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}