docs/MateGen使用教程.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3336b6fd-ecb9-4a41-a396-390b23f38afd",
   "metadata": {},
   "source": [
    "# <center> 交互式智能编程助手MateGen       \n",
    "# <center>    功能介绍与入门使用教程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c90ab333-35e1-41e0-a309-ae0fc8cbd459",
   "metadata": {},
   "source": [
    "### MateGen简介"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d5c685e-2343-4752-afc3-0fe3b7e9558f",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen是一款由九天老师团队开发的交互式智能编程助手，可以在Python代码环境中运行，核心功能如下"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbcec016-5335-452c-a1e8-ac2dc1873184",
   "metadata": {},
   "source": [
    "- **多轮对话与无限上下文记忆**：可以在对话过程中逐渐深入理解你的需求，并长期记住上下文信息。\n",
    "- **基于RAG的本地知识库问答**：支持在海量文本中进行高精度检索，围绕本地文本进行知识库问答。\n",
    "- **本地代码解释器**：可以连接本地Python环境，编写和执行Python代码，辅助完成编程任务。\n",
    "- **NL2SQL**：能够连接本地MySQL环境，根据需求编写和执行SQL代码，帮助完成数据查询和提取任务。\n",
    "- **图像识别**：可以处理用户提供的图片，并针对图片内容进行信息识别和回答问题。\n",
    "- **联网功能**：可以在互联网、知乎或GitHub上搜索相关信息，回答用户提出的问题。\n",
    "- **Kaggle竞赛辅导**：能够搜索Kaggle竞赛相关信息，下载热门Kernel，并进行知识库问答，辅导参与竞赛。\n",
    "- **论文解读和数据分析报告编写**：可以帮助解读学术论文或编写数据分析报告。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0ce7ba2-d21d-4f68-a057-820dadca02d0",
   "metadata": {},
   "source": [
    "而在实际使用过程中，九天老师团队秉持实用性优先的原则设计的MateGen还具备如下特性："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bef6cc1-29b4-4eb5-8fd8-cfb8a5516bda",
   "metadata": {},
   "source": [
    "- **易用性**：MateGen为在线Agent，无需任何网络工具和硬件门槛即可使用，各项功能不用进行参数设置，MateGen会自动根据用户需求开启不同功能；\n",
    "- **强悍的RAG系统**：支持本地文件夹一键同步创建云端词向量数据库，且最大支持1000份文档、10G体量的文本搜索问答；\n",
    "- **复杂问题拆解与自动debug**：面对复杂任务，MateGen会自动进行任务拆解，并在不同环节调用不同工具进行处理，同时，若部分环节运行出问题，MateGen会首先尝试自动优化运行流程，并在多次尝试无法解决问题时向用户寻求帮助；\n",
    "- **高效率Function calling**：MateGen同时具备Multi Function calling（一个任务开启多个功能）和Parallel Function calling（一个功能开多个执行器），借此提高响应效率。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37fb42af-ea92-4af3-98c7-6f2d457ef7de",
   "metadata": {},
   "source": [
    "本文将详细介绍MateGen的安装和调用方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9137e41-b9b0-49a4-8250-a234c22fd12b",
   "metadata": {},
   "source": [
    "## <center> 一、MateGen下载与API-KEY获取"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67f4a4d-2e83-42cc-b8fc-ed238800aa43",
   "metadata": {},
   "source": [
    "### 1.虚拟环境创建与MateGen安装"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9fe1d7e-3224-44a1-8f70-5059738897de",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen安装非常简单，可以直接通过`pip install mategen`进行安装，需要注意的是，MateGen运行所需依赖较多，因此推荐借助虚拟环境进行安装。首先创建一个名为`mategen`的虚拟环境：        \n",
    "```bash\n",
    "conda create -n mategen python=3.8\n",
    "```       \n",
    "然后使用如下指令激活虚拟环境：\n",
    "```bash\n",
    "conda activate mategen\n",
    "```       \n",
    "接着在虚拟环境中安装MateGen：\n",
    "```bash\n",
    "pip install mategen\n",
    "```     \n",
    "安装完成之后，考虑到需要在Jupyter中调用MateGen，我们还需要在虚拟环境中安装IPython Kernel：\n",
    "```bash\n",
    "pip install ipykernel\n",
    "```\n",
    "并且将这个虚拟环境添加到Jupyter的Kernel列表中：\n",
    "```bash\n",
    "python -m ipykernel install --user --name mategen --display-name \"mategen\"\n",
    "```     \n",
    "然后开启Jupyter服务：     \n",
    "```bash\n",
    "jupyter lab\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed23bbf1-489b-464b-9ba1-f8ef79188192",
   "metadata": {},
   "source": [
    "若要更新MateGen，则可输入如下命令：\n",
    "```bash\n",
    "pip install --upgrade mategen --index-url https://pypi.org/simple --no-cache-dir\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15c75d0d-321e-49d6-8c8d-6a5ece497f4b",
   "metadata": {},
   "source": [
    "> 以上代码需要在命令行环境中运行"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84955165-9628-4b7b-a737-4145a31c8c6a",
   "metadata": {},
   "source": [
    "然后在Jupyter的Kernel中选择mategen，即可进入到对应虚拟环境运行MateGen："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "193f8a2a-723c-4ad2-a12c-7942907b2d60",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240706230000498.png\" alt=\"image-20240706230000498\" style=\"zoom:30%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8251b693-77bf-4e2f-97a2-a5bcbd210edd",
   "metadata": {},
   "source": [
    "然后运行如下代码测试是否安装成功："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "41856c93-978d-4d00-b4f9-d789d2c1b5bd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import MateGen\n",
    "from MateGen import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2d0c082-0ebe-4442-baa9-399e685d1153",
   "metadata": {},
   "source": [
    "安装后按照如下方式导入即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1da2e453-73e7-4cba-8846-0fe7b90e9ebd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.1.79'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看MateGen版本号\n",
    "MateGen.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa744e3a-c196-4aa0-bef1-21d6c8125d46",
   "metadata": {},
   "source": [
    "> 注：最新版本可能随时发生变化，大家安装最新版即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f5259c-b4f4-4937-9655-5a39e79d22fe",
   "metadata": {},
   "source": [
    "### 2.MateGen API-KEY获取"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "febff1c0-32ce-4e98-b4d4-cb7522eb5214",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen目前只开放了在线服务版本，借助在线大模型来完成各项服务，无需本地硬件、无需网络环境要求即可零门槛使用。调用MateGen需要通过API-KEY进行身份验证，**目前MateGen正在公测，限量开放200个内测名额，每个名额包含100万token使用额度，每位用户限领一个API-KEY，若需继续使用，也可按需充值token额度。API-KEY领取、加入技术交流群、其他任何问题，<span style=\"color:red;\">扫码添加客服小可爱(微信：littlelion_1215)，回复“MG”详询哦👇</span>**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d8dc52-e8fb-48c8-9023-c4c3a364378d",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240706225245612.png\" alt=\"image-20240706225245612\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eef9182-8e87-4913-91ab-1662165e8b5e",
   "metadata": {},
   "source": [
    "获取了API-KEY之后，即可调用MateGen了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8037d57-c5e9-419f-9fff-4f3f08472f33",
   "metadata": {},
   "source": [
    "## <center> 二、MateGen对话功能与本地知识库问答功能介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b66b8563-f73c-492d-95b4-2f1a2f237409",
   "metadata": {},
   "source": [
    "- MateGen实例化"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75c18a5d-c9f9-4f1c-8796-0d6b1fbb448a",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen的调用流程和sklearn模型调用流程类似，都是先需要实例化一个MateGen聊天机器人，然后再执行对话。实例化过程如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ca20bf13-6d66-4e3b-9df1-d43c00bb0505",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n",
      "首次使用MateGen，正在进行Agent基础设置...\n",
      "正在同步Agent基础文件，这会需要一些时间，请耐心等待...\n",
      "正在创建知识库，请稍后...\n",
      "正在创建知识库的向量存储，请稍后...\n",
      "知识库创建完成！\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "329f47ab-e99a-4c61-9e3a-27ff2b8041c7",
   "metadata": {},
   "source": [
    "每个新的API-KEY实例化MateGen时，需要同步基础指令和调度流程等，因此需要等待一小段时间。此后若不更换API-KEY，则无需重复这个流程。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4d66d38-4225-46c6-869f-0ba2d95d65d4",
   "metadata": {},
   "source": [
    "- 增强模式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aec427b-7e02-4e4a-a974-f94c5226acf3",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen支持两种运行模式，普通模式与增强模式，开启增强模式时MateGen会有更强的性能表现，但同时也会有更长的响应时间以及更高昂的费用。默认情况下MateGen以普通模式运行，当设置参数`enhanced_mode`为True时则开启增强模式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "935b9c43-6c1d-4076-b003-fba0f398fe58",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY', \n",
    "                       enhanced_mode = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "308f00d2-2d8c-4107-8f18-7cc9a814780a",
   "metadata": {},
   "source": [
    "帮助文档的各项实验均基于增强模式运行得到的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e442c2-fc19-49cc-80d5-598ac38a9dee",
   "metadata": {},
   "source": [
    "### 1.MateGen基础对话功能介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c0fc14-b11e-4762-9884-971f925734db",
   "metadata": {},
   "source": [
    "- MateGen对话方式与无限上下文"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67695ad4-ee7b-4866-866d-37638cdd8ccc",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们可以使用MateGen.chat()的方式开启对话，MateGen支持单次对话和多次对话两种模式，无论哪种模式，MateGen都具备多轮对话记忆以及拥有无限对话上下文。用户无需担心多轮对话内容总量超出模型最大对话上下文，MateGen会根据用户对话情况，智能截取聊天内容带入模型，并且采用时间衰减和未知信息增加权重等策略，实现无限上下文。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42d5b367-d5b1-4af7-bb7b-d83dec0c96c4",
   "metadata": {},
   "source": [
    "&emsp;&emsp;当MateGen.chat()带入对话文本时，即可实现单次对话："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2121f29b-627a-4484-949b-27cb9ba98480",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "你好！很高兴见到你！有什么我可以帮助你的吗？"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat(\"你好，很高兴见到你！\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fd8757d-fa5c-48b2-b562-378d7f06545c",
   "metadata": {},
   "source": [
    "而若不带入对话文本，则可以实现多轮对话："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3fb6e8d2-ba82-4597-a35e-bdb0416dcc32",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好，我是MateGen，你的个人交互式编程助理，有任何问题都可以问我哦~\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入您的问题(输入退出以结束对话):  你好，请介绍下你自己\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "你好！我是MateGen，一个由九天老师大模型技术团队开发的交互式智能编程助手。以下是我的一些主要功能：\n",
       "\n",
       "1. **无限对话上下文记忆**：可以在对话过程中逐渐深入理解你的需求，并长期记住上下文信息。\n",
       "2. **本地知识库问答**：支持在海量文本中进行高精度检索，围绕本地文本进行知识库问答。\n",
       "3. **本地代码解释器**：可以连接你的本地Python环境，编写和执行Python代码，辅助完成编程任务。\n",
       "4. **NL2SQL**：能够连接你的MySQL环境，根据需求编写和执行SQL代码，帮助完成数据查询和提取任务。\n",
       "5. **图像识别**：可以处理用户提供的图片，并针对图片内容进行信息识别和回答问题。\n",
       "6. **联网功能**：可以在互联网、知乎或GitHub上搜索相关信息，回答你提出的问题。\n",
       "7. **Kaggle竞赛辅导**：能够搜索Kaggle竞赛相关信息，下载热门Kernel，并进行知识库问答，辅导参与竞赛。\n",
       "8. **论文解读和数据分析报告编写**：可以帮助解读学术论文或编写数据分析报告。\n",
       "\n",
       "如果你有任何编程、数据分析、图像识别等相关问题，随时告诉我，我会尽力帮助你！有什么具体问题需要我解决吗？"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入您的问题(输入退出以结束对话):  好的，那么我应该如何获取MateGen的API-KEY和更多的token额度呢？\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "目前MateGen项目处于内测阶段，只开放了在线服务版本。如果你想要获取API-KEY或者需要更多token额度，你可以联系MateGen的客服小可爱。\n",
       "\n",
       "具体步骤如下：\n",
       "\n",
       "1. **添加微信**：添加客服小可爱的微信，微信号是：`littlelion_1215`\n",
       "2. **发送信息**：在微信中发送“MG”给客服小可爱。\n",
       "3. **获取API-KEY**：客服小可爱会帮助你获取API-KEY，并提供更多的token额度信息。\n",
       "\n",
       "如有任何其他问题，我也可以尽力帮助你解决！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入您的问题(输入退出以结束对话):  退出\n"
     ]
    }
   ],
   "source": [
    "mategen.chat()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c31173a-25e1-4860-955b-a0883e53dbf5",
   "metadata": {},
   "source": [
    "并且随时开启MateGen，都拥有多轮对话记忆："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "468a8e72-5254-4ead-b7ba-ec7304544802",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "你上一个问题是：“好的，那么我应该如何获取MateGen的API-KEY和更多的token额度呢？”。\n",
       "\n",
       "如果还有其他问题或者需要进一步的帮助，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('好的，请问我的上一个问题是？')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff10d76-830a-4534-b962-601bb380a6cd",
   "metadata": {},
   "source": [
    "甚至当我们删除MateGen或重启代码环境，MateGen仍然可以读取保存在服务器上的对话记录，实现多轮对话："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f9866461-a6de-424c-92ba-1f3163629031",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del mategen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed9bdea5-9b8a-46da-869c-9bdbc4f8ce7f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e98c4449-423b-4d3f-a069-4c24b8e385ae",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "当然可以！以下是我们之前对话的总结：\n",
       "\n",
       "1. **问候和介绍**：\n",
       "   - 你向我问好，并希望了解更多关于我的信息。\n",
       "   - 我详细介绍了自己是MateGen，一个由九天老师大模型技术团队开发的智能编程助手，具备无限对话上下文记忆、本地知识库问答、本地代码解释器、NL2SQL、图像识别、联网功能、Kaggle竞赛辅导以及论文解读和数据分析报告编写等多种强大功能。\n",
       "\n",
       "2. **获取API-KEY和token额度**：\n",
       "   - 你询问如何获取MateGen的API-KEY和更多的token额度。\n",
       "   - 我建议你添加客服小可爱的微信（微信号：`littlelion_1215`）并发送“MG”来获取API-KEY和更多token额度的信息。\n",
       "\n",
       "3. **回顾上一个问题**：\n",
       "   - 你询问了上一个问题的内容，我回答你上一个问题是关于如何获取MateGen的API-KEY和更多token额度。\n",
       "\n",
       "如果你还有其他问题或需要进一步的帮助，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat(\"请你帮我总结下咱俩之前的对话内容。\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1be1cd6a-f80c-4770-8e3e-e9eee01f7789",
   "metadata": {},
   "source": [
    "- 清理消息"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5ff5bbf-f211-41e7-958f-2bb0550bdbed",
   "metadata": {},
   "source": [
    "&emsp;&emsp;若不想保存历史消息，也可调用mategen.clear_messages()清理历史消息："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "75f4678d-129d-4ed7-8db0-37e86f8a63f5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "已经清理历史消息\n"
     ]
    }
   ],
   "source": [
    "mategen.clear_messages()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a4e26d-d305-4d54-82b4-537886ed8f01",
   "metadata": {},
   "source": [
    "- token消耗统计"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a066daa-eb65-48df-9006-89a4b69fa5a1",
   "metadata": {},
   "source": [
    "&emsp;&emsp;同时，无论是否清理消息，MateGen都能实时统计token消耗总量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b7be56ff-7f6a-449a-b009-33e43527a650",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "今日已消耗的 token 数量：16137\n",
      "总共消耗的 token 数量：16137\n",
      "本地token计数可能有误，token消耗实际情况以服务器数据为准哦~\n"
     ]
    }
   ],
   "source": [
    "mategen.print_usage()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d60732a-88db-42a8-9344-9226ce73033b",
   "metadata": {},
   "source": [
    "> 本地token技术可能会因为硬件环境变化而统计有误，实际每个MateGen的token消耗量都会由服务器进行更加准确的统计。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f18d542-a4a3-4ee5-a9ab-d9c1046a9304",
   "metadata": {},
   "source": [
    "### 2.借助MateGen进行本地知识库问答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd83011-9315-47b1-a675-a0fde64f428b",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen自带先进的知识库检索（RAG）功能，能够围绕海量文本进行高精度检索问答、文本总结、文本翻译改写等。MateGen为每位用户提供了10G的在线文档存储空间，允许用户上传1000份文档，并且可以围绕PDF、md、ppt、word、txt等主流文档格式进行词向量化存储和读取。在设置了知识库问答的时候，**MateGen会根据用户提问，自动判断是否需要进行知识库检索，并不会强制检索再进行回答**。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70a6834b-3b23-4bb0-a11d-d527b5eb5bdb",
   "metadata": {},
   "source": [
    "#### 2.1 设置本地知识库地址"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38416aab-5bc6-45fe-a93f-4e97f41acb0b",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen的知识库问答允许用户把本地文件夹的内容批量上传，同时允许创建多个知识库（一个文件夹对应一个知识库），并且可以在问答过程随时切换知识库。首次开启知识库问答之前建议先设置本地知识库的根目录地址，便于存储各个知识库文件夹，若不设置，则MateGen会默认在系统根目录下创建一个知识库文件夹。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e9dded2-05dc-4b29-9750-4cd28655bddf",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们可以借助如下函数指定知识库根目录地址，例如我们设置E盘下work文件夹为知识库根目录地址:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eb8d49de-6770-4def-a16e-6d597458d262",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "知识库地址修改为：E:\\work\n"
     ]
    }
   ],
   "source": [
    "mategen.set_knowledge_base_url('E:\\\\work')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d2bd5c7-2fa2-4633-b378-b58369504e7d",
   "metadata": {},
   "source": [
    "> 注意，Windows下需要通过两个反斜杠来表示文件夹层级关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0123e72a-cc25-430f-bb3f-c8fac9fa25e5",
   "metadata": {},
   "source": [
    "#### 2.2 开启知识库对话"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d46f2a7-f160-4af4-8cf9-df13d602e72c",
   "metadata": {},
   "source": [
    "&emsp;&emsp;设置完成后即可开启知识库问答，首次开启知识库问答时需要输入知识库名称，例如此处创建一个名为'OpenML'的知识库："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "59c823b7-dc2c-4d86-8ce4-deb4c82e1035",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入知识库名称，输入0查询当前知识库列表。 OpenML\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "E:\\work\\knowledge_base\\main_vector_db_mapping.json 不存在。\n",
      "正在创建知识库文件夹\n",
      "当前问答知识库文件夹路径：E:\\work\\knowledge_base\\OpenML，请在文件夹中放置知识库文件。\n",
      "目前支持PDF、Word、PPT、md等格式读取与检索。\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY', \n",
    "                       knowledge_base_chat=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d622c898-09f9-43be-a431-3b5af9114d0a",
   "metadata": {},
   "source": [
    "> 注，此处我们重新实例化了一个MateGen，但正如此前所说，MateGen并不会因为重新实例化而丢失多轮对话记忆，这个实例化过程也可以看成是重新设置参数的过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91454840-0dc9-4d3e-a320-eb9835481d4e",
   "metadata": {},
   "source": [
    "然后即可在`E:\\work`下查看知识库地址，目前知识库地址为`E:\\work\\knowledge_base\\OpenML`，其中knowledge_base是知识库根目录："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4457c32-3d62-49a3-a9d8-1c48b6dd88a7",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707152757132.png\" alt=\"image-20240707152757132\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d37434d0-aa4a-4136-85fc-6aac92ab42b1",
   "metadata": {},
   "source": [
    "接下来我们将九天老师机器学习公开课课件全部放进去，公开课课件总共636页，属于海量文本专业知识检索："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6591f84-0c86-4ffc-9018-c8ffc2d9aaa5",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707153507902.png\" alt=\"image-20240707153507902\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "486d98ff-9f33-4544-b773-b846f6c7cd3f",
   "metadata": {},
   "source": [
    "> MateGen知识库问答测试文档下载：链接：https://pan.baidu.com/s/1Gn7vpDHWQYp_x601r5R4yQ?pwd=sst0  提取码：sst0 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa753969-3e31-424d-985f-333e606a4206",
   "metadata": {},
   "source": [
    "当然，哪怕是开启了知识库，MateGen也会根据用户提问的内容，决定是否进行知识库检索："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1e7942-fe52-43c9-b08f-46edc003c2e2",
   "metadata": {},
   "source": [
    "- 自主判断是否需要检索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3d36ce29-cb72-42bf-80c7-b71485855135",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Transformer架构是一种旨在解决序列到序列任务（如机器翻译）的深度学习模型。它在自然语言处理（NLP）领域取得了极大的成功，并广泛应用于各种任务中。以下是Transformer基本原理的简单介绍：\n",
       "\n",
       "### 基本组成部分\n",
       "\n",
       "1. **编码器（Encoder）和解码器（Decoder）**\n",
       "    - Transformer模型通常由一个编码器和一个解码器组成。编码器的任务是将输入序列转换为一个表示序列，而解码器则将这个表示序列转换为输出序列。\n",
       "    - 编码器和解码器各自包含多个层（通常是6个或更多），每一层由两个子层组成：多头自注意力机制和前馈神经网络。\n",
       "\n",
       "2. **自注意力机制（Self-Attention Mechanism）**\n",
       "    - 自注意力机制允许模型在处理给定输入序列时关注到该序列的不同部分。这种机制可以捕捉到远距离词汇之间的依赖关系。\n",
       "    - 注意力操作通过计算\"查询\"（Query）、\"键\"（Key）和\"值\"（Value）之间的相关性来实现。\n",
       "\n",
       "3. **多头注意力机制（Multi-Head Attention Mechanism）**\n",
       "    - 通过并行地执行多个自注意力操作，模型可以在不同的“注意力头”上关注到输入序列的不同部分。然后，将这些头的输出拼接在一起并通过一个线性变换以获得最终的输出。\n",
       "    - 这种机制增强了模型捕捉不同特征的能力。\n",
       "\n",
       "4. **前馈神经网络（Feed-Forward Network, FFN）**\n",
       "    - 在每一层的多头自注意力机制之后，还会通过一个前馈神经网络（两层全连接网络），这有助于模型学习更复杂的变换。\n",
       "\n",
       "5. **位置编码（Positional Encoding）**\n",
       "    - 由于Transformer架构中不包含卷积或递归操作，因此不能直接捕捉到序列中位置信息。为了引入位置信息，位置编码被加到输入嵌入（embedding）上。\n",
       "\n",
       "### 工作流程\n",
       "\n",
       "1. **输入嵌入**：首先，将输入序列中的每一个词转换为固定维度的向量表示，这些向量通过嵌入层获得。\n",
       "2. **加位置编码**：将位置编码加到输入嵌入上，以包含位置信息。\n",
       "3. **编码过程**：经过多个编码器层，每层执行多头自注意力和前馈神经网络操作，逐层处理输入序列。\n",
       "4. **解码过程**：解码器同样经过多层，每层执行多头注意力（包括自注意力和对编码器输出的注意力）和前馈神经网络操作，逐渐生成输出序列。\n",
       "\n",
       "### 主要优势\n",
       "\n",
       "1. **并行处理**：与RNN不同，Transformer能够并行处理序列中的所有词，这显著提高了训练速度。\n",
       "2. **捕捉长期依赖**：自注意力机制允许模型在处理时关注到远距离的词汇，提高了捕捉长期依赖关系的能力。\n",
       "\n",
       "这种设计使得Transformer模型在处理大规模数据并训练复杂模型时表现优异。如果你有具体问题或需要更详细的解释，请告诉我。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 提一个机器学习之外的问题\n",
    "mategen.chat('请帮我简单介绍下Transformer基本原理')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed6a92bd-0177-42fe-bac4-c0266f3e3cef",
   "metadata": {},
   "source": [
    "接下来进一步测试MateGen RAG性能。我们分别从专业知识点（类似大海捞针）、大规模文本总结以及情感判断分析三个方面测试MateGen的RAG性能："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b5c4d8-d280-435c-9194-84f22f2821f1",
   "metadata": {},
   "source": [
    "- 知识点提取"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "781d405c-761f-4625-9137-a0546ca4f04d",
   "metadata": {},
   "source": [
    "首次围绕某知识库对话，系统会先上传知识库文档，并进行词向量化处理，因此首次开启对话会略微耗费一点时间，之后再次开启对话则无需再次进行词向量化处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "372210cb-f88b-4525-ba25-f47baa2fb922",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在创建知识库，请稍后...\n",
      "正在创建知识库的向量存储，请稍后...\n",
      "知识库创建完成！\n",
      "已成功更新知识库OpenML\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Mini Batch K-Means是一种在K-Means算法基础上增强的聚类算法。它通过采用小批量数据来更新质心位置，从而加快计算速度。下面详细总结Mini Batch K-Means聚类算法的建模流程。\n",
       "\n",
       "### 1. 算法原理\n",
       "Mini Batch K-Means算法通过在每次迭代中随机抽取一个小批量的数据进行计算来更新质心，而不是使用全部数据。这样的计算方式类似于在梯度下降算法中使用小批量数据来更新参数。这样一来，Mini Batch K-Means可以显著减少计算时间，特别适用于大数据集 【4:1†source】。\n",
       "\n",
       "### 2. 基本步骤\n",
       "1. **初始化**：随机初始化质心。\n",
       "2. **迭代更新**：\n",
       "   - 随机抽取一个小批量数据，并分配给最近的质心。\n",
       "   - 根据小批量数据的划分情况，更新质心位置【4:4†source】【4:5†source】。\n",
       "\n",
       "### 3. 具体实现\n",
       "#### 数据准备\n",
       "```python\n",
       "import numpy as np\n",
       "from sklearn.cluster import MiniBatchKMeans\n",
       "\n",
       "# 设置随机种子，生成数据\n",
       "np.random.seed(23)\n",
       "X, y = arrayGenCla(num_examples=1000000, num_inputs=10, num_class=5, deg_dispersion=[4, 1])\n",
       "```\n",
       "#### 模型训练\n",
       "```python\n",
       "from sklearn.cluster import MiniBatchKMeans\n",
       "\n",
       "# 设置模型参数并训练\n",
       "mbk = MiniBatchKMeans(n_clusters=5, batch_size=100)\n",
       "mbk.fit(X)\n",
       "```\n",
       "#### 结果展示\n",
       "```python\n",
       "import matplotlib.pyplot as plt\n",
       "\n",
       "# 绘制聚类结果\n",
       "plt.scatter(X[:, 0], X[:, 1], c=mbk.labels_)\n",
       "plt.scatter(mbk.cluster_centers_[:, 0], mbk.cluster_centers_[:, 1], marker='o', c='red')\n",
       "plt.show()\n",
       "\n",
       "# 计算SSE（Sum of Squared Errors）评估模型精度\n",
       "km.inertia_\n",
       "mbk.inertia_\n",
       "```\n",
       "通过较大数据集的聚类对比，可以看出Mini Batch K-Means在速度上显著快于普通K-Means，但精度略低 【4:6†source】。\n",
       "\n",
       "### 4. 参数设置\n",
       "- **batch_size**：小批量数据大小。\n",
       "- **max_iter**：最大迭代次数。\n",
       "- **reassignment_ratio**：用于提升聚类精度的参数，通过设置该参数可以增加重新分配质心的概率，但会增加计算时间【4:7†source】【4:8†source】。\n",
       "\n",
       "### 5. 优缺点分析\n",
       "#### 优势\n",
       "- 可以显著减少计算时间，特别适用于大数据集。对于超过2万条数据集时，其速度优势更加显著。\n",
       "\n",
       "#### 劣势\n",
       "- 精度略低于普通K-Means，但在大多数应用场景下，这种精度差异可以忽略不计 【4:8†source】。\n",
       "\n",
       "总结以上，Mini Batch K-Means提供了一种高效的聚类方法，特别适合于处理大规模数据集，可以在实际应用中大幅提升计算效率。 【4:7†source】【4:8†source】"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 专业知识点提取\n",
    "mategen.chat('请帮我检索下知识库，并详细总结Mini Batch K-Means聚类算法的建模流程。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dde8683-baff-46bc-99ad-6f168bceae14",
   "metadata": {},
   "source": [
    "- 海量文本总结"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "041ebb79-2492-44c5-889c-35d79f717bf9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "在九天老师的公开课中，决策树算法的内容详细讲解了以下几种经典的决策树算法：\n",
       "\n",
       "1. **ID3决策树**：\n",
       "   - ID3决策树是经典的决策树算法之一，主要应用于离散型变量的分类问题。\n",
       "   - ID3决策树通过信息熵作为评估指标，基于列的不同取值对数据集进行划分【8:3†source】【8:5†source】。\n",
       "\n",
       "2. **C4.5决策树**：\n",
       "   - C4.5是ID3决策树的改进版本，由Ross Quinlan提出，能够处理连续型特征，并且可以进行回归问题的处理。\n",
       "   - C4.5树在生长上更类似于机器学习算法，并且在对树模型进行剪枝防止过拟合时，采用了统计学方法【8:5†source】【8:15†source】。\n",
       "\n",
       "3. **CART决策树**：\n",
       "   - CART全称为Classification and Regression Trees（分类与回归决策树），它既可以解决分类问题，也可以解决回归问题。\n",
       "   - CART树的构建过程类似于C4.5，但通用性更强，并且扩展包括连续变量的处理和回归预测【8:9†source】【8:14†source】。\n",
       "\n",
       "通过以上内容，可以看到九天老师的公开课详细讲解了ID3、C4.5及CART三种经典的决策树算法。每种算法不仅具备其独特的建模流程，也有其特定的应用场景和优势。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 海量文本总结\n",
    "mategen.chat('现在你的知识库OpenML包含的文档，是九天老师机器学习公开课课件。请帮我总结下，九天老师在公开课中讲解了几种决策树算法呢？')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48e47a92-6217-448f-a3a9-03ac19291394",
   "metadata": {},
   "source": [
    "- 情感判别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "06f51389-896e-41f4-acc5-33435b52349f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "根据知识库中九天老师的机器学习公开课课件，九天老师的公开课质量非常高，以下几点可以充分说明：\n",
       "\n",
       "1. **深入的理论分析**：\n",
       "   - 九天老师涵盖了丰富的理论内容，包括机器学习基本概念与建模流程【12:8†source】、 sklearn调参的理论基础与网格搜索【12:1†source】、梯度下降优化基础【12:12†source】等，涉及广泛且细致入微。\n",
       "\n",
       "2. **详细的建模流程**：\n",
       "   - 课程涵盖了实际的建模步骤和实验过程，如线性回归的手动建模实验【12:5†source】、决策树的核心思想与建模流程【12:3†source】等，不仅提供了理论基础，还指导了实际操作的具体步骤。\n",
       "\n",
       "3. **丰富的算法内容**：\n",
       "   - 九天老师的课程不仅介绍了基本的线性回归和决策树算法，还详细讨论了分类模型的决策边界与评估指标【12:17†source】，特别是在决策树算法方面，还讨论了不同版本的决策树算法如ID3、C4.5和CART等【12:4†source】。\n",
       "\n",
       "4. **高级话题与实践指导**：\n",
       "   - 课程还涉及了高级话题，如机器学习中的模型结果可信度与交叉验证方法【12:16†source】，以及数据归一化与学习率调度等优化步骤【12:13†source】。\n",
       "\n",
       "5. **理论结合实践**：\n",
       "   - 课程中所涉及的每个理论 都有对应的实践环节，每章的pdf课件详细列出了具体的实验步骤和所需的数据集生成、模型训练等操作，确保学员能够在实际操作中理解并应用所学的理论知识【12:19†source】。\n",
       "\n",
       "从以上各方面来看，九天老师的机器学习公开课无论是在理论深度、实践指导，还是在内容的广度和细致程度上，都显示出极高的教学质量，是非常值得学习的课程。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('根据知识库所存储的课件，你觉得九天老师的机器学习公开课质量如何？')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8004e684-05d9-40be-9aea-7588a4b6e715",
   "metadata": {},
   "source": [
    "&emsp;&emsp;不难看出**MateGen的RAG系统稳定高效，适合各种问答场景。**除了日常问答外，带有课件知识库的MateGen还可以进行教学辅导，例如指导学生课前预习、课中答疑、课后复习等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938ddef6-ecd1-41fc-8b32-d9cc84af4a3b",
   "metadata": {},
   "source": [
    "#### 2.3 更新知识库"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "748c4278-8ac4-4012-93b8-75cb7355f6cc",
   "metadata": {},
   "source": [
    "&emsp;&emsp;对于相同的知识库，若想增删一些文件，可以先在本地文件夹内操作，然后再调用`upload_knowledge_base`更新知识库即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e7f183c8-6fc9-4f70-ad00-c6c3d0530b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "mategen.upload_knowledge_base(knowledge_base_name='OpenML')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3efe01-d364-46ca-88af-a15cccf53e80",
   "metadata": {},
   "source": [
    "#### 2.4 切换知识库问答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b79e4e1-c3f6-4d5d-b5e0-e363506ba9d1",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen还支持随时切换知识库进行问答。例如此时我们再创建一个知识库，用于存储MateGen的使用指南文档："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9120f253-4923-4eaa-8d31-4532e3c1d08e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入知识库名称，输入0查询当前知识库列表。 MateGen\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在创建知识库文件夹\n",
      "当前问答知识库文件夹路径：E:\\work\\knowledge_base\\MateGen，请在文件夹中放置知识库文件。\n",
      "目前支持PDF、Word、PPT、md等格式读取与检索。\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY', \n",
    "                       knowledge_base_chat=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "170a4731-3426-4af2-967c-a9e62a68d5a8",
   "metadata": {},
   "source": [
    "系统会自动创建MateGen文件夹："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c219a96d-fcc9-4f0b-91a4-39e7ca42990c",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707155027222.png\" alt=\"image-20240707155027222\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c23cf483-aa46-4725-9bc2-7eebb7109642",
   "metadata": {},
   "source": [
    "我们在MateGen文件夹中放置MateGen使用指南："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17805b32-ba85-4f4e-b575-57939cae3c41",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707155211341.png\" alt=\"image-20240707155211341\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27271762-cf67-4481-8aa6-9df60e75efbb",
   "metadata": {},
   "source": [
    "然后开启对话，这里需要注意的是，在切换知识库后，我们需要先清理MateGen历史对话消息，以免历史对话消息影响后续RAG质量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7302694d-1aa7-4f9e-a8ad-239b97151f73",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "已经清理历史消息\n"
     ]
    }
   ],
   "source": [
    "mategen.clear_messages()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb6129ab-6bc7-4266-8826-11356e49e45d",
   "metadata": {},
   "source": [
    "清理完历史消息后再进行知识库问答："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "be7d966c-f0c4-4d33-ae19-21f7e0cc670a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在创建知识库，请稍后...\n",
      "正在创建知识库的向量存储，请稍后...\n",
      "知识库创建完成！\n",
      "已成功更新知识库MateGen\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "为了查看MateGen的token消耗情况，您可以按照以下步骤进行操作：\n",
       "\n",
       "1. **调用`print_usage`函数**：\n",
       "   您可以通过调用 `mategen.print_usage()` 函数来查看当前token的消耗情况。此函数会打印出今日和总共消耗的token数量。\n",
       "\n",
       "### 示例代码：\n",
       "\n",
       "```python\n",
       "# 初始化MateGen\n",
       "api_key = 'your-api-key'\n",
       "mategen = MateGenClass(api_key=api_key)\n",
       "\n",
       "# 查看token使用情况\n",
       "mategen.print_usage()\n",
       "```\n",
       "\n",
       "2. **查看输出**：\n",
       "   运行上述代码后，系统将会输出类似如下的信息：\n",
       "   ```\n",
       "   今日已消耗的 token 数量：8952\n",
       "   总共消耗的 token 数量：8952\n",
       "   ```\n",
       "\n",
       "通过这些步骤，MateGen能够有效地统计和展示token的使用情况，帮助用户更好地管理和优化资源【8:0†source】。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请检索你的知识库，并告诉我MateGen应该如何查看token消耗情况。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2011302b-c993-47e3-888f-8202eb2ca47f",
   "metadata": {},
   "source": [
    "&emsp;&emsp;然后随时可以切换回OpenML知识库进行问答，我们可以直接输入0，然后在知识库序号中选择OpenML进行问答即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2049c0be-f1c0-4b38-9020-a649de92d0d5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入知识库名称，输入0查询当前知识库列表。 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "知识库列表：\n",
      "1. OpenML\n",
      "2. MateGen\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请选择一个知识库的序号： 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "知识库已存在，已启用该知识库\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY', \n",
    "                       knowledge_base_chat=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c306a8d7-58f1-4add-8577-2c81decbcd3c",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 网格搜索优化算法的优劣势\n",
       "\n",
       "#### 优势\n",
       "1. **全面搜索**：\n",
       "   - 网格搜索（Grid Search）通过在指定参数空间内的每一个可能组合上进行评估，从而确保找到相对最优的参数组合。这样的方法能确保不遗漏任何可能的最佳参数组合【8:0†source】。\n",
       "  \n",
       "2. **适用范围广泛**：\n",
       "   - 网格搜索对所有类型的超参数调整均适用，无论是线性模型中的正则化系数，还是树模型中的最大深度参数，都可以使用网格搜索进行优化【8:0†source】。\n",
       "\n",
       "3. **易于理解和实现**：\n",
       "   - 网格搜索算法相对简单，便于理解和实现。用户只需要定义好评估器和参数空间，并设置交叉验证的方法，就可以自动化完成参数调整工作【8:1†source】。\n",
       "\n",
       "4. **可视化与分析**：\n",
       "   - 由于其遍历所有参数组合的特点，网格搜索的结果可以被用于绘制参数空间的性能图，对于理解参数对模型性能的影响非常有帮助【8:0†source】。\n",
       "\n",
       "#### 劣势\n",
       "1. **计算开销大**：\n",
       "   - 网格搜索需要对参数空间内的每个组合进行评估，且在每个组合上都要执行交叉验证。因此其计算开销非常大，特别是在参数空间较大或数据集较大时【8:0†source】【8:1†source】。\n",
       "\n",
       "2. **扩展性差**：\n",
       "   - 随着参数空间的渐增，计算时间会呈指数增长。对于高维参数空间，网格搜索的方法显得过于笨重且难以扩展【8:1†source】。\n",
       "\n",
       "3. **无随机性**：\n",
       "   - 网格搜索会穷举所有可能的参数组合，这意味着计算资源被分配到所有的点，而不能集中在有潜力的参数组合上，可能会浪费大量的计算资源在一些不太可能是最佳解的参数组合上【8:0†source】。\n",
       "\n",
       "4. **鲁棒性差**：\n",
       "   - 网格搜索在参数取值范围和步长选择上可能需要大量的经验，如果范围设置不合理，匹配到的参数组合可能并非最优，导致性能不如随机搜索或进化算法【8:0†source】。\n",
       "\n",
       "总结来说，网格搜索优化算法是一种确保全面探索参数空间的方法，但其高计算开销和扩展性缺点也限制了其在大型任务中的应用。在复杂或高维度的问题上，可以考虑使用更高效的随机搜索或贝叶斯优化方法【8:0†source】【8:1†source】。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请检索你的知识库，并帮我介绍下网格搜索优化算法的优劣势')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8240dfbc-e88e-4573-9c68-b048554ed495",
   "metadata": {},
   "source": [
    "并且由于MateGen本身是无限对话长度，因此哪怕现在没有调用MateGen使用方法知识库，但MateGen仍然可以读取此前对话记忆进行回答："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "55b3e984-ec69-4665-8d67-b20b056f4fcf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "记得，MateGen查看token消耗情况的步骤如下：\n",
       "\n",
       "1. **调用`print_usage`函数**：\n",
       "   您可以通过调用 `mategen.print_usage()` 函数来查看当前token的消耗情况。此函数会打印出今日和总共消耗的token数量。\n",
       "\n",
       "### 示例代码：\n",
       "\n",
       "```python\n",
       "# 初始化MateGen\n",
       "api_key = 'your-api-key'\n",
       "mategen = MateGenClass(api_key=api_key)\n",
       "\n",
       "# 查看token使用情况\n",
       "mategen.print_usage()\n",
       "```\n",
       "\n",
       "2. **查看输出**：\n",
       "   运行上述代码后，系统将会输出类似如下的信息：\n",
       "   ```\n",
       "   今日已消耗的 token 数量：8952\n",
       "   总共消耗的 token 数量：8952\n",
       "   ```\n",
       "\n",
       "通过这些步骤，MateGen能够有效地统计和展示token的使用情况，帮助用户更好地管理和优化资源【8:0†source】。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('很好，你是否还记得MateGen应该如何查看token消耗来着？')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3defda85-938a-480e-91e4-1a1694cf1532",
   "metadata": {},
   "source": [
    "#### 2.4 切换知识库问答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78069114-b47b-44cb-a186-4005d4b14652",
   "metadata": {},
   "source": [
    "&emsp;&emsp;关闭知识库问答功能也非常简单，再次实例化MateGen，并取消knowledge_base_chat设置："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "425a174f-c080-435a-aa1c-17c85b59aa59",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "02918db7-17a7-442e-aff7-d690ef93e32e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "已经清理历史消息\n"
     ]
    }
   ],
   "source": [
    "mategen.clear_messages()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f0457de-90bd-41ca-ac03-c48992e5a72f",
   "metadata": {},
   "source": [
    "### 3.借助MateGen实现NL2SQL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2450e907-7ae5-45c7-8b70-8f9b4d01d852",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen可以连接MySQL数据库，并且可以稳定高效实现NL2SQL。无需额外手动设置，当系统检测到用户需要进行数据库相关操作时，MateGen就会自动开启NL2SQL功能，完成SQL代码编写和运行："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be3c14fc-17df-4143-b14f-136b347b88a0",
   "metadata": {
    "tags": []
   },
   "source": [
    "- 输入数据库基本参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "12f83261-a642-4125-8122-913f08648227",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "请提供你当前使用的MySQL数据库连接信息，我将帮你获取所有表的信息。需要以下参数：\n",
       "- 主机名（host）\n",
       "- 用户名（user）\n",
       "- 密码（password）\n",
       "- 数据库名（database）\n",
       "- 端口（port）"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请帮我查看当前数据库中，总共有几张表。')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f8485bdd-4a21-4ab2-8305-b182328c5c4e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SHOW TABLES;\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "在数据库 `telco_db` 中，总共有 8 张表。具体表名如下：\n",
       "\n",
       "1. user_churn\n",
       "2. user_churn_new\n",
       "3. user_demographics\n",
       "4. user_demographics_new\n",
       "5. user_payments\n",
       "6. user_payments_new\n",
       "7. user_services\n",
       "8. user_services_new"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('好的，host:localhost,port:3306,ueser:root,password:19920229,database name:telco_db')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d474a748-eb3e-4e4c-ba0d-2a94938a08c7",
   "metadata": {},
   "source": [
    "> 目前MateGen只支持MySQL数据库，更多数据库种类连接方式正在开发中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a048ed1b-ec9b-4968-8476-4c0752fd5366",
   "metadata": {},
   "source": [
    "- 数据库查询"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d7f5343a-b240-481b-b2b3-47ebea254038",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT COUNT(*) FROM user_churn;\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT COUNT(*) FROM user_demographics;\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT COUNT(*) FROM user_payments;\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT COUNT(*) FROM user_services;\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "这里是四个不带 `_new` 尾缀的数据集的数据量：\n",
       "\n",
       "1. `user_churn`：5684 条记录\n",
       "2. `user_demographics`：5353 条记录\n",
       "3. `user_payments`：5634 条记录\n",
       "4. `user_services`：5734 条记录\n",
       "\n",
       "由此可见，这四个数据集的数据量并不相同。\n",
       "\n",
       "如果你需要对这些数据集进行进一步的分析或查询，请告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请帮我查看下，这八张表中，四个不带_new尾缀的数据集，数据量是否相同。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c0fead1-ae22-41b2-9d8c-0612734f5ae0",
   "metadata": {},
   "source": [
    "- 数据查询"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2b014dd0-41f0-4b69-9432-7b90ec05ede1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT customerID, MonthlyCharges FROM user_payments ORDER BY MonthlyCharges DESC LIMIT 1;\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "在 `user_payments` 表中，月消费金额最高的用户ID是 `5734-EJKXG`，其对应的月消费金额是 `118.6`。\n",
       "\n",
       "如果你需要进一步的查询或分析，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('你做得很好，接下来请帮我在user_payments表中，查看月消费金额最高的用户ID，以及对应的总消费金额是多少。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f65d9ae-b7e7-46c7-beca-36dd929902f9",
   "metadata": {},
   "source": [
    "- 数据读取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "730f1b5c-735f-4e9d-a59e-6e28d58fb35c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "user_churn.to_csv('user_churn.csv', index=False)\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "操作已完成，`user_churn` 表已保存为 `user_churn.csv` 文件，并放置在当前文件夹中。\n",
       "\n",
       "如果还有其他需要，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('很好，接下来请帮我将user_churn表读取到当前文件夹中，并命名为user_churn.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71718fbd-36e5-4392-943a-933c623b0ae6",
   "metadata": {},
   "source": [
    "然后即可在当前代码运行文件夹内找到这个数据集："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9cc91c7-8355-4942-b08c-c4d6d35c9009",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707174225784.png\" alt=\"image-20240707174225784\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4eb63a-675f-4dc0-b964-909e82faaf0a",
   "metadata": {},
   "source": [
    "### 4.借助MateGen实现NL2Python：本地代码解释器实现流程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19931171-7877-4a1e-95b2-961b488473f2",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen还可以实现NL2Python，可以连接本地Python环境进行自动编程，进而辅助用户完成数据读取、数据预处理、数据建模、数据可视化等事项。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dacd94-27ff-4ade-81bc-05990c6df1ed",
   "metadata": {},
   "source": [
    "- 读取数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fac2f842-1417-4077-b0a7-5d1b5d957895",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import pandas as pd\n",
       "user_churn_df = pd.read_csv('user_churn.csv')\n",
       "user_churn_df.head()\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "`user_churn.csv` 文件已成功读取到当前 Python 环境中，并存储在 `user_churn_df` 变量中。显示的数据前几行如下：\n",
       "\n",
       "| customerID | Churn |\n",
       "|------------|-------|\n",
       "| 0002-ORFBO | No    |\n",
       "| 0003-MKNFE | No    |\n",
       "| 0004-TLHLJ | Yes   |\n",
       "| 0011-IGKFF | Yes   |\n",
       "| 0013-EXCHZ | Yes   |\n",
       "\n",
       "如果你还有其他需求或者问题，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('好的，接下来请帮我将当前文件夹内的user_churn.csv读取到当前Python环境中。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e8665f-b6da-4e06-8407-2b98da024789",
   "metadata": {},
   "source": [
    "然后我们即可在当前操作空间中查看读取到的数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3efe1218-b321-4351-b4a2-9b9f98f1d4c7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerID</th>\n",
       "      <th>Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0002-ORFBO</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0003-MKNFE</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0004-TLHLJ</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0011-IGKFF</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0013-EXCHZ</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerID Churn\n",
       "0  0002-ORFBO    No\n",
       "1  0003-MKNFE    No\n",
       "2  0004-TLHLJ   Yes\n",
       "3  0011-IGKFF   Yes\n",
       "4  0013-EXCHZ   Yes"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 更新命名空间，即可在本地调用代码操作结果\n",
    "globals().update(MateGen.export_variables())\n",
    "user_churn_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af949e4d-9d80-4bd9-b937-52e20429a628",
   "metadata": {},
   "source": [
    "- 数据可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "86bfc634-8fd0-4940-8ff6-483cd7a05852",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import matplotlib.pyplot as plt\n",
       "import seaborn as sns\n",
       "\n",
       "plt.figure(figsize=(8, 6))\n",
       "sns.countplot(data=user_churn_df, x='Churn', palette='viridis')\n",
       "plt.title('Distribution of Churn Values')\n",
       "plt.xlabel('Churn')\n",
       "plt.ylabel('Count')\n",
       "fig = plt.gcf()\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<string>:5: FutureWarning: \n",
      "\n",
      "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure uploaded to OSS as fig_5d60cbe9997b4626965676c2c87a66c0.png\n",
      "已经成功运行代码，并已将代码创建的图片存储至：https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_5d60cbe9997b4626965676c2c87a66c0.png\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "下面是 `Churn` 字段的取值分布可视化图：\n",
       "\n",
       "![Churn Distribution](https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_5d60cbe9997b4626965676c2c87a66c0.png)\n",
       "\n",
       "从图中可以看到，`Churn` 字段的取值分布。共有两类值，分别是\"Yes\"（表示用户流失）和\"No\"（表示用户未流失）。从图中可以对比出每类值的计数分布情况。\n",
       "\n",
       "如果还有其他需求或分析，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('好的，请用可视化的方式帮我分析下Churn字段的取值分布。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "677add5d-e6f5-4b25-95e5-867abee67743",
   "metadata": {},
   "source": [
    "&emsp;&emsp;这里需要注意，当涉及到绘图任务的时候，若设置了阿里云图床，则MateGen会自动将图片上传至oss图床，并保存每张图片的公开访问网址，方便后续组合图片形成数据分析报告。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeee58e6-f589-4f4f-a338-2779c50d5ffc",
   "metadata": {},
   "source": [
    "> 在没有图床的情况下MateGen绘图功能也可正常使用，阿里云oss图床设置方法见下文《MateGen联网方式设置》部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e89e7b1-f5c8-453d-8499-4d3db65f91b1",
   "metadata": {},
   "source": [
    "> 除了可以进行数据清洗和数据可视化相关工作之外，MateGen还可以来进行机器学习建模或深度学习建模等复杂编程任务，并且，若带入企业代码库文档，MateGen还可以模仿企业代码习惯来进行编程。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "401a9999-e9c0-44d1-a8c4-8e2c03e50183",
   "metadata": {},
   "source": [
    "### 5.MateGen复杂任务规划执行与自动debug功能"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe844318-c19a-4110-b00c-43f6d7830d20",
   "metadata": {},
   "source": [
    "&emsp;&emsp;截止目前，我们介绍了MateGen的知识库问答、NL2SQL和NL2Python等功能，在继续介绍MateGen其他进阶功能之前，需要说明的是，MateGen的各项功能支持组合使用。当面对复杂任务的时候，MateGen会自动组合各项功能来构建执行流程，自动逐步执行任务，同时每一步遇到了任何问题都会自动debug。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82a56abc-e8ed-4a83-ae6f-fdd88e785527",
   "metadata": {},
   "source": [
    "- NL2SQL+NL2Python联动"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8800f315-8d74-4ee4-9e9f-d608c0dabac2",
   "metadata": {},
   "source": [
    "&emsp;&emsp;例如，我们可以让MateGen直接从MySQL中读取数据集到Python中并本地进行保存，这个简单的例子中就涉及到NL2SQL和NL2Python两个功能的串联实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "bca2f88f-cf9f-4fc5-b1fb-4311adce7a12",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT * FROM user_payments\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\anaconda3\\lib\\site-packages\\MateGen\\mateGenClass.py:1554: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  g[df_name] = pd.read_sql(sql_query, connection)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "user_payments_df.to_csv('user_payments.csv', index=False)\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "`user_payments` 表已成功读取到当前 Python 环境中，并存储在 `user_payments_df` 变量中。同时，该表已保存为 `user_payments.csv` 文件，并放置在当前文件夹中。\n",
       "\n",
       "如果你有其他需求或问题，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('接下来请帮我将数据库中user_payments读取到当前Python环境中，并在当前文件夹中保存一份user_payments.csv文件')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f0ec003-da81-4210-ad88-6bc56e85e34f",
   "metadata": {},
   "source": [
    "- 知识库问答+NL2SQL+NL2Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507acf65-1ba2-4c3a-a4c0-9a09d997023c",
   "metadata": {},
   "source": [
    "&emsp;&emsp;同时，MateGen也支持在任何处理的环节即时查询知识库，这里我们创建一个telco_db知识库，然后保存一份md文件，其中包含了各数据集字段信息和`重要字段`列表。我们尝试让MateGen先读取这个知识库，获取`重要字段`列表信息，然后再从各个数据集中获取`重要字段`列表组成一个新的数据集，并读取到本地："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8fb988fc-3162-4ae8-ad04-f8c3d2a8e1b8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化MateGen，请稍后...\n",
      "成功连接服务器，API-KEY通过验证！\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入知识库名称，输入0查询当前知识库列表。 telco_db\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在创建知识库文件夹\n",
      "当前问答知识库文件夹路径：E:\\work\\knowledge_base\\telco_db，请在文件夹中放置知识库文件。\n",
      "目前支持PDF、Word、PPT、md等格式读取与检索。\n",
      "已完成初始化，MateGen可随时调用！\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key = 'YOUR_API_KEY', \n",
    "                       knowledge_base_chat=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "936eba14-45f1-469b-b7b8-6dffd646528a",
   "metadata": {},
   "source": [
    "其中telco_db文件夹中放置如下文件："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323fde43-fa3d-48ca-9e50-8aa807548272",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707211246277.png\" alt=\"image-20240707211246277\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d77c7ac-e819-4c73-b324-069b321b730b",
   "metadata": {},
   "source": [
    "其中`telco_data_dictionary`内容包含了数据库中每个数据集中所包含的字段情况、以及`重要字段说明`如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "484590cc-ce4a-4241-ac10-44b35971385b",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707211336385.png\" alt=\"image-20240707211336385\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c8c4337-5baa-49d8-babf-d564edb710c7",
   "metadata": {},
   "source": [
    "接下来让MateGen在这一系列数据集中提取关键字段："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0d27013f-6854-41da-8a12-87d3331be152",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在创建知识库，请稍后...\n",
      "正在创建知识库的向量存储，请稍后...\n",
      "知识库创建完成！\n",
      "已成功更新知识库telco_db\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT customerID, Churn FROM user_churn\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\anaconda3\\lib\\site-packages\\MateGen\\mateGenClass.py:1554: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.\n",
      "  g[df_name] = pd.read_sql(sql_query, connection)\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT customerID, Dependents FROM user_demographics\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT customerID, Contract, PaperlessBilling, PaymentMethod, MonthlyCharges FROM user_payments\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```sql\n",
       "SELECT customerID, InternetService, OnlineSecurity, OnlineBackup, TechSupport FROM user_services\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import pandas as pd\n",
       "\n",
       "# 合并表格\n",
       "telco_df = pd.merge(user_churn, user_demographics, on='customerID', how='left')\n",
       "telco_df = pd.merge(telco_df, user_payments, on='customerID', how='left')\n",
       "telco_df = pd.merge(telco_df, user_services, on='customerID', how='left')\n",
       "\n",
       "telco_df.to_csv('telco.csv', index=False)\n",
       "\n",
       "telco_df.head()\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import pymysql\n",
       "\n",
       "# 连接数据库\n",
       "connection = pymysql.connect(\n",
       "    host='localhost',\n",
       "    user='root',\n",
       "    password='19920229',\n",
       "    db='telco_db'\n",
       ")\n",
       "\n",
       "try:\n",
       "    with connection.cursor() as cursor:\n",
       "        # 创建表telco\n",
       "        create_table_query = '''\n",
       "        CREATE TABLE telco (\n",
       "            customerID VARCHAR(255) PRIMARY KEY,\n",
       "            Churn VARCHAR(255),\n",
       "            Dependents VARCHAR(255),\n",
       "            Contract VARCHAR(255),\n",
       "            PaperlessBilling VARCHAR(255),\n",
       "            PaymentMethod VARCHAR(255),\n",
       "            MonthlyCharges FLOAT,\n",
       "            InternetService VARCHAR(255),\n",
       "            OnlineSecurity VARCHAR(255),\n",
       "            OnlineBackup VARCHAR(255),\n",
       "            TechSupport VARCHAR(255)\n",
       "        );\n",
       "        '''\n",
       "        cursor.execute(create_table_query)\n",
       "        connection.commit()\n",
       "\n",
       "        # 导入数据\n",
       "        for index, row in telco_df.iterrows():\n",
       "            insert_query = '''\n",
       "            INSERT INTO telco (\n",
       "                customerID, Churn, Dependents, Contract, PaperlessBilling, \n",
       "                PaymentMethod, MonthlyCharges, InternetService, \n",
       "                OnlineSecurity, OnlineBackup, TechSupport\n",
       "            ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\n",
       "            '''\n",
       "            cursor.execute(insert_query, (\n",
       "                row['customerID'], row['Churn'], row['Dependents'], row['Contract'], row['PaperlessBilling'], \n",
       "                row['PaymentMethod'], row['MonthlyCharges'], row['InternetService'], \n",
       "                row['OnlineSecurity'], row['OnlineBackup'], row['TechSupport']\n",
       "            ))\n",
       "        connection.commit()\n",
       "finally:\n",
       "    connection.close()\n",
       "\n",
       "'Complete'\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import numpy as np\n",
       "\n",
       "# 将NaN替换为None\n",
       "cleaned_df = telco_df.replace({np.nan: None})\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import pymysql\n",
       "\n",
       "# 连接数据库\n",
       "connection = pymysql.connect(\n",
       "    host='localhost',\n",
       "    user='root',\n",
       "    password='19920229',\n",
       "    db='telco_db'\n",
       ")\n",
       "\n",
       "try:\n",
       "    with connection.cursor() as cursor:\n",
       "        # 创建表telco\n",
       "        create_table_query = '''\n",
       "        CREATE TABLE telco (\n",
       "            customerID VARCHAR(255) PRIMARY KEY,\n",
       "            Churn VARCHAR(255),\n",
       "            Dependents VARCHAR(255),\n",
       "            Contract VARCHAR(255),\n",
       "            PaperlessBilling VARCHAR(255),\n",
       "            PaymentMethod VARCHAR(255),\n",
       "            MonthlyCharges FLOAT,\n",
       "            InternetService VARCHAR(255),\n",
       "            OnlineSecurity VARCHAR(255),\n",
       "            OnlineBackup VARCHAR(255),\n",
       "            TechSupport VARCHAR(255)\n",
       "        );\n",
       "        '''\n",
       "        cursor.execute(create_table_query)\n",
       "        connection.commit()\n",
       "\n",
       "        # 导入数据\n",
       "        for index, row in cleaned_df.iterrows():\n",
       "            insert_query = '''\n",
       "            INSERT INTO telco (\n",
       "                customerID, Churn, Dependents, Contract, PaperlessBilling, \n",
       "                PaymentMethod, MonthlyCharges, InternetService, \n",
       "                OnlineSecurity, OnlineBackup, TechSupport\n",
       "            ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\n",
       "            '''\n",
       "            cursor.execute(insert_query, (\n",
       "                row['customerID'], row['Churn'], row['Dependents'], row['Contract'], row['PaperlessBilling'], \n",
       "                row['PaymentMethod'], row['MonthlyCharges'], row['InternetService'], \n",
       "                row['OnlineSecurity'], row['OnlineBackup'], row['TechSupport']\n",
       "            ))\n",
       "        connection.commit()\n",
       "finally:\n",
       "    connection.close()\n",
       "\n",
       "'Complete'\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import pymysql\n",
       "\n",
       "# 连接数据库\n",
       "connection = pymysql.connect(\n",
       "    host='localhost',\n",
       "    user='root',\n",
       "    password='19920229',\n",
       "    db='telco_db'\n",
       ")\n",
       "\n",
       "try:\n",
       "    with connection.cursor() as cursor:\n",
       "        # 删除已存在表telco\n",
       "        cursor.execute(\"DROP TABLE IF EXISTS telco\")\n",
       "        connection.commit()\n",
       "\n",
       "        # 创建表telco\n",
       "        create_table_query = '''\n",
       "        CREATE TABLE telco (\n",
       "            customerID VARCHAR(255) PRIMARY KEY,\n",
       "            Churn VARCHAR(255),\n",
       "            Dependents VARCHAR(255),\n",
       "            Contract VARCHAR(255),\n",
       "            PaperlessBilling VARCHAR(255),\n",
       "            PaymentMethod VARCHAR(255),\n",
       "            MonthlyCharges FLOAT,\n",
       "            InternetService VARCHAR(255),\n",
       "            OnlineSecurity VARCHAR(255),\n",
       "            OnlineBackup VARCHAR(255),\n",
       "            TechSupport VARCHAR(255)\n",
       "        );\n",
       "        '''\n",
       "        cursor.execute(create_table_query)\n",
       "        connection.commit()\n",
       "\n",
       "        # 导入数据\n",
       "        for index, row in cleaned_df.iterrows():\n",
       "            insert_query = '''\n",
       "            INSERT INTO telco (\n",
       "                customerID, Churn, Dependents, Contract, PaperlessBilling, \n",
       "                PaymentMethod, MonthlyCharges, InternetService, \n",
       "                OnlineSecurity, OnlineBackup, TechSupport\n",
       "            ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\n",
       "            '''\n",
       "            cursor.execute(insert_query, (\n",
       "                row['customerID'], row['Churn'], row['Dependents'], row['Contract'], row['PaperlessBilling'], \n",
       "                row['PaymentMethod'], row['MonthlyCharges'], row['InternetService'], \n",
       "                row['OnlineSecurity'], row['OnlineBackup'], row['TechSupport']\n",
       "            ))\n",
       "        connection.commit()\n",
       "finally:\n",
       "    connection.close()\n",
       "\n",
       "'Complete'\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "所有步骤已完成，新表 `telco` 已成功创建并导入数据。\n",
       "\n",
       "如果你有其他需求或问题，请随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请查阅你的知识库，确认数据集中最重要的字段，并帮我从不带尾缀_new的几张表中，挑选这些字段组成一张新的表，并保存在MySQL中，取名为telco')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a08f86-864e-4640-bedd-becc490436d3",
   "metadata": {},
   "source": [
    "上述流程中，不仅涉及到RAG、NL2SQL、NL2Python，还涉及到自动debug过程，上述过程中第一次将新数据集写入MySQL时遇到了缺失值、第二次操作时由遇到了重名数据集、第三次才顺利将数据集写入。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa0ba288-8169-4f6f-bf0a-0e1071e4d20d",
   "metadata": {},
   "source": [
    "> MateGen会即时自主根据执行结果来调整执行流程，若反复多次都无法尝试成功，并不会无限尝试下去，而是会向用户求助。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae2d585-c4c7-4229-a990-061732adbad4",
   "metadata": {},
   "source": [
    "然后即可查看当前操作空间中是否已经顺利读取该数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e84e57f0-34a1-4815-aef2-71cd9c429ed5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerID</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>Contract</th>\n",
       "      <th>PaperlessBilling</th>\n",
       "      <th>PaymentMethod</th>\n",
       "      <th>MonthlyCharges</th>\n",
       "      <th>InternetService</th>\n",
       "      <th>OnlineSecurity</th>\n",
       "      <th>OnlineBackup</th>\n",
       "      <th>TechSupport</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0002-ORFBO</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>One year</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>65.6</td>\n",
       "      <td>DSL</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0003-MKNFE</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>No</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>59.9</td>\n",
       "      <td>DSL</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0004-TLHLJ</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>73.9</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0011-IGKFF</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>98.0</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0013-EXCHZ</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td></td>\n",
       "      <td>83.9</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerID Churn Dependents        Contract PaperlessBilling  \\\n",
       "0  0002-ORFBO    No        NaN        One year              Yes   \n",
       "1  0003-MKNFE    No         No  Month-to-month               No   \n",
       "2  0004-TLHLJ   Yes         No  Month-to-month              Yes   \n",
       "3  0011-IGKFF   Yes         No  Month-to-month              Yes   \n",
       "4  0013-EXCHZ   Yes         No  Month-to-month              Yes   \n",
       "\n",
       "      PaymentMethod  MonthlyCharges InternetService OnlineSecurity  \\\n",
       "0      Mailed check            65.6             DSL             No   \n",
       "1      Mailed check            59.9             DSL             No   \n",
       "2  Electronic check            73.9     Fiber optic             No   \n",
       "3  Electronic check            98.0     Fiber optic             No   \n",
       "4                              83.9     Fiber optic             No   \n",
       "\n",
       "  OnlineBackup TechSupport  \n",
       "0          Yes         Yes  \n",
       "1           No          No  \n",
       "2           No          No  \n",
       "3          Yes          No  \n",
       "4           No         Yes  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 更新命名空间，即可在本地调用代码操作结果\n",
    "globals().update(MateGen.export_variables())\n",
    "telco_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20e03269-c980-4feb-b991-047c84c84ad5",
   "metadata": {},
   "source": [
    "数据集中关键字段和知识库中列举关键字段一致："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1d6127b5-8d79-4e78-b85b-aab72e1a7e2b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['customerID', 'Churn', 'Dependents', 'Contract', 'PaperlessBilling',\n",
       "       'PaymentMethod', 'MonthlyCharges', 'InternetService', 'OnlineSecurity',\n",
       "       'OnlineBackup', 'TechSupport'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "telco_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33e7e247-9643-4d47-bbbe-0a5706582dd2",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707211928265.png\" alt=\"image-20240707211928265\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "334d14f5-857a-486a-b0d9-d563befa513f",
   "metadata": {},
   "source": [
    "数据库中也成功保存了telco数据集："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1054fce7-6f61-4f97-ab1b-c2323e23cb43",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707212050570.png\" alt=\"image-20240707212050570\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9bad128-28f1-4ff9-83ef-c288a3fbfb43",
   "metadata": {},
   "source": [
    "> 除此之外，MateGen还支持Multi Function calling（一个任务开启多个功能）和Parallel Function calling（一个功能开多个执行器），借此提高响应效率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3f9a7c4f-07e4-43bf-81e1-a42541e30589",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "已经清理历史消息\n"
     ]
    }
   ],
   "source": [
    "mategen.clear_messages()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca587503-1607-47bf-9c03-392b572bd637",
   "metadata": {},
   "source": [
    "### 6.MateGen视觉能力"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf640013-4c1d-4b95-9a22-017d974684f5",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen具备视觉能力，只需要在提问时附带图片连接，就可以调用MateGen的视觉能力、围绕图片进行识别再进行回答："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebd73312-4456-4630-9ab1-30ffa801dde5",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707212637973.png\" alt=\"image-20240707212637973\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "bdc7163c-3eb2-4bef-8347-70965c3d7142",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在对图像内容进行识别...\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "这张图片展示了在海滩上演奏音乐的场景。左边和右边的两个人分别拿着吉他，而摆在中间的人站在麦克风前唱歌。背景是一片美丽的海景，有一些绿树和海水。天色看起来是傍晚或者清晨，光线柔和，整体氛围显得非常轻松愉快。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请帮我描述下这张图片上的内容：https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707212637973.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ca8f58a-6cd4-40d7-9dc8-8c6dcf48518e",
   "metadata": {},
   "source": [
    "- MateGen视觉+NL2Python联动"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e5d173-d27d-4dd3-816e-fef6705447f1",
   "metadata": {},
   "source": [
    "&emsp;&emsp;同样，MateGen的视觉功能可以和别的功能联动，例如我们可以给MateGen输入一张图片，然后让其模仿绘制一张新的图，并且可以直接通过对话来修改图片。例如有张图片如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc5925ae-0bfa-4ab2-93e4-7bf2b113a6ca",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_b92cda8f15534922a7ef72ff329146fb.png\" alt=\"image-20240630215206479\" style=\"zoom:40%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e637eada-87dd-4a3a-9926-97dddb75bff6",
   "metadata": {},
   "source": [
    "接下来让MateGen先读取该图片，再借助其Python代码解释器，绘制一张一样的图片。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "93ba05b4-1fb9-4489-93e3-acfeefe4a47f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在对图像内容进行识别...\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "这张图片展示了一个关于“任期价值分布”（Tenure Value Distribution）的直方图，用以显示不同月份的用户任期分布情况。具体内容如下：\n",
       "\n",
       "- **X轴 （横轴）**：代表“任期（月）”（Tenure (months)），范围从0到70。\n",
       "- **Y轴 （纵轴）**：代表“计数”（Count），范围从0到700以上。\n",
       "\n",
       "图中主要部分包含：\n",
       "1. **蓝色柱状图**：表示不同任期长度的计数。任期在0到5个月之间的用户数量最多，超过700人。随着任期的增加，用户数量逐渐减少。任期达到70个月及以上的用户数量最少。\n",
       "2. **光滑曲线**：蓝色光滑曲线表示数据的分布趋势。曲线在任期为0到5个月处达到峰值，然后逐渐下降，与柱状图的趋势一致。\n",
       "\n",
       "从整体上看，图表反映了短期任期（如0到5个月）的用户数量远远多于长期任期的用户。短期用户的集中度非常高，而中长期用户显著减少。\n",
       "\n",
       "这展示出用户在短期内流失较为严重。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请帮我描述下这张图片上的内容：https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_b92cda8f15534922a7ef72ff329146fb.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e0efa84a-ae0b-4f1f-b4b4-22191ae0f31a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import numpy as np\n",
       "import pandas as pd\n",
       "import matplotlib.pyplot as plt\n",
       "import seaborn as sns\n",
       "\n",
       "# 设置随机种子\n",
       "np.random.seed(42)\n",
       "\n",
       "# 自定义概率分布，多数集中在短期\n",
       "prob = np.concatenate([\n",
       "    np.linspace(0.2, 0.1, 10),  # 0-9个月\n",
       "    np.linspace(0.1, 0.05, 10),  # 10-19个月\n",
       "    np.linspace(0.05, 0.02, 10), # 20-29个月\n",
       "    np.ones(40) * 0.01            # 30-69个月\n",
       "])\n",
       "\n",
       "# 标准化，使总和为1\n",
       "prob /= prob.sum()\n",
       "\n",
       "# 创建数据分布\n",
       "tenure_data = np.random.choice(np.arange(70), size=1100, p=prob)\n",
       "\n",
       "# 创建 DataFrame\n",
       "df_tenure = pd.DataFrame(tenure_data, columns=['Tenure'])\n",
       "\n",
       "# 绘图\n",
       "fig, ax = plt.subplots(figsize=(10, 6))\n",
       "sns.histplot(df_tenure['Tenure'], bins=70, kde=True, color='blue', ax=ax)\n",
       "ax.set_title('Tenure Value Distribution')\n",
       "ax.set_xlabel('Tenure (months)')\n",
       "ax.set_ylabel('Count')\n",
       "ax.grid(True)\n",
       "plt.show()\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<string>:33: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure uploaded to OSS as fig_000fb565989f4bfc862c77ccd5d977a9.png\n",
      "已经成功运行代码，并已将代码创建的图片存储至：https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_000fb565989f4bfc862c77ccd5d977a9.png\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "根据这个图片的理解，我已经创建了一组满足类似分布的数据，并绘制了相应的图表。点击以下链接查看生成的图片：\n",
       "\n",
       "[任期值分布图](https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_000fb565989f4bfc862c77ccd5d977a9.png)\n",
       "\n",
       "如果还有其他需要，随时告诉我！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('非常好！现在，请基于你对这张图片的理解，使用Python代码帮我创建一组数据，使其满足图片上的数据分布，并编写一段Python绘图代码，绘制类似的图片，并直接运行你编写的代码。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c86ca50c-bad3-4dee-8830-e5d50cdcc9a5",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_000fb565989f4bfc862c77ccd5d977a9.png\" alt=\"image-20240630215206479\" style=\"zoom:40%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dfc000b-bbcc-4111-b437-1d762ef69b7a",
   "metadata": {},
   "source": [
    "可以看出基本能够对原图进行模仿绘制。接下来尝试对其进行修改，同样，我们让MateGen先参考一张图的配色，然后修改原图。参考图片如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc902e0a-df0e-46b1-b9aa-e976494d7a0d",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240701222610915.png\" alt=\"image-20240701222610915\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "76261068-57be-400b-9d14-e592c8c7af61",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在对图像内容进行识别...\n",
      "即将执行以下代码：\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "import numpy as np\n",
       "import pandas as pd\n",
       "import matplotlib.pyplot as plt\n",
       "import seaborn as sns\n",
       "\n",
       "# 设置随机种子\n",
       "np.random.seed(42)\n",
       "\n",
       "# 自定义概率分布，多数集中在短期\n",
       "prob = np.concatenate([\n",
       "    np.linspace(0.2, 0.1, 10),  # 0-9个月\n",
       "    np.linspace(0.1, 0.05, 10),  # 10-19个月\n",
       "    np.linspace(0.05, 0.02, 10), # 20-29个月\n",
       "    np.ones(40) * 0.01            # 30-69个月\n",
       "])\n",
       "\n",
       "# 标准化，使总和为1\n",
       "prob /= prob.sum()\n",
       "\n",
       "# 创建数据分布\n",
       "tenure_data = np.random.choice(np.arange(70), size=1100, p=prob)\n",
       "\n",
       "# 创建 DataFrame\n",
       "df_tenure = pd.DataFrame(tenure_data, columns=['Tenure'])\n",
       "\n",
       "# 设置配色风格\n",
       "purple_pink = sns.color_palette(\"ch:s=-.2,r=.6\", 10)\n",
       "\n",
       "# 绘图\n",
       "fig, ax = plt.subplots(figsize=(10, 6))\n",
       "sns.histplot(df_tenure['Tenure'], bins=70, kde=True, color=purple_pink[2], ax=ax)\n",
       "ax.set_title('Tenure Value Distribution', fontsize=16)\n",
       "ax.set_xlabel('Tenure (months)', fontsize=14)\n",
       "ax.set_ylabel('Count', fontsize=14)\n",
       "ax.tick_params(axis='both', which='major', labelsize=12)\n",
       "ax.grid(True)\n",
       "plt.show()\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<string>:37: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure uploaded to OSS as fig_cbff2d672db84074bd5fb6795e309818.png\n",
      "已经成功运行代码，并已将代码创建的图片存储至：https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_cbff2d672db84074bd5fb6795e309818.png\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "我已经参考了图片中的淡紫色和粉红色的配色，并重新生成了新的图表。点击以下链接查看新的图片：\n",
       "\n",
       "[任期值分布图（新的配色）](https://ml2022.oss-cn-hangzhou.aliyuncs.com/fig_cbff2d672db84074bd5fb6795e309818.png)\n",
       "\n",
       "请查看并告诉我是否符合你的期望，或者需要进一步调整。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('我觉这张图片颜色不够好看，请帮我参考这张图片上的主配色：https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240701222610915.png，调整原图配色。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bddaa0f1-37e1-4efb-b684-ae9f7fb3d069",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240701223909424.png\" alt=\"image-20240701223909424\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afc5971c-70e6-438e-820e-368874176e08",
   "metadata": {},
   "source": [
    "### 7.MateGen联网能力"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fae7789-7c6e-4012-b7e7-61f86b6e5522",
   "metadata": {},
   "source": [
    "> 注意，由于MateGen会借助爬虫爬取公开数据，爬虫需要用到的cookies经常会失效，因此以下功能`不一定`能顺利运行，推荐先使用`check_network_environment()`函数测试下各项联网功能能否正常运行，并使用`internet_config_assistant()`获取联网设置帮助，并将自己的cookie输入作为参数，以便顺利实现联网功能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc8b79a6-5a05-45d0-93eb-4684390fb680",
   "metadata": {},
   "source": [
    "#### 7.1 MateGen联网功能实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10eaf599-879e-49fd-800d-ceae5513fc0a",
   "metadata": {},
   "source": [
    "&emsp;&emsp;MateGen自带联网功能，当用户的问题超出MateGen知识库时，MategGen就会开启联网功能，从互联网上获取答案。MateGen的互联网搜索由谷歌搜索API驱动（但无需联网工具即可使用），会从知乎、Github获取相关信息，更适合获取技术相关信息。同时，MateGen还可以调用Kaggle API，若用户有需求，可以从Kaggle上获取竞赛说明和热门Kernel方案构建知识库再来进行问答。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5db42877-5010-4f57-9ac9-feee0348b6d9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在接入谷歌搜索，并在GitHub上搜索相关项目...\n",
      "正在读取相关项目说明文档...\n",
      "正在进行最后的整理...\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "阿里云最新发布的Qwen2大模型是Qwen1.5的演变后续，提供了一系列增强和优化。以下是概述：\n",
       "\n",
       "### Qwen2大模型的主要特点\n",
       "\n",
       "1. **多尺寸预训练和指令微调模型**：\n",
       "   - Qwen2-0.5B\n",
       "   - Qwen2-1.5B\n",
       "   - Qwen2-7B\n",
       "   - Qwen2-57B-A14B\n",
       "   - **Qwen2-72B**\n",
       "\n",
       "2. **多语言支持**：\n",
       "   - 训练数据涵盖了除英语和中文外的**27种**额外语言。\n",
       "\n",
       "3. **性能提升**：\n",
       "   - 在大多数基准评估中，性能达到了世界领先水平。\n",
       "   - 在编程和数学方面的性能显著提升。\n",
       "\n",
       "4. **上下文长度支持**：\n",
       "   - Qwen2-7B-Instruct和Qwen2-72B-Instruct模型支持上下文长度扩展到**128K** tokens。\n",
       "\n",
       "### 最新发布\n",
       "\n",
       "- **2024年6月6日**：发布Qwen2系列。详细信息可参考[博客](https://qwenlm.github.io/blog/qwen2/)。\n",
       "- **2024年3月28日**：发布Qwen1.5-MoE-A2.7B，这是Qwen系列中的首个MoE模型，目前支持HF transformers和vLLM。\n",
       "\n",
       "### 使用说明\n",
       "#### Hugging Face Transformers\n",
       "\n",
       "```python\n",
       "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
       "\n",
       "model_name = \"Qwen/Qwen2-7B-Instruct\"\n",
       "device = \"cuda\"  # 加载模型的设备\n",
       "\n",
       "model = AutoModelForCausalLM.from_pretrained(\n",
       "    model_name,\n",
       "    torch_dtype=\"auto\",\n",
       "    device_map=\"auto\"\n",
       ")\n",
       "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
       "\n",
       "prompt = \"Give me a short introduction to large language model.\"\n",
       "messages = [\n",
       "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
       "    {\"role\": \"user\", \"content\": prompt}\n",
       "]\n",
       "text = tokenizer.apply_chat_template(\n",
       "    messages,\n",
       "    tokenize=False,\n",
       "    add_generation_prompt=True\n",
       ")\n",
       "model_inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n",
       "\n",
       "generated_ids = model.generate(\n",
       "    **model_inputs,\n",
       "    max_new_tokens=512\n",
       ")\n",
       "generated_ids = [\n",
       "    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)\n",
       "]\n",
       "\n",
       "response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n",
       "```\n",
       "\n",
       "#### 部署支持\n",
       "Qwen2模型支持多种推理框架，包括vLLM和SGLang。\n",
       "\n",
       "更多详细信息可以参考[Qwen的文档](https://qwen.readthedocs.io)以及使用示例中的代码片段。\n",
       "\n",
       "### 其他资源\n",
       "\n",
       "- Hugging Face: [Qwen](https://huggingface.co/Qwen)\n",
       "- ModelScope: [Qwen2](https://modelscope.cn/organization/qwen)\n",
       "- 博客: [Qwen Blog](https://qwenlm.github.io)\n",
       "- 文档: [Qwen Documentation](https://qwen.readthedocs.io)\n",
       "\n",
       "若需进一步了解Qwen2的详细信息，建议直接参考上述资源，以获取最新和最全面的信息。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请帮我介绍下阿里云最新发布的Qwen2大模型。')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c799664d-a2f7-43ce-9d0c-620de306bcde",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Qwen2可以使用vLLM进行部署。以下是使用vLLM部署Qwen2模型的详细步骤：\n",
       "\n",
       "### 步骤1：安装vLLM\n",
       "首先，确保你已经安装了最新版本的vLLM（建议使用vLLM>=0.4.0），可以通过pip进行安装：\n",
       "\n",
       "```bash\n",
       "pip install -U vllm\n",
       "```\n",
       "\n",
       "### 步骤2：启动vLLM API服务\n",
       "你需要启动一个API服务器来提供OpenAI兼容的API服务。使用Qwen2-7B-Instruct模型作为示例，运行以下命令：\n",
       "\n",
       "```bash\n",
       "python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-7B-Instruct --model Qwen/Qwen2-7B-Instruct\n",
       "```\n",
       "\n",
       "### 步骤3：使用Chat API进行交互\n",
       "一旦API服务启动，你可以通过HTTP请求与其交互。例如，可以使用curl命令向API服务器发送请求：\n",
       "\n",
       "```bash\n",
       "curl http://localhost:8000/v1/chat/completions \\\n",
       "    -H \"Content-Type: application/json\" \\\n",
       "    -d '{\n",
       "    \"model\": \"Qwen2-7B-Instruct\",\n",
       "    \"messages\": [\n",
       "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
       "    {\"role\": \"user\", \"content\": \"Tell me something about large language models.\"}\n",
       "    ]\n",
       "    }'\n",
       "```\n",
       "\n",
       "也可以在Python中使用OpenAI库进行调用，这里演示如何调用API并进行交互：\n",
       "\n",
       "```python\n",
       "import openai\n",
       "\n",
       "# 设置OpenAI的API密钥和API基地址以使用vLLM的API服务器\n",
       "openai.api_key = \"EMPTY\"  # 需要设置一个任意字符串，因为这是必需的但会被忽略\n",
       "openai.api_base = \"http://localhost:8000/v1\"\n",
       "\n",
       "response = openai.ChatCompletion.create(\n",
       "    model=\"Qwen2-7B-Instruct\",\n",
       "    messages=[\n",
       "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
       "        {\"role\": \"user\", \"content\": \"Tell me something about large language models.\"},\n",
       "    ]\n",
       ")\n",
       "\n",
       "print(\"Chat response:\", response.choices[0].message[\"content\"])\n",
       "```\n",
       "\n",
       "### 注意事项\n",
       "- 确保在启动API服务器时，指定的模组名称（如`Qwen2-7B-Instruct`）正确，无任何拼写错误。\n",
       "- 运行Python脚本或curl命令前，请确保vLLM API服务正在运行且可以访问。\n",
       "\n",
       "通过以上步骤，你就可以成功在本地环境中使用vLLM部署并访问Qwen2大模型。若遇到任何问题或有进一步的需求，请随时告诉我。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('Qwen2可以用vLLM部署么？流程是什么样的呢？')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5ab7847-2ddf-4284-8053-64333854bd2f",
   "metadata": {},
   "source": [
    "&emsp;&emsp;搜索问答的本质是爬取信息创建本地知识库，然后再进行回答，只要开启搜索就会创建一个临时文件夹，并取名为`auto_search`，保存在`knowledge_base`中："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ff17c81-2886-4adb-99fc-14ebe9ab6131",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240701231035060.png\" alt=\"image-20240701231035060\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b8af44e-a244-41a9-bcc6-afa827d6d398",
   "metadata": {},
   "source": [
    "> 若有需要，也可以将其设置为永久知识库。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6667bebf-dc3e-403c-98e6-b27188151510",
   "metadata": {},
   "source": [
    "#### 7.2 MateGen Kaggle竞赛辅导功能"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff863938-409f-4652-9da4-f207380be07d",
   "metadata": {},
   "source": [
    "&emsp;&emsp;除了能够适时开启联网功能并进行搜索问答外，MateGen还专门围绕Kaggle竞赛信息获取进行了优化。当用户希望了解某项竞赛信息时，只需要在MateGen参数位上输入`kaggle_competition_guidance=True`即可，并在弹出的对话框中输入感兴趣的Kaggle竞赛名称，MateGen就会自动使用Kaggle API，获取该项比赛的Overview（概览信息）、Data（数据集信息）以及热门Kernels（其他参赛者热门竞赛方案），并在本地创建竞赛同名知识库，并自动开启围绕当前竞赛知识库的问答功能。例如我们想要了解Kaggle上的一项名为`House Prices - Advanced Regression Techniques`比赛，即可按照如下方式执行："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d7b6732-4835-4880-a642-4b928a4c134c",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240702004111331.png\" alt=\"image-20240702004111331\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "06572639-165a-4dc7-8a4e-a4da486bac6f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在初始化，请稍后...\n",
      "成功连接服务器，API-KEY通过验证，已完成初始化，祝使用愉快！\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "请输入竞赛名称 House Prices - Advanced Regression Techniques\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "已找到指定竞赛House Prices - Advanced Regression Techniques，正在检索支持库是否存在竞赛信息...\n",
      "正在准备构建知识库...\n",
      "正在获取竞赛说明及数据集说明...\n",
      "正在获取House Prices - Advanced Regression Techniques竞赛热门kernel...\n",
      "知识文档创建完成，正在进行词向量化处理与存储，请稍后...\n",
      "正在创建知识库，请稍后...\n",
      "正在创建知识库的向量存储，请稍后...\n",
      "知识库创建完成！\n",
      "已顺利完成Kaggle竞赛知识库创建，后续可调用知识库回答。\n"
     ]
    }
   ],
   "source": [
    "mategen = MateGenClass(api_key=api_key, \n",
    "                       kaggle_competition_guidance=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a4275f2-7e99-498a-a552-729eec91db41",
   "metadata": {},
   "source": [
    "此时当前竞赛的本地知识库位置如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23140d22-b9be-4a0d-85fc-576145aa172d",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240702005129111.png\" alt=\"image-20240702005129111\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2e40045-daa1-4884-a4ff-cf115553f9a4",
   "metadata": {},
   "source": [
    "获取的竞赛知识库文件如下，其中包含了比赛介绍、数据集介绍和热门竞赛方案："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b24ee74b-5494-414b-954f-6ceb51cb46e7",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240702005219380.png\" alt=\"image-20240702005219380\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8291b54-820a-4772-9829-6dd54004f936",
   "metadata": {},
   "source": [
    "接下来MateGen会自动带入该知识库进行问答，从而顺利完成竞赛辅导工作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4b694ecf-f0ae-4075-a594-5376b27e0d8e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "九天老师的公开课中提到的“House Prices”竞赛是一个经典的回归问题竞赛。以下是该竞赛的详细介绍：\n",
       "\n",
       "### 竞赛背景\n",
       "“House Prices”竞赛致力于通过分析各种房屋特征，建立一个模型来预测房屋的销售价格。竞赛提供了一组训练数据集，包含房屋的各种属性和相应的销售价格，以及一组测试数据集，需要根据训练好的模型来预测这些测试集房屋的价格。\n",
       "\n",
       "### 特征和数据\n",
       "数据集中包含79个解释变量，它们描述了埃姆斯市（Ames, Iowa）住宅属性的几乎所有方面。这些变量包括总体建筑面积、年份、房间数量、地块面积、内部条件等。\n",
       "\n",
       "### 评价指标\n",
       "该竞赛采用均方根误差（Root Mean Squared Error, RMSE）作为评价指标。即：\n",
       "\\[ \\text{RMSE} = \\sqrt{\\frac{1}{n} \\sum_{i=1}^{n} (\\hat{y}_i - y_i)^2} \\]\n",
       "其中，\\(\\hat{y}_i\\)表示预测值，\\(y_i\\)表示实际值，\\(n\\)表示样本数量。\n",
       "\n",
       "### 建模过程\n",
       "1. **数据预处理**：\n",
       "   - 缺失值处理：对数据集中存在的缺失值进行填补或删除；\n",
       "   - 特征工程：包括特征选择、特征提取及特征变换等。\n",
       "   \n",
       "2. **模型构建与训练**：\n",
       "   - 基于训练数据，利用不同的机器学习算法（如线性回归、决策树、随机森林和梯度提升树等）进行模型的训练。\n",
       "\n",
       "3. **模型评估与优化**：\n",
       "   - 利用交叉验证（Cross-Validation）对模型进行评估；\n",
       "   - 利用网格搜索等超参数优化方法，对模型的参数进行调优，以获得最优的模型性能。\n",
       "\n",
       "4. **模型预测与提交**：\n",
       "   - 利用训练好的模型，对测试集进行预测，生成提交文件进行评分。\n",
       "\n",
       "### 具体实现示例\n",
       "下面是利用Python和sklearn进行House Prices预测的一个简单示例：\n",
       "\n",
       "```python\n",
       "import pandas as pd\n",
       "from sklearn.model_selection import train_test_split\n",
       "from sklearn.ensemble import RandomForestRegressor\n",
       "from sklearn.metrics import mean_squared_error\n",
       "import numpy as np\n",
       "\n",
       "# 读取数据\n",
       "train = pd.read_csv('train.csv')\n",
       "test = pd.read_csv('test.csv')\n",
       "\n",
       "# 特征选择和数据预处理\n",
       "X = train.drop(['SalePrice', 'Id'], axis=1)\n",
       "y = train['SalePrice']\n",
       "\n",
       "# 简单的数值型特征填充\n",
       "X = X.select_dtypes(include=[np.number]).fillna(X.mean())\n",
       "\n",
       "# 数据集拆分\n",
       "X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)\n",
       "\n",
       "# 模型训练\n",
       "model = RandomForestRegressor(n_estimators=100, random_state=42)\n",
       "model.fit(X_train, y_train)\n",
       "\n",
       "# 模型评估\n",
       "y_pred = model.predict(X_val)\n",
       "rmse = np.sqrt(mean_squared_error(y_val, y_pred))\n",
       "print(f'Validation RMSE: {rmse}')\n",
       "\n",
       "# 预测测试集\n",
       "test = test.select_dtypes(include=[np.number]).fillna(X.mean())\n",
       "predictions = model.predict(test.drop('Id', axis=1))\n",
       "\n",
       "# 生成提交文件\n",
       "submission = pd.DataFrame({'Id': test['Id'], 'SalePrice': predictions})\n",
       "submission.to_csv('submission.csv', index=False)\n",
       "```\n",
       "\n",
       "### 竞赛优点\n",
       "- 丰富的数据集：丰富的特征变量，为特征工程和模型构建提供了充分的数据支持；\n",
       "- 实践性强：房价预测是一个现实中的常见问题，通过该竞赛可以很好地将理论与实践结合；\n",
       "- 对模型能力的全面考察：竞赛不仅考察选手的数据预处理和特征工程能力，还深入考察模型构建、参数调优及结果分析能力。\n",
       "\n",
       "### 结论\n",
       "通过参与“House Prices”竞赛，选手能够完整地体验一个机器学习项目的全流程，有助于提升其实践能力和理论水平。\n",
       "\n",
       "【12:2†source】【12:16†source】【12:19†source】"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('请检索知识库，帮我详细介绍下House Prices这个竞赛')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5d5ba1a4-e9e2-4d51-ba51-b4af01116b1d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "▌ MateGen初始化完成，欢迎使用！"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "九天老师的公开课课件中提供了几份高分竞赛方案，每个方案均有其独特的思路和方法。这里以其中一份竞赛方案为例，详细介绍其思路。\n",
       "\n",
       "### 高分竞赛方案思路\n",
       "\n",
       "**竞赛名称**：House Prices: Advanced Regression Techniques\n",
       "\n",
       "1. **特征工程**：\n",
       "   - **缺失值处理**：处理数据集中缺失值的部分，例如使用中位数或均值填补数值型变量的缺失值，使用众数填补类别型变量的缺失值。\n",
       "   - **特征变换**：对数值型特征进行变换，包括对数变换、Box-Cox变换等，以减少特征的偏态分布。\n",
       "   - **构造新特征**：结合现有特征生成新特征，例如将每个房屋的建成年份和翻新年份结合生成“房龄”特征。\n",
       "\n",
       "2. **模型选择与训练**：\n",
       "   - **单模型训练**：\n",
       "     - 使用多种回归算法训练模型，包括线性回归、Ridge回归、Lasso回归、弹性网络回归、随机森林回归、梯度提升树（Gradient Boosting Machine, GBM）等。\n",
       "   - **集成学习**：\n",
       "     - 采用Stacking的方法，将多个基础模型的预测结果作为新特征，进一步训练元模型（Meta Learner），常用Logistic Regression或GBR作为元模型训练。\n",
       "   \n",
       "3. **模型评估与选择**：\n",
       "   - **交叉验证**：使用交叉验证（Cross-Validation）评估模型性能，选择性能最佳的模型。\n",
       "   - **模型调优**：通过网格搜索（Grid Search）或随机搜索（Random Search）等方法优化模型超参数，进一步提升模型性能。\n",
       "\n",
       "4. **预测与结果提交**：\n",
       "   - 使用训练好的最优模型对测试数据集进行预测，生成预测结果。\n",
       "   - 预测结果与实际销售价格进行比较，计算均方根误差（Root Mean Squared Error, RMSE）作为评价指标。\n",
       "\n",
       "5. **代码示例**：\n",
       "   ```python\n",
       "   import pandas as pd\n",
       "   import numpy as np\n",
       "   from sklearn.model_selection import train_test_split, cross_val_score\n",
       "   from sklearn.linear_model import Ridge, Lasso\n",
       "   from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n",
       "   from sklearn.metrics import mean_squared_error\n",
       "   from sklearn.pipeline import make_pipeline\n",
       "   from sklearn.preprocessing import RobustScaler\n",
       "\n",
       "   # 读取数据\n",
       "   train = pd.read_csv('train.csv')\n",
       "   test = pd.read_csv('test.csv')\n",
       "\n",
       "   # 特征工程示例\n",
       "   train['TotalSF'] = train['TotalBsmtSF'] + train['1stFlrSF'] + train['2ndFlrSF']\n",
       "   test['TotalSF'] = test['TotalBsmtSF'] + test['1stFlrSF'] + test['2ndFlrSF']\n",
       "   y = train['SalePrice']\n",
       "\n",
       "   # 特征选择及划分训练与验证集\n",
       "   X = train.drop(['SalePrice', 'Id'], axis=1).select_dtypes(include=[np.number]).fillna(0)\n",
       "   X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)\n",
       "\n",
       "   # 构建模型\n",
       "   models = [\n",
       "       make_pipeline(RobustScaler(), Lasso(alpha=0.0005, random_state=1)),\n",
       "       make_pipeline(RobustScaler(), Ridge(alpha=1.0)),\n",
       "       RandomForestRegressor(n_estimators=100, random_state=1),\n",
       "       GradientBoostingRegressor(n_estimators=3000, learning_rate=0.05, max_depth=4, \n",
       "                                 max_features='sqrt', min_samples_leaf=15, min_samples_split=10, \n",
       "                                 loss='huber', random_state =5)\n",
       "   ]\n",
       "\n",
       "   # 评估模型\n",
       "   for model in models:\n",
       "       model.fit(X_train, y_train)\n",
       "       score = cross_val_score(model, X_train, y_train, cv=5, scoring='neg_mean_squared_error')\n",
       "       print(f\"{model.steps[0][0]} RMSE: {np.sqrt(-score.mean())}\")\n",
       "\n",
       "   # 最优模型预测\n",
       "   final_model = models[3]\n",
       "   final_model.fit(X_train, y_train)\n",
       "   y_pred = final_model.predict(X_val)\n",
       "   rmse = np.sqrt(mean_squared_error(y_val, y_pred))\n",
       "   print(f'Validation RMSE: {rmse}')\n",
       "\n",
       "   # 测试集预测及生成提交文件\n",
       "   X_test = test.drop(['Id'], axis=1).select_dtypes(include=[np.number]).fillna(0)\n",
       "   test_preds = final_model.predict(X_test)\n",
       "   submission = pd.DataFrame({'Id': test['Id'], 'SalePrice': test_preds})\n",
       "   submission.to_csv('submission.csv', index=False)\n",
       "   ```\n",
       "\n",
       "### 小结\n",
       "通过以上竞赛思路，可以看到：\n",
       "- 详细的特征工程处理和高效的特征选择是高分方案的基础。\n",
       "- 多模型训练和集成学习提升了模型的预测能力和稳健性。\n",
       "- 通过交叉验证和网格搜索等方法调整模型超参数，确保模型的最优性能。\n",
       "\n",
       "这种系统化的方法不仅适用于House Prices竞赛，亦可适用于其他类似的回归问题。    "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mategen.chat('在你的知识库中，有什么比较精彩的高分竞赛方案么？能帮我分享下他们的思路么？')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d84ec9-b156-4bf2-85ef-5d7c98170bf8",
   "metadata": {},
   "source": [
    "#### 7.3 MateGen联网测试与设置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "065fdb77-7cf0-42e9-b95b-9511db2430e3",
   "metadata": {},
   "source": [
    "&emsp;&emsp;不过需要注意的是，MateGen的联网功能有一定使用门槛，有时会因为Cookie失效等原因无法顺利联网，因此MateGen自带了联网检测工具和设置函数，若原始Cookie失效，则可以调用相关实用函数手动输入新的Cookie来重启联网功能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08a4b408-6074-4d28-8fd8-adf630e5d503",
   "metadata": {},
   "source": [
    "- MateGen联网检测函数：check_network_environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "10118a6b-fdeb-4c7b-84b7-56d51371274e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在检测MateGen联网环境。MateGen支持连接谷歌搜索、知乎、Github和Kaggle，且可以借助阿里云oss对象存储，将生成的图片存储到oss上。\n",
      "    默认配置文件下除oss外上述功能可以正常运行，但伴随时间推移，部分cookie可能会失效。\n",
      "    因此，MateGen也支持用户自定义修改相关配置。即将开启联网功能检测...\n",
      "即将检测谷歌搜索API能否正常运行\n",
      "测试成功：API响应包含预期的数据。\n",
      "谷歌搜索API可正常运行\n",
      "即将检测能否连接知乎进行问答\n",
      "测试成功：爬虫响应正常。\n",
      "可以连接知乎进行问答\n",
      "即将检测能否连接Github进行问答\n",
      "测试成功：GitHub token 正常工作。\n",
      "可以连接Github进行问答\n",
      "即将检测能否连接Kaggle进行问答\n",
      "测试成功：Kaggle API 响应正常。\n",
      "可以连接Kaggle进行问答\n",
      "即将检测阿里云oss对象存储设置是否成功\n",
      "请先设置相关变量取值\n",
      "阿里云oss对象存储设置不成功，，请调用internet_config_assistant()函数获取帮助文档并进行设置\n"
     ]
    }
   ],
   "source": [
    "check_network_environment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3068071d-78d5-418e-93d6-38ec8e2f1009",
   "metadata": {},
   "source": [
    "&emsp;&emsp;check_network_environment会进行5项联网功能检测，首次使用MateGen时会发现oss对象存储设置不成功，是因为oss对象存储没有设置初始变量值，需要用户自行注册阿里云并获取oss存储相关信息，才能开启MateGen的oss存储功能。不过oss设置并不影响MateGen的联网功能，没有oss只会影响MateGen绘图功能无法自动保存到图床上。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8759c36-d9b1-49dc-a45c-82b421247ef0",
   "metadata": {},
   "source": [
    "- MateGen联网设置函数：internet_config_assistant"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3832ea80-c194-4289-9b9f-973e3eb12dc7",
   "metadata": {},
   "source": [
    "&emsp;&emsp;当然，我们可以调用internet_config_assistant函数获取各类不同网络环境设置指南和设置入口，例如，如果我们希望设置阿里云oss对象存储，则可以按照如下方式进行操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8ff784a7-20ac-42c6-a194-451e99d73132",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "你好，欢迎MateGen互联网配置助手，请输入你希望获取的帮助类型：1.获取帮助文档；2.进行互联网参数配置；0.退出 1\n",
      "请输入想要获得的帮助文档类型:        1.谷歌搜索API获取与配置方法；\n",
      "        2.知乎搜索cookie获取与配置方法；\n",
      "        3.Github token获取与配置方法； \n",
      "        4.Kaggle搜索cookie获取与配置方法；\n",
      "        5.阿里云oss对象存储配置方法；\n",
      "        6.返回上一级 \n",
      "        0.退出 5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "以下是《阿里云oss对象存储配置方法》详情\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "# 阿里云oss设置\r\n",
       "\r\n",
       "​\t首先，注册并登录阿里云，并点击右上方控制台：\r\n",
       "\r\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240706190716644.png\" alt=\"image-20240706190716644\" style=\"zoom:50%;\" />\r\n",
       "\r\n",
       "然后搜索oss对象存储：https://oss.console.aliyun.com/overview，点击创建Bucket：\r\n",
       "\r\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240706190820973.png\" alt=\"image-20240706190820973\" style=\"zoom:50%;\" />\r\n",
       "\r\n",
       "然后即可获得ENDPOINT='https://oss-cn-hangzhou.aliyuncs.com' （杭州地区，若是北京地区，则是https://oss-cn-beijing.aliyuncs.com） ，以及BUCKET_NAME（即创建的bucket名称）。\r\n",
       "\r\n",
       "接下来创建ACCESS_KEY_ID和ACCESS_KEY_SECRET，点击Access管理，创建账户及API-KEY即可。\r\n",
       "\r\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240706191115348.png\" alt=\"image-20240706191115348\" style=\"zoom:50%;\" />\r\n",
       "\r\n",
       "\r\n",
       "\r\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 获取帮助\n",
    "internet_config_assistant()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5972776e-e301-41e5-a16d-024f936591fd",
   "metadata": {},
   "source": [
    "然后再进行设置即可："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5d06031-7869-4964-a930-83590b111cd3",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240707222652585.png\" alt=\"image-20240707222652585\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5b50334-d983-43c5-b243-753029e6752e",
   "metadata": {},
   "source": [
    "&emsp;&emsp;至此MateGen核心功能全部介绍完毕，欢迎各位用户参与使用并提交反馈。**目前MateGen正在公测，限量开放200个内测名额，每个名额包含100万token使用额度，每位用户限领一个API-KEY，若需继续使用，也可按需充值token额度。API-KEY领取、加入技术交流群、其他任何问题，<span style=\"color:red;\">扫码添加客服小可爱(微信：littlelion_1215)，回复“MG”详询哦👇</span>**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b01373ca-7377-4421-9c72-ec74c9375f13",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240706225245612.png\" alt=\"image-20240706225245612\" style=\"zoom:50%;\" />"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mategen",
   "language": "python",
   "name": "mategen"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}