notebook/4-1-distributed-scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed scraping: multiprocessing\n",
    "\n",
    "**Speed up scraping by distributed crawling and parsing. I'm going to scrape [my website](https://morvanzhou.github.io/) but in a local server \"http://127.0.0.1:4000/\" to eliminate different downloading speed. This test is more accurate in time measuring. You can use \"https://morvanzhou.github.io/\" instead, because you cannot access \"http://127.0.0.1:4000/\".**\n",
    "\n",
    "**We gonna scrape all web pages in my website and reture the title and url for each page.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import multiprocessing as mp\n",
    "import time\n",
    "from urllib.request import urlopen, urljoin\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "\n",
    "base_url = \"http://127.0.0.1:4000/\"\n",
    "# base_url = 'https://morvanzhou.github.io/'\n",
    "\n",
    "# DON'T OVER CRAWL THE WEBSITE OR YOU MAY NEVER VISIT AGAIN\n",
    "if base_url != \"http://127.0.0.1:4000/\":\n",
    "    restricted_crawl = True\n",
    "else:\n",
    "    restricted_crawl = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Create a crawl function to open a url in parallel.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def crawl(url):\n",
    "    response = urlopen(url)\n",
    "    time.sleep(0.1)             # slightly delay for downloading\n",
    "    return response.read().decode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Create a parse function to find all results we need in parallel**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse(html):\n",
    "    soup = BeautifulSoup(html, 'lxml')\n",
    "    urls = soup.find_all('a', {\"href\": re.compile('^/.+?/$')})\n",
    "    title = soup.find('h1').get_text().strip()\n",
    "    page_urls = set([urljoin(base_url, url['href']) for url in urls])\n",
    "    url = soup.find('meta', {'property': \"og:url\"})['content']\n",
    "    return title, page_urls, url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normal way\n",
    "\n",
    "**Do not use multiprocessing, test the speed. Firstly, set what urls we have already seen and what we haven't in a python set.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nDistributed Crawling...\n\nDistributed Parsing...\n\nAnalysing...\n1 教程 http://localhost:4000\n\nDistributed Crawling...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nDistributed Parsing...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nAnalysing...\n2 其他教学系列 http://localhost:4000tutorials/others/\n3 Linux 简易教学 http://localhost:4000tutorials/others/linux-basic/\n4 近期更新 http://localhost:4000recent-posts/\n5 正则表达式 http://localhost:4000tutorials/python-basic/basic/13-10-regular-expression/\n6 从头开始做一个汽车状态分类器1: 分析数据 http://localhost:4000tutorials/machine-learning/ML-practice/build-car-classifier-from-scratch1/\n7 Threading 多线程教程系列 http://localhost:4000tutorials/python-basic/threading/\n8 强化学习 Reinforcement Learning 教程系列 http://localhost:4000tutorials/machine-learning/reinforcement-learning/\n9 从头开始做一个汽车状态分类器2: 搭建模型 http://localhost:4000tutorials/machine-learning/ML-practice/build-car-classifier-from-scratch2/\n10 我的一点点背景资料 (Mofan's Background) http://localhost:4000about/\n11 Numpy & Pandas 教程系列 http://localhost:4000tutorials/data-manipulation/np-pd/\n12 推荐学习顺序 http://localhost:4000learning-steps/\n13 为了更优秀 http://localhost:4000support/\n14 机器学习实践 http://localhost:4000tutorials/machine-learning/ML-practice/\n15 Git 版本管理 教程系列 http://localhost:4000tutorials/others/git/\n16 multiprocessing 多进程教程系列 http://localhost:4000tutorials/python-basic/multiprocessing/\n17 网页爬虫教程系列 http://localhost:4000tutorials/data-manipulation/scraping/\n18 进化算法 Evolutionary Strategies 教程系列 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/\n19 Keras 教程系列 http://localhost:4000tutorials/machine-learning/keras/\n20 Python基础 教程系列 http://localhost:4000tutorials/python-basic/\n21 Pytorch 教程系列 http://localhost:4000tutorials/machine-learning/torch/\n22 Tkinter GUI 教程系列 http://localhost:4000tutorials/python-basic/tkinter/\n23 有趣的机器学习系列 http://localhost:4000tutorials/machine-learning/ML-intro/\n24 数据处理教程系列 http://localhost:4000tutorials/data-manipulation/\n25 从头开始做一个机器手臂5 完善测试 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch5/\n26 基础教程系列 http://localhost:4000tutorials/python-basic/basic/\n27 机器学习系列 http://localhost:4000tutorials/machine-learning/\n28 Sklearn 通用机器学习 教程系列 http://localhost:4000tutorials/machine-learning/sklearn/\n29 Tensorflow 教程系列 http://localhost:4000tutorials/machine-learning/tensorflow/\n30 从头开始做一个机器手臂1 搭建结构 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch1/\n31 说吧~ http://localhost:4000discuss/\n32 Matplotlib 画图教程系列 http://localhost:4000tutorials/data-manipulation/plt/\n33 Theano 教程系列 http://localhost:4000tutorials/machine-learning/theano/\n\nDistributed Crawling...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nDistributed Parsing...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nAnalysing...\n34 Actor Critic (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-1-actor-critic/\n35 Numpy 基础运算1 http://localhost:4000tutorials/data-manipulation/np-pd/2-3-np-math1/\n36 读写文件 3 http://localhost:4000tutorials/python-basic/basic/08-3-read-file3/\n37 小例子 http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-1-general-rl/\n38 Why Keras? http://localhost:4000tutorials/machine-learning/keras/1-1-why/\n39 什么是 强化学习 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-1-A-RL/\n40 Actor Critic http://localhost:4000tutorials/machine-learning/ML-intro/4-08-AC/\n41 临时修改 (stash) http://localhost:4000tutorials/others/git/4-4-stash/\n42 什么是 DQN http://localhost:4000tutorials/machine-learning/torch/4-05-A-DQN/\n43 messagebox 弹窗 http://localhost:4000tutorials/python-basic/tkinter/2-10-messagebox/\n44 Canvas 画布 http://localhost:4000tutorials/python-basic/tkinter/2-07-canvas/\n45 交叉验证 2 Cross-validation http://localhost:4000tutorials/machine-learning/sklearn/3-3-cross-validation2/\n46 Image 图片 http://localhost:4000tutorials/data-manipulation/plt/3-4-image/\n47 线程锁 Lock http://localhost:4000tutorials/python-basic/threading/6-lock/\n48 什么是 Asynchronous Advantage Actor-Critic (A3C) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-3-A1-A3C/\n49 神经进化 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-01-neuro-evolution/\n50 Prioritized Experience Replay (DQN) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-6-prioritized-replay/\n51 RNN 循环神经网络 (回归) http://localhost:4000tutorials/machine-learning/torch/4-03-RNN-regression/\n52 Classification 分类学习 http://localhost:4000tutorials/machine-learning/tensorflow/5-01-classifier/\n53 什么是 Policy Gradients http://localhost:4000tutorials/machine-learning/reinforcement-learning/5-1-A-PG/\n54 Sarsa http://localhost:4000tutorials/machine-learning/ML-intro/4-04-sarsa/\n55 LSTM RNN 循环神经网络 (LSTM) http://localhost:4000tutorials/machine-learning/ML-intro/2-4-LSTM/\n56 神经网络在干嘛 http://localhost:4000tutorials/machine-learning/tensorflow/1-3-what-does-NN-do/\n57 为什么用 Matplotlib http://localhost:4000tutorials/data-manipulation/plt/1-1-why/\n58 while 循环 http://localhost:4000tutorials/python-basic/basic/03-1-while/\n59 Legend 图例 http://localhost:4000tutorials/data-manipulation/plt/2-5-lagend/\n60 基本用法 http://localhost:4000tutorials/data-manipulation/plt/2-1-basic-usage/\n61 什么是 tkinter 窗口 http://localhost:4000tutorials/python-basic/basic/13-07-tkinter/\n62 模块安装 http://localhost:4000tutorials/python-basic/basic/07-1-module-install/\n63 课程要求 http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-2-requirment/\n64 GAN (Generative Adversarial Nets 生成对抗网络) http://localhost:4000tutorials/machine-learning/torch/4-06-GAN/\n65 Pandas 基本介绍 http://localhost:4000tutorials/data-manipulation/np-pd/3-1-pd-intro/\n66 if elif else 判断 http://localhost:4000tutorials/python-basic/basic/04-3-if-elif-else/\n67 自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/ML-intro/2-5-autoencoder/\n68 什么是自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/keras/2-6-A-autoencoder/\n69 自编码 Autoencoder (非监督学习) http://localhost:4000tutorials/machine-learning/tensorflow/5-11-autoencoder/\n70 Dueling DQN (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-7-dueling-DQN/\n71 for 循环 http://localhost:4000tutorials/python-basic/basic/03-2-for/\n72 Linux 基本指令 mkdir, rmdir 和 rm http://localhost:4000tutorials/others/linux-basic/2-03-basic-command/\n73 Numpy 基础运算2 http://localhost:4000tutorials/data-manipulation/np-pd/2-4-np-math2/\n74 Github 在线代码管理 http://localhost:4000tutorials/others/git/5-1-github/\n75 CNN 卷积神经网络 3 http://localhost:4000tutorials/machine-learning/tensorflow/5-05-CNN3/\n76 Asynchronous Advantage Actor-Critic (A3C) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-3-A3C/\n77 Regressor 回归 http://localhost:4000tutorials/machine-learning/keras/2-1-regressor/\n78 Q-learning 思维决策 http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-3-tabular-q2/\n79 Subplot 分格显示 http://localhost:4000tutorials/data-manipulation/plt/4-2-subplot2/\n80 Pandas plot 出图 http://localhost:4000tutorials/data-manipulation/np-pd/3-8-pd-plot/\n81 批标准化 (Batch Normalization) http://localhost:4000tutorials/machine-learning/ML-intro/3-08-batch-normalization/\n82 Policy Gradients http://localhost:4000tutorials/machine-learning/ML-intro/4-07-PG/\n83 什么是多线程 Threading http://localhost:4000tutorials/python-basic/threading/1-why/\n84 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/ML-intro/2-8-gradient-descent/\n85 为什么 Torch 是动态的 http://localhost:4000tutorials/machine-learning/torch/5-01-dynamic/\n86 保存提取 http://localhost:4000tutorials/machine-learning/torch/3-04-save-reload/\n87 添加线程 Thread http://localhost:4000tutorials/python-basic/threading/2-add-thread/\n88 DQN 神经网络 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-2-DQN2/\n89 什么是自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/torch/4-04-A-autoencoder/\n90 Autoencoder 自编码 http://localhost:4000tutorials/machine-learning/keras/2-6-autoencoder/\n91 元组 列表 http://localhost:4000tutorials/python-basic/basic/11-1-array-list/\n92 Sarsa-lambda http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-3-tabular-sarsa-lambda/\n93 设置坐标轴2 http://localhost:4000tutorials/data-manipulation/plt/2-4-axis2/\n94 函数参数 http://localhost:4000tutorials/python-basic/basic/05-2-def-parameters/\n95 神经网络在做什么 http://localhost:4000tutorials/machine-learning/theano/1-3-NN-job/\n96 GIL 不一定有效率 http://localhost:4000tutorials/python-basic/threading/5-GIL/\n97 Pandas 设置值 http://localhost:4000tutorials/data-manipulation/np-pd/3-3-pd-assign/\n98 Pandas 选择数据 http://localhost:4000tutorials/data-manipulation/np-pd/3-2-pd-indexing/\n99 CNN 卷积神经网络 http://localhost:4000tutorials/machine-learning/torch/4-01-CNN/\n100 Placeholder 传入值 http://localhost:4000tutorials/machine-learning/tensorflow/2-5-placeholde/\n101 Tensorboard 可视化好帮手 1 http://localhost:4000tutorials/machine-learning/tensorflow/4-1-tensorboard1/\n102 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/theano/1-1-A-ANN-and-NN/\n103 回到从前 (checkout 针对单个文件) http://localhost:4000tutorials/others/git/3-2-checkout/\n104 为什么用强化学习 Why? http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-1-why/\n105 Linux 基本指令 touch, cp 和 mv http://localhost:4000tutorials/others/linux-basic/2-02-basic-command/\n106 OpenAI gym 环境库 http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-4-gym/\n107 NEAT 强化学习 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-03-neat-reinforcement-learning/\n108 什么是卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/tensorflow/5-03-A-CNN/\n109 Tensorflow 2017 更新 http://localhost:4000tutorials/machine-learning/tensorflow/5-14-tf2017/\n110 continue & break http://localhost:4000tutorials/python-basic/basic/13-01-continue-break/\n111 multiprocessing 什么是多进程 http://localhost:4000tutorials/python-basic/basic/13-06-multiprocessing/\n112 Saver 保存读取 http://localhost:4000tutorials/machine-learning/tensorflow/5-06-save/\n113 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/1-1-A-ANN-and-NN/\n114 什么是 Sarsa http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-1-A-sarsa/\n115 神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/ML-intro/2-1-NN/\n116 通用学习模式 http://localhost:4000tutorials/machine-learning/sklearn/2-2-general-pattern/\n117 第一个版本库 Repository http://localhost:4000tutorials/others/git/2-1-repository/\n118 Dropout 缓解过拟合 http://localhost:4000tutorials/machine-learning/torch/5-03-dropout/\n119 Sarsa 算法更新 http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-1-tabular-sarsa1/\n120 set 找不同 http://localhost:4000tutorials/python-basic/basic/13-09-set/\n121 多维列表 http://localhost:4000tutorials/python-basic/basic/11-3-multi-dim-list/\n122 检验神经网络 (Evaluation) http://localhost:4000tutorials/machine-learning/sklearn/3-2-A-Evaluate-NN/\n123 3D 数据 http://localhost:4000tutorials/data-manipulation/plt/3-5-3d/\n124 Regression 回归例子 http://localhost:4000tutorials/machine-learning/theano/3-2-regression/\n125 激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/ML-intro/3-04-activation-function/\n126 Scatter 散点图 http://localhost:4000tutorials/data-manipulation/plt/3-1-scatter/\n127 什么是神经网络进化 (Neuro-Evolution) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-00-neuro-evolution/\n128 什么是循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/tensorflow/5-07-A-RNN/\n129 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/tensorflow/1-1-B-NN/\n130 DQN 算法更新 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-1-DQN1/\n131 怎么样从 MacOS 或 Linux 通过 SSH 远程 Linux http://localhost:4000tutorials/others/linux-basic/4-01-ssh-from-linux-or-mac/\n132 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/keras/1-1-B-NN/\n133 区分类型 (分类) http://localhost:4000tutorials/machine-learning/torch/3-02-classification/\n134 Optimizer 优化器 http://localhost:4000tutorials/machine-learning/torch/3-06-optimizer/\n135 AlphaGo Zero 为什么更厉害? http://localhost:4000tutorials/machine-learning/ML-intro/4-11-AlphaGo-zero/\n136 Deep Deterministic Policy Gradient (DDPG) http://localhost:4000tutorials/machine-learning/ML-intro/4-09-DDPG/\n137 if 判断 http://localhost:4000tutorials/python-basic/basic/04-1-if/\n138 class 类 init 功能 http://localhost:4000tutorials/python-basic/basic/09-2-class-init/\n139 Numpy copy & deep copy http://localhost:4000tutorials/data-manipulation/np-pd/2-8-np-copy/\n140 Tensorboard 可视化好帮手 2 http://localhost:4000tutorials/machine-learning/tensorflow/4-2-tensorboard2/\n141 Frame 框架 http://localhost:4000tutorials/python-basic/tkinter/2-09-frame/\n142 Microbial Genetic Algorithm http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-04-microbial-genetic-algorithm/\n143 NEAT 监督学习 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-02-neat-supervised-learning/\n144 安装 http://localhost:4000tutorials/python-basic/basic/01-1-install/\n145 怎么样从 Windows 通过 SSH 远程 Linux http://localhost:4000tutorials/others/linux-basic/4-02-ssh-from-windows/\n146 什么是 Actor Critic http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-1-A-AC/\n147 遗传算法 (Genetic Algorithm) http://localhost:4000tutorials/machine-learning/ML-intro/5-01-genetic-algorithm/\n148 CNN 卷积神经网络 http://localhost:4000tutorials/machine-learning/keras/2-3-CNN/\n149 选择好特征 (Good Features) http://localhost:4000tutorials/machine-learning/ML-intro/3-03-choose-feature/\n150 什么是 Q Leaning http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-2-A-q-learning/\n151 Evolution Strategy 强化学习 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-04-evolution-strategy-reinforcement-learning/\n152 兼容 backend http://localhost:4000tutorials/machine-learning/keras/1-3-backend/\n153 什么是过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/theano/3-5-A-overfitting/\n154 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/theano/1-1-C-gradient-descent/\n155 Numpy 和 Pandas 安装 http://localhost:4000tutorials/data-manipulation/np-pd/1-2-install/\n156 例子 旅行商人问题 (Travel Sales Problem) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-03-genetic-algorithm-travel-sales-problem/\n157 Distributed Proximal Policy Optimization (DPPO) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-4-DPPO/\n158 Pytorch 安装 http://localhost:4000tutorials/machine-learning/torch/1-2-install/\n159 基础数学运算 http://localhost:4000tutorials/python-basic/basic/02-2-basic-math/\n160 激励函数 (Activation) http://localhost:4000tutorials/machine-learning/torch/2-03-activation/\n161 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/tensorflow/1-1-D-feature-representation/\n162 例子3 结果可视化 http://localhost:4000tutorials/machine-learning/tensorflow/3-3-visualize-result/\n163 Activation function 激励函数 http://localhost:4000tutorials/machine-learning/theano/2-4-activation/\n164 可视化结果 回归例子 http://localhost:4000tutorials/machine-learning/theano/3-3-visualize/\n165 什么是循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/torch/4-02-A-RNN/\n166 什么是 Deep Deterministic Policy Gradient (DDPG) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-2-A-DDPG/\n167 例子 登录窗口1 http://localhost:4000tutorials/python-basic/tkinter/3-01-example1/\n168 怎么样用 TeamViewer 和 VNC 从远程控制电脑 http://localhost:4000tutorials/others/linux-basic/4-04-teamviewer-vnc/\n169 什么是循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/keras/2-4-A-RNN/\n170 Pandas 导入导出 http://localhost:4000tutorials/data-manipulation/np-pd/3-5-pd-to/\n171 选择学习方法 http://localhost:4000tutorials/machine-learning/sklearn/2-1-select-method/\n172 class 类 http://localhost:4000tutorials/python-basic/basic/09-1-class/\n173 什么是激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/theano/2-4-A-activation-function/\n174 特征标准化 (Feature Normalization) http://localhost:4000tutorials/machine-learning/ML-intro/3-02-normalization/\n175 Policy Gradients 思维决策 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/5-2-policy-gradient-softmax2/\n176 RNN LSTM (回归例子可视化) http://localhost:4000tutorials/machine-learning/tensorflow/5-10-RNN4/\n177 关系拟合 (回归) http://localhost:4000tutorials/machine-learning/torch/3-01-regression/\n178 正规化 Normalization http://localhost:4000tutorials/machine-learning/sklearn/3-1-normalization/\n179 GPU 加速运算 http://localhost:4000tutorials/machine-learning/torch/5-02-GPU/\n180 DQN 强化学习 http://localhost:4000tutorials/machine-learning/torch/4-05-DQN/\n181 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/ML-intro/2-7-feature-representation/\n182 加速神经网络训练 (Speed Up Training) http://localhost:4000tutorials/machine-learning/ML-intro/3-06-speed-up-learning/\n183 交叉验证 1 Cross-validation http://localhost:4000tutorials/machine-learning/sklearn/3-2-cross-validation1/\n184 读写文件 1 http://localhost:4000tutorials/python-basic/basic/08-1-read-file/\n185 回到从前 (reset) http://localhost:4000tutorials/others/git/3-1-reset/\n186 什么是 Tkinter http://localhost:4000tutorials/python-basic/tkinter/1-1-why/\n187 Theano 安装 http://localhost:4000tutorials/machine-learning/theano/1-2-install/\n188 Save 保存 提取 http://localhost:4000tutorials/machine-learning/theano/3-6-save/\n189 sklearn 常用属性与功能 http://localhost:4000tutorials/machine-learning/sklearn/2-4-model-attributes/\n190 变量 (Variable) http://localhost:4000tutorials/machine-learning/torch/2-02-variable/\n191 threading 什么是多线程 http://localhost:4000tutorials/python-basic/basic/13-05-threading/\n192 RNN LSTM 循环神经网络 (分类例子) http://localhost:4000tutorials/machine-learning/tensorflow/5-08-RNN2/\n193 记录修改 (log & diff) http://localhost:4000tutorials/others/git/2-2-modified/\n194 dictionary 字典 http://localhost:4000tutorials/python-basic/basic/11-4-dictionary/\n195 Session 会话控制 http://localhost:4000tutorials/machine-learning/tensorflow/2-3-session/\n196 神经网络进化 (Neuro-Evolution) http://localhost:4000tutorials/machine-learning/ML-intro/5-03-neuro-evolution/\n197 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/tensorflow/1-1-C-gradient-descent/\n198 例子3 添加层 def add_layer() http://localhost:4000tutorials/machine-learning/tensorflow/3-1-add-layer/\n199 Scale 尺度 http://localhost:4000tutorials/python-basic/tkinter/2-05-scale/\n200 例子 配对句子 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-02-genetic-algorithm-match-phrase/\n201 rebase 分支冲突 http://localhost:4000tutorials/others/git/4-3-rebase/\n202 Batch Normalization 批标准化 http://localhost:4000tutorials/machine-learning/torch/5-04-batch-normalization/\n203 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/theano/1-1-D-feature-representation/\n204 自己的云计算, 把 Linux 当成你的云计算平台 http://localhost:4000tutorials/others/linux-basic/5-01-remote-learning/\n205 例子 登录窗口2 http://localhost:4000tutorials/python-basic/tkinter/3-02-example2/\n206 机器学习 (Machine Learning) http://localhost:4000tutorials/machine-learning/sklearn/1-1-A-ML/\n207 自己的模块 http://localhost:4000tutorials/python-basic/basic/12-2-personal-module/\n208 Matplotlib 安装 http://localhost:4000tutorials/data-manipulation/plt/1-2-install/\n209 进化策略 (Evolution Strategy) http://localhost:4000tutorials/machine-learning/ML-intro/5-02-evolution-strategy/\n210 list 列表 http://localhost:4000tutorials/python-basic/basic/11-2-list/\n211 Double DQN (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-5-double_DQN/\n212 Numpy 的创建 array http://localhost:4000tutorials/data-manipulation/np-pd/2-2-np-array/\n213 遗传算法 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-01-genetic-algorithm/\n214 Torch 或 Numpy http://localhost:4000tutorials/machine-learning/torch/2-01-torch-numpy/\n215 快速搭建法 http://localhost:4000tutorials/machine-learning/torch/3-03-fast-nn/\n216 例子 登录窗口3 http://localhost:4000tutorials/python-basic/tkinter/3-03-example3/\n217 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/ML-intro/2-0-ANN-and-NN/\n218 zip lambda map http://localhost:4000tutorials/python-basic/basic/13-03-zip-lambda-map/\n219 Batch Normalization 批标准化 http://localhost:4000tutorials/machine-learning/tensorflow/5-13-BN/\n220 pack grid place 放置位置 http://localhost:4000tutorials/python-basic/tkinter/2-11-pack-grid-place/\n221 scope 命名方法 http://localhost:4000tutorials/machine-learning/tensorflow/5-12-scope/\n222 Subplot 多合一显示 http://localhost:4000tutorials/data-manipulation/plt/4-1-subpot1/\n223 L1 / L2 正规化 (Regularization) http://localhost:4000tutorials/machine-learning/ML-intro/3-09-l1l2regularization/\n224 强化学习方法汇总 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/ML-intro/4-02-RL-methods/\n225 次坐标轴 http://localhost:4000tutorials/data-manipulation/plt/4-4-sencondary-axis/\n226 什么是 LSTM 循环神经网络 http://localhost:4000tutorials/machine-learning/keras/2-4-B-LSTM/\n227 Asynchronous Advantage Actor-Critic (A3C) http://localhost:4000tutorials/machine-learning/ML-intro/4-10-A3C/\n228 强化学习 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/ML-intro/4-01-RL/\n229 Numpy array 分割 http://localhost:4000tutorials/data-manipulation/np-pd/2-7-np-split/\n230 Numpy array 合并 http://localhost:4000tutorials/data-manipulation/np-pd/2-6-np-concat/\n231 if else 判断 http://localhost:4000tutorials/python-basic/basic/04-2-if-else/\n232 进程锁 Lock http://localhost:4000tutorials/python-basic/multiprocessing/7-lock/\n233 import 模块 http://localhost:4000tutorials/python-basic/basic/12-1-import/\n234 RNN Regressor 循环神经网络 http://localhost:4000tutorials/machine-learning/keras/2-5-RNN-LSTM-Regressor/\n235 sklearn 强大数据库 http://localhost:4000tutorials/machine-learning/sklearn/2-3-database/\n236 Regularization 正规化 http://localhost:4000tutorials/machine-learning/theano/3-5-regularization/\n237 效率对比 threading & multiprocessing http://localhost:4000tutorials/python-basic/multiprocessing/4-comparison/\n238 Why Numpy & Pandas? http://localhost:4000tutorials/data-manipulation/np-pd/1-1-why/\n239 共享内存 shared memory http://localhost:4000tutorials/python-basic/multiprocessing/6-shared-memory/\n240 什么是激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/torch/2-03-A-activation-function/\n241 Menubar 菜单 http://localhost:4000tutorials/python-basic/tkinter/2-08-menubar/\n242 Annotation 标注 http://localhost:4000tutorials/data-manipulation/plt/2-6-annotation/\n243 RNN 循环神经网络 (分类) http://localhost:4000tutorials/machine-learning/torch/4-02-RNN-classification/\n244 什么是进化策略 (Evolution Strategy) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-00-evolution-strategy/\n245 例子3 建造神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/3-2-create-NN/\n246 (1+1)-ES http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-02-evolution-strategy-1+1-ES/\n247 保存模型 http://localhost:4000tutorials/machine-learning/sklearn/3-5-save/\n248 进化算法 简介 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/1-01-intro/\n249 Sarsa 思维决策 http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-2-tabular-sarsa2/\n250 Dropout 解决 overfitting http://localhost:4000tutorials/machine-learning/tensorflow/5-02-dropout/\n251 什么是激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/tensorflow/2-6-A-activation-function/\n252 进程池 Pool http://localhost:4000tutorials/python-basic/multiprocessing/5-pool/\n253 Asyncio http://localhost:4000tutorials/data-manipulation/scraping/asyncio/\n254 添加进程 Process http://localhost:4000tutorials/python-basic/multiprocessing/2-add/\n255 存储进程输出 Queue http://localhost:4000tutorials/python-basic/multiprocessing/3-queue/\n256 过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/ML-intro/3-05-overfitting/\n257 图中图 http://localhost:4000tutorials/data-manipulation/plt/4-3-plot-in-plot/\n258 Save & reload 保存提取 http://localhost:4000tutorials/machine-learning/keras/3-1-save/\n259 Checkbutton 勾选项 http://localhost:4000tutorials/python-basic/tkinter/2-06-checkbutton/\n260 分支 (branch) http://localhost:4000tutorials/others/git/4-1-branch/\n261 Q-learning 算法更新 http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-2-tabular-q1/\n262 变量 variable http://localhost:4000tutorials/python-basic/basic/02-3-variable/\n263 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/torch/1-1-C-gradient-descent/\n264 生成对抗网络 (GAN) http://localhost:4000tutorials/machine-learning/ML-intro/2-6-GAN/\n265 什么是 Sarsa(lambda) http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-3-A-sarsa-lambda/\n266 Pandas 合并 merge http://localhost:4000tutorials/data-manipulation/np-pd/3-7-pd-merge/\n267 遗传算法 (Genetic Algorithm) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-00-genetic-algorithm/\n268 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/keras/1-1-D-feature-representation/\n269 什么是批标准化 (Batch Normalization) http://localhost:4000tutorials/machine-learning/torch/5-04-A-batch-normalization/\n270 自己的云计算, 多电脑共享你云端文件 http://localhost:4000tutorials/others/linux-basic/5-02-share-folder/\n271 从头开始做一个机器手臂3 写动态环境 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch3/\n272 定义 Layer 类 http://localhost:4000tutorials/machine-learning/theano/3-1-layer/\n273 什么是 LSTM 循环神经网络 http://localhost:4000tutorials/machine-learning/torch/4-02-B-LSTM/\n274 pickle 保存数据 http://localhost:4000tutorials/python-basic/basic/13-08-pickle/\n275 Classifier 分类 http://localhost:4000tutorials/machine-learning/keras/2-2-classifier/\n276 CNN 卷积神经网络 1 http://localhost:4000tutorials/machine-learning/tensorflow/5-03-CNN1/\n277 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/torch/1-1-D-feature-representation/\n278 什么是生成对抗网络 (GAN) http://localhost:4000tutorials/machine-learning/torch/4-06-A-GAN/\n279 强化学习方法汇总 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-1-B-RL-methods/\n280 Keras 安装 http://localhost:4000tutorials/machine-learning/keras/1-2-install/\n281 Entry & Text 输入, 文本框 http://localhost:4000tutorials/python-basic/tkinter/2-02-entry-text/\n282 RNN 循环神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/5-07-RNN1/\n283 Policy Gradients 算法更新 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/5-1-policy-gradient-softmax1/\n284 Function 用法 http://localhost:4000tutorials/machine-learning/theano/2-2-function/\n285 优化器 optimizer http://localhost:4000tutorials/machine-learning/tensorflow/3-4-optimizer/\n286 Pandas 合并 concat http://localhost:4000tutorials/data-manipulation/np-pd/3-6-pd-concat/\n287 Tensorflow 安装 http://localhost:4000tutorials/machine-learning/tensorflow/1-2-install/\n288 从头开始做一个机器手臂4 加入强化学习算法 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch4/\n289 Deep Deterministic Policy Gradient (DDPG) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-2-DDPG/\n290 什么是卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/keras/2-3-A-CNN/\n291 什么是批标准化 (Batch Normalization) http://localhost:4000tutorials/machine-learning/tensorflow/5-13-A-batch-normalization/\n292 什么是过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/tensorflow/5-02-A-overfitting/\n293 函数默认参数 http://localhost:4000tutorials/python-basic/basic/05-3-def-default-parameters/\n294 Linux 文件权限 http://localhost:4000tutorials/others/linux-basic/3-01-file-permissions/\n295 Animation 动画 http://localhost:4000tutorials/data-manipulation/plt/5-1-animation/\n296 全局 & 局部 变量 http://localhost:4000tutorials/python-basic/basic/06-1-global-local/\n297 从头开始做一个机器手臂2 写静态环境 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch2/\n298 Natural Evolution Strategy http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-03-evolution-strategy-natural-evolution-strategy/\n299 Pandas 处理丢失数据 http://localhost:4000tutorials/data-manipulation/np-pd/3-4-pd-nan/\n300 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/theano/1-1-B-NN/\n301 Numpy 索引 http://localhost:4000tutorials/data-manipulation/np-pd/2-5-np-indexing/\n302 Radiobutton 选择按钮 http://localhost:4000tutorials/python-basic/tkinter/2-04-radiobutton/\n303 激励函数 Activation Function http://localhost:4000tutorials/machine-learning/tensorflow/2-6-activation/\n304 Why Theano? http://localhost:4000tutorials/machine-learning/theano/1-1-why/\n305 Classification 分类学习 http://localhost:4000tutorials/machine-learning/theano/3-4-classification/\n306 了解网页结构 http://localhost:4000tutorials/data-manipulation/scraping/1-01-understand-website/\n307 Variable 变量 http://localhost:4000tutorials/machine-learning/tensorflow/2-4-variable/\n308 加速神经网络训练 (Speed Up Training) http://localhost:4000tutorials/machine-learning/tensorflow/3-4-A-speed-up-learning/\n309 Linux 基本指令 nano 和 cat http://localhost:4000tutorials/others/linux-basic/2-04-basic-command/\n310 处理结构 http://localhost:4000tutorials/machine-learning/tensorflow/2-1-structure/\n311 循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/ML-intro/2-3-RNN/\n312 Bar 柱状图 http://localhost:4000tutorials/data-manipulation/plt/3-2-Bar/\n313 Shared 变量 http://localhost:4000tutorials/machine-learning/theano/2-3-shared-variable/\n314 总结和更多 http://localhost:4000tutorials/machine-learning/theano/4-1-summary/\n315 print 功能 http://localhost:4000tutorials/python-basic/basic/02-1-print/\n316 merge 分支冲突 http://localhost:4000tutorials/others/git/4-2-merge-conflict/\n317 join 功能 http://localhost:4000tutorials/python-basic/threading/3-join/\n318 Why Git? http://localhost:4000tutorials/others/git/1-1-why/\n319 RNN Classifier 循环神经网络 http://localhost:4000tutorials/machine-learning/keras/2-4-RNN-classifier/\n320 def 函数 http://localhost:4000tutorials/python-basic/basic/05-1-def/\n321 安装 Ubuntu 17.10 http://localhost:4000tutorials/others/linux-basic/1-2-install/\n322 Linux 基本指令 ls 和 cd http://localhost:4000tutorials/others/linux-basic/2-01-basic-command/\n323 Sklearn 安装 http://localhost:4000tutorials/machine-learning/sklearn/1-2-install/\n324 AutoEncoder (自编码/非监督学习) http://localhost:4000tutorials/machine-learning/torch/4-04-autoencoder/\n325 DQN http://localhost:4000tutorials/machine-learning/ML-intro/4-06-DQN/\n326 为什么用 Numpy 还是慢, 你用对了吗? http://localhost:4000tutorials/data-manipulation/np-pd/4-1-speed-up-numpy/\n327 什么是过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/torch/5-03-A-overfitting/\n328 机器学习 (Machine Learning) http://localhost:4000tutorials/machine-learning/ML-intro/1-1-machine-learning/\n329 Sarsa(lambda) http://localhost:4000tutorials/machine-learning/ML-intro/4-05-sarsa-lambda/\n330 卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/ML-intro/2-2-CNN/\n331 交叉验证 3 Cross-validation http://localhost:4000tutorials/machine-learning/sklearn/3-4-cross-validation3/\n332 例子2 http://localhost:4000tutorials/machine-learning/tensorflow/2-2-example2/\n333 什么是 DQN http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-1-A-DQN/\n334 为什么选 Tensorflow? http://localhost:4000tutorials/machine-learning/tensorflow/1-1-why/\n335 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/keras/1-1-A-ANN-and-NN/\n336 设置坐标轴1 http://localhost:4000tutorials/data-manipulation/plt/2-3-axis1/\n337 Why Sklearn? http://localhost:4000tutorials/machine-learning/sklearn/1-1-why/\n338 Q Leaning http://localhost:4000tutorials/machine-learning/ML-intro/4-03-q-learning/\n339 进化策略 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-01-evolution-strategy/\n340 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/keras/1-1-C-gradient-descent/\n341 什么是 Multiprocessing http://localhost:4000tutorials/python-basic/multiprocessing/1-why/\n342 Listbox 列表部件 http://localhost:4000tutorials/python-basic/tkinter/2-03-listbox/\n343 Numpy 属性 http://localhost:4000tutorials/data-manipulation/np-pd/2-1-np-attributes/\n344 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/torch/1-1-A-ANN-and-NN/\n345 RNN LSTM (回归例子) http://localhost:4000tutorials/machine-learning/tensorflow/5-09-RNN3/\n346 CNN 卷积神经网络 2 http://localhost:4000tutorials/machine-learning/tensorflow/5-04-CNN2/\n347 figure 图像 http://localhost:4000tutorials/data-manipulation/plt/2-2-figure/\n348 用 Tensorflow 可视化梯度下降 http://localhost:4000tutorials/machine-learning/tensorflow/5-15-tf-gradient-descent/\n349 储存进程结果 Queue http://localhost:4000tutorials/python-basic/threading/4-queue/\n350 Git 安装 http://localhost:4000tutorials/others/git/1-2-install/\n351 tick 能见度 http://localhost:4000tutorials/data-manipulation/plt/2-7-tick-visibility/\n352 给你的 Ubuntu 安装软件 http://localhost:4000tutorials/others/linux-basic/1-4-install-software/\n353 基本用法 http://localhost:4000tutorials/machine-learning/theano/2-1-basic-usage/\n354 加速神经网络训练 (Speed Up Training) http://localhost:4000tutorials/machine-learning/torch/3-06-A-speed-up-learning/\n355 Contours 等高线图 http://localhost:4000tutorials/data-manipulation/plt/3-3-contours/\n356 处理不均衡数据 (Imbalanced data) http://localhost:4000tutorials/machine-learning/ML-intro/3-07-imbalanced-data/\n357 批训练 http://localhost:4000tutorials/machine-learning/torch/3-05-train-on-batch/\n358 DQN 思维决策 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-3-DQN3/\n359 快速了解 Ubuntu 17.10 基本界面 http://localhost:4000tutorials/others/linux-basic/1-3-system-overview/\n360 Why Pytorch? http://localhost:4000tutorials/machine-learning/torch/1-1-why/\n361 什么是卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/torch/4-01-A-CNN/\n362 Label & Button 标签和按钮 http://localhost:4000tutorials/python-basic/tkinter/2-01-label-button/\n363 读写文件 2 http://localhost:4000tutorials/python-basic/basic/08-2-read-file2/\n364 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/torch/1-1-B-NN/\n365 什么是自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/tensorflow/5-11-A-autoencoder/\n366 Why Linux? http://localhost:4000tutorials/others/linux-basic/1-1-why/\n367 怎么样从手机 (Android安卓/IOS苹果) 通过 SSH 远程 Linux http://localhost:4000tutorials/others/linux-basic/4-03-ssh-from-phone/\n368 copy & deepcopy 浅复制 & 深复制 http://localhost:4000tutorials/python-basic/basic/13-04-copy/\n369 检验神经网络 (Evaluation) http://localhost:4000tutorials/machine-learning/ML-intro/3-01-Evaluate-NN/\n370 input 输入 http://localhost:4000tutorials/python-basic/basic/10-1-input/\n371 什么是 LSTM 循环神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/5-07-B-LSTM/\n372 try 错误处理 http://localhost:4000tutorials/python-basic/basic/13-02-try/\nTotal time: 52.3 s\n"
     ]
    }
   ],
   "source": [
    "unseen = set([base_url,])\n",
    "seen = set()\n",
    "\n",
    "count, t1 = 1, time.time()\n",
    "\n",
    "while len(unseen) != 0:                 # still get some url to visit\n",
    "    if restricted_crawl and len(seen) > 20:\n",
    "            break\n",
    "        \n",
    "    print('\\nDistributed Crawling...')\n",
    "    htmls = [crawl(url) for url in unseen]\n",
    "\n",
    "    print('\\nDistributed Parsing...')\n",
    "    results = [parse(html) for html in htmls]\n",
    "\n",
    "    print('\\nAnalysing...')\n",
    "    seen.update(unseen)         # seen the crawled\n",
    "    unseen.clear()              # nothing unseen\n",
    "\n",
    "    for title, page_urls, url in results:\n",
    "        print(count, title, url)\n",
    "        count += 1\n",
    "        unseen.update(page_urls - seen)     # get new url to crawl\n",
    "print('Total time: %.1f s' % (time.time()-t1, ))    # 53 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## multiprocessing\n",
    "**Create a process pool and scrape parallelly.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nDistributed Crawling...\n\nDistributed Parsing...\n\nAnalysing...\n1 教程 http://localhost:4000\n\nDistributed Crawling...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nDistributed Parsing...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nAnalysing...\n2 其他教学系列 http://localhost:4000tutorials/others/\n3 Linux 简易教学 http://localhost:4000tutorials/others/linux-basic/\n4 近期更新 http://localhost:4000recent-posts/\n5 正则表达式 http://localhost:4000tutorials/python-basic/basic/13-10-regular-expression/\n6 从头开始做一个汽车状态分类器1: 分析数据 http://localhost:4000tutorials/machine-learning/ML-practice/build-car-classifier-from-scratch1/\n7 Threading 多线程教程系列 http://localhost:4000tutorials/python-basic/threading/\n8 强化学习 Reinforcement Learning 教程系列 http://localhost:4000tutorials/machine-learning/reinforcement-learning/\n9 从头开始做一个汽车状态分类器2: 搭建模型 http://localhost:4000tutorials/machine-learning/ML-practice/build-car-classifier-from-scratch2/\n10 我的一点点背景资料 (Mofan's Background) http://localhost:4000about/\n11 Numpy & Pandas 教程系列 http://localhost:4000tutorials/data-manipulation/np-pd/\n12 推荐学习顺序 http://localhost:4000learning-steps/\n13 为了更优秀 http://localhost:4000support/\n14 机器学习实践 http://localhost:4000tutorials/machine-learning/ML-practice/\n15 Git 版本管理 教程系列 http://localhost:4000tutorials/others/git/\n16 multiprocessing 多进程教程系列 http://localhost:4000tutorials/python-basic/multiprocessing/\n17 网页爬虫教程系列 http://localhost:4000tutorials/data-manipulation/scraping/\n18 进化算法 Evolutionary Strategies 教程系列 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/\n19 Keras 教程系列 http://localhost:4000tutorials/machine-learning/keras/\n20 Python基础 教程系列 http://localhost:4000tutorials/python-basic/\n21 Pytorch 教程系列 http://localhost:4000tutorials/machine-learning/torch/\n22 Tkinter GUI 教程系列 http://localhost:4000tutorials/python-basic/tkinter/\n23 有趣的机器学习系列 http://localhost:4000tutorials/machine-learning/ML-intro/\n24 数据处理教程系列 http://localhost:4000tutorials/data-manipulation/\n25 从头开始做一个机器手臂5 完善测试 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch5/\n26 基础教程系列 http://localhost:4000tutorials/python-basic/basic/\n27 机器学习系列 http://localhost:4000tutorials/machine-learning/\n28 Sklearn 通用机器学习 教程系列 http://localhost:4000tutorials/machine-learning/sklearn/\n29 Tensorflow 教程系列 http://localhost:4000tutorials/machine-learning/tensorflow/\n30 从头开始做一个机器手臂1 搭建结构 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch1/\n31 说吧~ http://localhost:4000discuss/\n32 Matplotlib 画图教程系列 http://localhost:4000tutorials/data-manipulation/plt/\n33 Theano 教程系列 http://localhost:4000tutorials/machine-learning/theano/\n\nDistributed Crawling...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nDistributed Parsing...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nAnalysing...\n34 Actor Critic (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-1-actor-critic/\n35 Numpy 基础运算1 http://localhost:4000tutorials/data-manipulation/np-pd/2-3-np-math1/\n36 读写文件 3 http://localhost:4000tutorials/python-basic/basic/08-3-read-file3/\n37 小例子 http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-1-general-rl/\n38 Why Keras? http://localhost:4000tutorials/machine-learning/keras/1-1-why/\n39 什么是 强化学习 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-1-A-RL/\n40 Actor Critic http://localhost:4000tutorials/machine-learning/ML-intro/4-08-AC/\n41 临时修改 (stash) http://localhost:4000tutorials/others/git/4-4-stash/\n42 什么是 DQN http://localhost:4000tutorials/machine-learning/torch/4-05-A-DQN/\n43 messagebox 弹窗 http://localhost:4000tutorials/python-basic/tkinter/2-10-messagebox/\n44 Canvas 画布 http://localhost:4000tutorials/python-basic/tkinter/2-07-canvas/\n45 交叉验证 2 Cross-validation http://localhost:4000tutorials/machine-learning/sklearn/3-3-cross-validation2/\n46 Image 图片 http://localhost:4000tutorials/data-manipulation/plt/3-4-image/\n47 线程锁 Lock http://localhost:4000tutorials/python-basic/threading/6-lock/\n48 什么是 Asynchronous Advantage Actor-Critic (A3C) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-3-A1-A3C/\n49 神经进化 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-01-neuro-evolution/\n50 Prioritized Experience Replay (DQN) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-6-prioritized-replay/\n51 RNN 循环神经网络 (回归) http://localhost:4000tutorials/machine-learning/torch/4-03-RNN-regression/\n52 Classification 分类学习 http://localhost:4000tutorials/machine-learning/tensorflow/5-01-classifier/\n53 什么是 Policy Gradients http://localhost:4000tutorials/machine-learning/reinforcement-learning/5-1-A-PG/\n54 Sarsa http://localhost:4000tutorials/machine-learning/ML-intro/4-04-sarsa/\n55 LSTM RNN 循环神经网络 (LSTM) http://localhost:4000tutorials/machine-learning/ML-intro/2-4-LSTM/\n56 神经网络在干嘛 http://localhost:4000tutorials/machine-learning/tensorflow/1-3-what-does-NN-do/\n57 为什么用 Matplotlib http://localhost:4000tutorials/data-manipulation/plt/1-1-why/\n58 while 循环 http://localhost:4000tutorials/python-basic/basic/03-1-while/\n59 Legend 图例 http://localhost:4000tutorials/data-manipulation/plt/2-5-lagend/\n60 基本用法 http://localhost:4000tutorials/data-manipulation/plt/2-1-basic-usage/\n61 什么是 tkinter 窗口 http://localhost:4000tutorials/python-basic/basic/13-07-tkinter/\n62 模块安装 http://localhost:4000tutorials/python-basic/basic/07-1-module-install/\n63 课程要求 http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-2-requirment/\n64 GAN (Generative Adversarial Nets 生成对抗网络) http://localhost:4000tutorials/machine-learning/torch/4-06-GAN/\n65 Pandas 基本介绍 http://localhost:4000tutorials/data-manipulation/np-pd/3-1-pd-intro/\n66 if elif else 判断 http://localhost:4000tutorials/python-basic/basic/04-3-if-elif-else/\n67 自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/ML-intro/2-5-autoencoder/\n68 什么是自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/keras/2-6-A-autoencoder/\n69 自编码 Autoencoder (非监督学习) http://localhost:4000tutorials/machine-learning/tensorflow/5-11-autoencoder/\n70 Dueling DQN (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-7-dueling-DQN/\n71 for 循环 http://localhost:4000tutorials/python-basic/basic/03-2-for/\n72 Linux 基本指令 mkdir, rmdir 和 rm http://localhost:4000tutorials/others/linux-basic/2-03-basic-command/\n73 Numpy 基础运算2 http://localhost:4000tutorials/data-manipulation/np-pd/2-4-np-math2/\n74 Github 在线代码管理 http://localhost:4000tutorials/others/git/5-1-github/\n75 CNN 卷积神经网络 3 http://localhost:4000tutorials/machine-learning/tensorflow/5-05-CNN3/\n76 Asynchronous Advantage Actor-Critic (A3C) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-3-A3C/\n77 Regressor 回归 http://localhost:4000tutorials/machine-learning/keras/2-1-regressor/\n78 Q-learning 思维决策 http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-3-tabular-q2/\n79 Subplot 分格显示 http://localhost:4000tutorials/data-manipulation/plt/4-2-subplot2/\n80 Pandas plot 出图 http://localhost:4000tutorials/data-manipulation/np-pd/3-8-pd-plot/\n81 批标准化 (Batch Normalization) http://localhost:4000tutorials/machine-learning/ML-intro/3-08-batch-normalization/\n82 Policy Gradients http://localhost:4000tutorials/machine-learning/ML-intro/4-07-PG/\n83 什么是多线程 Threading http://localhost:4000tutorials/python-basic/threading/1-why/\n84 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/ML-intro/2-8-gradient-descent/\n85 为什么 Torch 是动态的 http://localhost:4000tutorials/machine-learning/torch/5-01-dynamic/\n86 保存提取 http://localhost:4000tutorials/machine-learning/torch/3-04-save-reload/\n87 添加线程 Thread http://localhost:4000tutorials/python-basic/threading/2-add-thread/\n88 DQN 神经网络 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-2-DQN2/\n89 什么是自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/torch/4-04-A-autoencoder/\n90 Autoencoder 自编码 http://localhost:4000tutorials/machine-learning/keras/2-6-autoencoder/\n91 元组 列表 http://localhost:4000tutorials/python-basic/basic/11-1-array-list/\n92 Sarsa-lambda http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-3-tabular-sarsa-lambda/\n93 设置坐标轴2 http://localhost:4000tutorials/data-manipulation/plt/2-4-axis2/\n94 函数参数 http://localhost:4000tutorials/python-basic/basic/05-2-def-parameters/\n95 神经网络在做什么 http://localhost:4000tutorials/machine-learning/theano/1-3-NN-job/\n96 GIL 不一定有效率 http://localhost:4000tutorials/python-basic/threading/5-GIL/\n97 Pandas 设置值 http://localhost:4000tutorials/data-manipulation/np-pd/3-3-pd-assign/\n98 Pandas 选择数据 http://localhost:4000tutorials/data-manipulation/np-pd/3-2-pd-indexing/\n99 CNN 卷积神经网络 http://localhost:4000tutorials/machine-learning/torch/4-01-CNN/\n100 Placeholder 传入值 http://localhost:4000tutorials/machine-learning/tensorflow/2-5-placeholde/\n101 Tensorboard 可视化好帮手 1 http://localhost:4000tutorials/machine-learning/tensorflow/4-1-tensorboard1/\n102 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/theano/1-1-A-ANN-and-NN/\n103 回到从前 (checkout 针对单个文件) http://localhost:4000tutorials/others/git/3-2-checkout/\n104 为什么用强化学习 Why? http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-1-why/\n105 Linux 基本指令 touch, cp 和 mv http://localhost:4000tutorials/others/linux-basic/2-02-basic-command/\n106 OpenAI gym 环境库 http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-4-gym/\n107 NEAT 强化学习 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-03-neat-reinforcement-learning/\n108 什么是卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/tensorflow/5-03-A-CNN/\n109 Tensorflow 2017 更新 http://localhost:4000tutorials/machine-learning/tensorflow/5-14-tf2017/\n110 continue & break http://localhost:4000tutorials/python-basic/basic/13-01-continue-break/\n111 multiprocessing 什么是多进程 http://localhost:4000tutorials/python-basic/basic/13-06-multiprocessing/\n112 Saver 保存读取 http://localhost:4000tutorials/machine-learning/tensorflow/5-06-save/\n113 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/1-1-A-ANN-and-NN/\n114 什么是 Sarsa http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-1-A-sarsa/\n115 神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/ML-intro/2-1-NN/\n116 通用学习模式 http://localhost:4000tutorials/machine-learning/sklearn/2-2-general-pattern/\n117 第一个版本库 Repository http://localhost:4000tutorials/others/git/2-1-repository/\n118 Dropout 缓解过拟合 http://localhost:4000tutorials/machine-learning/torch/5-03-dropout/\n119 Sarsa 算法更新 http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-1-tabular-sarsa1/\n120 set 找不同 http://localhost:4000tutorials/python-basic/basic/13-09-set/\n121 多维列表 http://localhost:4000tutorials/python-basic/basic/11-3-multi-dim-list/\n122 检验神经网络 (Evaluation) http://localhost:4000tutorials/machine-learning/sklearn/3-2-A-Evaluate-NN/\n123 3D 数据 http://localhost:4000tutorials/data-manipulation/plt/3-5-3d/\n124 Regression 回归例子 http://localhost:4000tutorials/machine-learning/theano/3-2-regression/\n125 激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/ML-intro/3-04-activation-function/\n126 Scatter 散点图 http://localhost:4000tutorials/data-manipulation/plt/3-1-scatter/\n127 什么是神经网络进化 (Neuro-Evolution) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-00-neuro-evolution/\n128 什么是循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/tensorflow/5-07-A-RNN/\n129 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/tensorflow/1-1-B-NN/\n130 DQN 算法更新 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-1-DQN1/\n131 怎么样从 MacOS 或 Linux 通过 SSH 远程 Linux http://localhost:4000tutorials/others/linux-basic/4-01-ssh-from-linux-or-mac/\n132 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/keras/1-1-B-NN/\n133 区分类型 (分类) http://localhost:4000tutorials/machine-learning/torch/3-02-classification/\n134 Optimizer 优化器 http://localhost:4000tutorials/machine-learning/torch/3-06-optimizer/\n135 AlphaGo Zero 为什么更厉害? http://localhost:4000tutorials/machine-learning/ML-intro/4-11-AlphaGo-zero/\n136 Deep Deterministic Policy Gradient (DDPG) http://localhost:4000tutorials/machine-learning/ML-intro/4-09-DDPG/\n137 if 判断 http://localhost:4000tutorials/python-basic/basic/04-1-if/\n138 class 类 init 功能 http://localhost:4000tutorials/python-basic/basic/09-2-class-init/\n139 Numpy copy & deep copy http://localhost:4000tutorials/data-manipulation/np-pd/2-8-np-copy/\n140 Tensorboard 可视化好帮手 2 http://localhost:4000tutorials/machine-learning/tensorflow/4-2-tensorboard2/\n141 Frame 框架 http://localhost:4000tutorials/python-basic/tkinter/2-09-frame/\n142 Microbial Genetic Algorithm http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-04-microbial-genetic-algorithm/\n143 NEAT 监督学习 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-02-neat-supervised-learning/\n144 安装 http://localhost:4000tutorials/python-basic/basic/01-1-install/\n145 怎么样从 Windows 通过 SSH 远程 Linux http://localhost:4000tutorials/others/linux-basic/4-02-ssh-from-windows/\n146 什么是 Actor Critic http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-1-A-AC/\n147 遗传算法 (Genetic Algorithm) http://localhost:4000tutorials/machine-learning/ML-intro/5-01-genetic-algorithm/\n148 CNN 卷积神经网络 http://localhost:4000tutorials/machine-learning/keras/2-3-CNN/\n149 选择好特征 (Good Features) http://localhost:4000tutorials/machine-learning/ML-intro/3-03-choose-feature/\n150 什么是 Q Leaning http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-2-A-q-learning/\n151 Evolution Strategy 强化学习 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/4-04-evolution-strategy-reinforcement-learning/\n152 兼容 backend http://localhost:4000tutorials/machine-learning/keras/1-3-backend/\n153 什么是过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/theano/3-5-A-overfitting/\n154 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/theano/1-1-C-gradient-descent/\n155 Numpy 和 Pandas 安装 http://localhost:4000tutorials/data-manipulation/np-pd/1-2-install/\n156 例子 旅行商人问题 (Travel Sales Problem) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-03-genetic-algorithm-travel-sales-problem/\n157 Distributed Proximal Policy Optimization (DPPO) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-4-DPPO/\n158 Pytorch 安装 http://localhost:4000tutorials/machine-learning/torch/1-2-install/\n159 基础数学运算 http://localhost:4000tutorials/python-basic/basic/02-2-basic-math/\n160 激励函数 (Activation) http://localhost:4000tutorials/machine-learning/torch/2-03-activation/\n161 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/tensorflow/1-1-D-feature-representation/\n162 例子3 结果可视化 http://localhost:4000tutorials/machine-learning/tensorflow/3-3-visualize-result/\n163 Activation function 激励函数 http://localhost:4000tutorials/machine-learning/theano/2-4-activation/\n164 可视化结果 回归例子 http://localhost:4000tutorials/machine-learning/theano/3-3-visualize/\n165 什么是循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/torch/4-02-A-RNN/\n166 什么是 Deep Deterministic Policy Gradient (DDPG) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-2-A-DDPG/\n167 例子 登录窗口1 http://localhost:4000tutorials/python-basic/tkinter/3-01-example1/\n168 怎么样用 TeamViewer 和 VNC 从远程控制电脑 http://localhost:4000tutorials/others/linux-basic/4-04-teamviewer-vnc/\n169 什么是循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/keras/2-4-A-RNN/\n170 Pandas 导入导出 http://localhost:4000tutorials/data-manipulation/np-pd/3-5-pd-to/\n171 选择学习方法 http://localhost:4000tutorials/machine-learning/sklearn/2-1-select-method/\n172 class 类 http://localhost:4000tutorials/python-basic/basic/09-1-class/\n173 什么是激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/theano/2-4-A-activation-function/\n174 特征标准化 (Feature Normalization) http://localhost:4000tutorials/machine-learning/ML-intro/3-02-normalization/\n175 Policy Gradients 思维决策 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/5-2-policy-gradient-softmax2/\n176 RNN LSTM (回归例子可视化) http://localhost:4000tutorials/machine-learning/tensorflow/5-10-RNN4/\n177 关系拟合 (回归) http://localhost:4000tutorials/machine-learning/torch/3-01-regression/\n178 正规化 Normalization http://localhost:4000tutorials/machine-learning/sklearn/3-1-normalization/\n179 GPU 加速运算 http://localhost:4000tutorials/machine-learning/torch/5-02-GPU/\n180 DQN 强化学习 http://localhost:4000tutorials/machine-learning/torch/4-05-DQN/\n181 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/ML-intro/2-7-feature-representation/\n182 加速神经网络训练 (Speed Up Training) http://localhost:4000tutorials/machine-learning/ML-intro/3-06-speed-up-learning/\n183 交叉验证 1 Cross-validation http://localhost:4000tutorials/machine-learning/sklearn/3-2-cross-validation1/\n184 读写文件 1 http://localhost:4000tutorials/python-basic/basic/08-1-read-file/\n185 回到从前 (reset) http://localhost:4000tutorials/others/git/3-1-reset/\n186 什么是 Tkinter http://localhost:4000tutorials/python-basic/tkinter/1-1-why/\n187 Theano 安装 http://localhost:4000tutorials/machine-learning/theano/1-2-install/\n188 Save 保存 提取 http://localhost:4000tutorials/machine-learning/theano/3-6-save/\n189 sklearn 常用属性与功能 http://localhost:4000tutorials/machine-learning/sklearn/2-4-model-attributes/\n190 变量 (Variable) http://localhost:4000tutorials/machine-learning/torch/2-02-variable/\n191 threading 什么是多线程 http://localhost:4000tutorials/python-basic/basic/13-05-threading/\n192 RNN LSTM 循环神经网络 (分类例子) http://localhost:4000tutorials/machine-learning/tensorflow/5-08-RNN2/\n193 记录修改 (log & diff) http://localhost:4000tutorials/others/git/2-2-modified/\n194 dictionary 字典 http://localhost:4000tutorials/python-basic/basic/11-4-dictionary/\n195 Session 会话控制 http://localhost:4000tutorials/machine-learning/tensorflow/2-3-session/\n196 神经网络进化 (Neuro-Evolution) http://localhost:4000tutorials/machine-learning/ML-intro/5-03-neuro-evolution/\n197 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/tensorflow/1-1-C-gradient-descent/\n198 例子3 添加层 def add_layer() http://localhost:4000tutorials/machine-learning/tensorflow/3-1-add-layer/\n199 Scale 尺度 http://localhost:4000tutorials/python-basic/tkinter/2-05-scale/\n200 例子 配对句子 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-02-genetic-algorithm-match-phrase/\n201 rebase 分支冲突 http://localhost:4000tutorials/others/git/4-3-rebase/\n202 Batch Normalization 批标准化 http://localhost:4000tutorials/machine-learning/torch/5-04-batch-normalization/\n203 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/theano/1-1-D-feature-representation/\n204 自己的云计算, 把 Linux 当成你的云计算平台 http://localhost:4000tutorials/others/linux-basic/5-01-remote-learning/\n205 例子 登录窗口2 http://localhost:4000tutorials/python-basic/tkinter/3-02-example2/\n206 机器学习 (Machine Learning) http://localhost:4000tutorials/machine-learning/sklearn/1-1-A-ML/\n207 自己的模块 http://localhost:4000tutorials/python-basic/basic/12-2-personal-module/\n208 Matplotlib 安装 http://localhost:4000tutorials/data-manipulation/plt/1-2-install/\n209 进化策略 (Evolution Strategy) http://localhost:4000tutorials/machine-learning/ML-intro/5-02-evolution-strategy/\n210 list 列表 http://localhost:4000tutorials/python-basic/basic/11-2-list/\n211 Double DQN (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-5-double_DQN/\n212 Numpy 的创建 array http://localhost:4000tutorials/data-manipulation/np-pd/2-2-np-array/\n213 遗传算法 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-01-genetic-algorithm/\n214 Torch 或 Numpy http://localhost:4000tutorials/machine-learning/torch/2-01-torch-numpy/\n215 快速搭建法 http://localhost:4000tutorials/machine-learning/torch/3-03-fast-nn/\n216 例子 登录窗口3 http://localhost:4000tutorials/python-basic/tkinter/3-03-example3/\n217 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/ML-intro/2-0-ANN-and-NN/\n218 zip lambda map http://localhost:4000tutorials/python-basic/basic/13-03-zip-lambda-map/\n219 Batch Normalization 批标准化 http://localhost:4000tutorials/machine-learning/tensorflow/5-13-BN/\n220 pack grid place 放置位置 http://localhost:4000tutorials/python-basic/tkinter/2-11-pack-grid-place/\n221 scope 命名方法 http://localhost:4000tutorials/machine-learning/tensorflow/5-12-scope/\n222 Subplot 多合一显示 http://localhost:4000tutorials/data-manipulation/plt/4-1-subpot1/\n223 L1 / L2 正规化 (Regularization) http://localhost:4000tutorials/machine-learning/ML-intro/3-09-l1l2regularization/\n224 强化学习方法汇总 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/ML-intro/4-02-RL-methods/\n225 次坐标轴 http://localhost:4000tutorials/data-manipulation/plt/4-4-sencondary-axis/\n226 什么是 LSTM 循环神经网络 http://localhost:4000tutorials/machine-learning/keras/2-4-B-LSTM/\n227 Asynchronous Advantage Actor-Critic (A3C) http://localhost:4000tutorials/machine-learning/ML-intro/4-10-A3C/\n228 强化学习 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/ML-intro/4-01-RL/\n229 Numpy array 分割 http://localhost:4000tutorials/data-manipulation/np-pd/2-7-np-split/\n230 Numpy array 合并 http://localhost:4000tutorials/data-manipulation/np-pd/2-6-np-concat/\n231 if else 判断 http://localhost:4000tutorials/python-basic/basic/04-2-if-else/\n232 进程锁 Lock http://localhost:4000tutorials/python-basic/multiprocessing/7-lock/\n233 import 模块 http://localhost:4000tutorials/python-basic/basic/12-1-import/\n234 RNN Regressor 循环神经网络 http://localhost:4000tutorials/machine-learning/keras/2-5-RNN-LSTM-Regressor/\n235 sklearn 强大数据库 http://localhost:4000tutorials/machine-learning/sklearn/2-3-database/\n236 Regularization 正规化 http://localhost:4000tutorials/machine-learning/theano/3-5-regularization/\n237 效率对比 threading & multiprocessing http://localhost:4000tutorials/python-basic/multiprocessing/4-comparison/\n238 Why Numpy & Pandas? http://localhost:4000tutorials/data-manipulation/np-pd/1-1-why/\n239 共享内存 shared memory http://localhost:4000tutorials/python-basic/multiprocessing/6-shared-memory/\n240 什么是激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/torch/2-03-A-activation-function/\n241 Menubar 菜单 http://localhost:4000tutorials/python-basic/tkinter/2-08-menubar/\n242 Annotation 标注 http://localhost:4000tutorials/data-manipulation/plt/2-6-annotation/\n243 RNN 循环神经网络 (分类) http://localhost:4000tutorials/machine-learning/torch/4-02-RNN-classification/\n244 什么是进化策略 (Evolution Strategy) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-00-evolution-strategy/\n245 例子3 建造神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/3-2-create-NN/\n246 (1+1)-ES http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-02-evolution-strategy-1+1-ES/\n247 保存模型 http://localhost:4000tutorials/machine-learning/sklearn/3-5-save/\n248 进化算法 简介 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/1-01-intro/\n249 Sarsa 思维决策 http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-2-tabular-sarsa2/\n250 Dropout 解决 overfitting http://localhost:4000tutorials/machine-learning/tensorflow/5-02-dropout/\n251 什么是激励函数 (Activation Function) http://localhost:4000tutorials/machine-learning/tensorflow/2-6-A-activation-function/\n252 进程池 Pool http://localhost:4000tutorials/python-basic/multiprocessing/5-pool/\n253 Asyncio http://localhost:4000tutorials/data-manipulation/scraping/asyncio/\n254 添加进程 Process http://localhost:4000tutorials/python-basic/multiprocessing/2-add/\n255 存储进程输出 Queue http://localhost:4000tutorials/python-basic/multiprocessing/3-queue/\n256 过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/ML-intro/3-05-overfitting/\n257 图中图 http://localhost:4000tutorials/data-manipulation/plt/4-3-plot-in-plot/\n258 Save & reload 保存提取 http://localhost:4000tutorials/machine-learning/keras/3-1-save/\n259 Checkbutton 勾选项 http://localhost:4000tutorials/python-basic/tkinter/2-06-checkbutton/\n260 分支 (branch) http://localhost:4000tutorials/others/git/4-1-branch/\n261 Q-learning 算法更新 http://localhost:4000tutorials/machine-learning/reinforcement-learning/2-2-tabular-q1/\n262 变量 variable http://localhost:4000tutorials/python-basic/basic/02-3-variable/\n263 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/torch/1-1-C-gradient-descent/\n264 生成对抗网络 (GAN) http://localhost:4000tutorials/machine-learning/ML-intro/2-6-GAN/\n265 什么是 Sarsa(lambda) http://localhost:4000tutorials/machine-learning/reinforcement-learning/3-3-A-sarsa-lambda/\n266 Pandas 合并 merge http://localhost:4000tutorials/data-manipulation/np-pd/3-7-pd-merge/\n267 遗传算法 (Genetic Algorithm) http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/2-00-genetic-algorithm/\n268 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/keras/1-1-D-feature-representation/\n269 什么是批标准化 (Batch Normalization) http://localhost:4000tutorials/machine-learning/torch/5-04-A-batch-normalization/\n270 自己的云计算, 多电脑共享你云端文件 http://localhost:4000tutorials/others/linux-basic/5-02-share-folder/\n271 从头开始做一个机器手臂3 写动态环境 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch3/\n272 定义 Layer 类 http://localhost:4000tutorials/machine-learning/theano/3-1-layer/\n273 什么是 LSTM 循环神经网络 http://localhost:4000tutorials/machine-learning/torch/4-02-B-LSTM/\n274 pickle 保存数据 http://localhost:4000tutorials/python-basic/basic/13-08-pickle/\n275 Classifier 分类 http://localhost:4000tutorials/machine-learning/keras/2-2-classifier/\n276 CNN 卷积神经网络 1 http://localhost:4000tutorials/machine-learning/tensorflow/5-03-CNN1/\n277 科普: 神经网络的黑盒不黑 http://localhost:4000tutorials/machine-learning/torch/1-1-D-feature-representation/\n278 什么是生成对抗网络 (GAN) http://localhost:4000tutorials/machine-learning/torch/4-06-A-GAN/\n279 强化学习方法汇总 (Reinforcement Learning) http://localhost:4000tutorials/machine-learning/reinforcement-learning/1-1-B-RL-methods/\n280 Keras 安装 http://localhost:4000tutorials/machine-learning/keras/1-2-install/\n281 Entry & Text 输入, 文本框 http://localhost:4000tutorials/python-basic/tkinter/2-02-entry-text/\n282 RNN 循环神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/5-07-RNN1/\n283 Policy Gradients 算法更新 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/5-1-policy-gradient-softmax1/\n284 Function 用法 http://localhost:4000tutorials/machine-learning/theano/2-2-function/\n285 优化器 optimizer http://localhost:4000tutorials/machine-learning/tensorflow/3-4-optimizer/\n286 Pandas 合并 concat http://localhost:4000tutorials/data-manipulation/np-pd/3-6-pd-concat/\n287 Tensorflow 安装 http://localhost:4000tutorials/machine-learning/tensorflow/1-2-install/\n288 从头开始做一个机器手臂4 加入强化学习算法 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch4/\n289 Deep Deterministic Policy Gradient (DDPG) (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/6-2-DDPG/\n290 什么是卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/keras/2-3-A-CNN/\n291 什么是批标准化 (Batch Normalization) http://localhost:4000tutorials/machine-learning/tensorflow/5-13-A-batch-normalization/\n292 什么是过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/tensorflow/5-02-A-overfitting/\n293 函数默认参数 http://localhost:4000tutorials/python-basic/basic/05-3-def-default-parameters/\n294 Linux 文件权限 http://localhost:4000tutorials/others/linux-basic/3-01-file-permissions/\n295 Animation 动画 http://localhost:4000tutorials/data-manipulation/plt/5-1-animation/\n296 全局 & 局部 变量 http://localhost:4000tutorials/python-basic/basic/06-1-global-local/\n297 从头开始做一个机器手臂2 写静态环境 http://localhost:4000tutorials/machine-learning/ML-practice/RL-build-arm-from-scratch2/\n298 Natural Evolution Strategy http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-03-evolution-strategy-natural-evolution-strategy/\n299 Pandas 处理丢失数据 http://localhost:4000tutorials/data-manipulation/np-pd/3-4-pd-nan/\n300 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/theano/1-1-B-NN/\n301 Numpy 索引 http://localhost:4000tutorials/data-manipulation/np-pd/2-5-np-indexing/\n302 Radiobutton 选择按钮 http://localhost:4000tutorials/python-basic/tkinter/2-04-radiobutton/\n303 激励函数 Activation Function http://localhost:4000tutorials/machine-learning/tensorflow/2-6-activation/\n304 Why Theano? http://localhost:4000tutorials/machine-learning/theano/1-1-why/\n305 Classification 分类学习 http://localhost:4000tutorials/machine-learning/theano/3-4-classification/\n306 了解网页结构 http://localhost:4000tutorials/data-manipulation/scraping/1-01-understand-website/\n307 Variable 变量 http://localhost:4000tutorials/machine-learning/tensorflow/2-4-variable/\n308 加速神经网络训练 (Speed Up Training) http://localhost:4000tutorials/machine-learning/tensorflow/3-4-A-speed-up-learning/\n309 Linux 基本指令 nano 和 cat http://localhost:4000tutorials/others/linux-basic/2-04-basic-command/\n310 处理结构 http://localhost:4000tutorials/machine-learning/tensorflow/2-1-structure/\n311 循环神经网络 RNN (Recurrent Neural Network) http://localhost:4000tutorials/machine-learning/ML-intro/2-3-RNN/\n312 Bar 柱状图 http://localhost:4000tutorials/data-manipulation/plt/3-2-Bar/\n313 Shared 变量 http://localhost:4000tutorials/machine-learning/theano/2-3-shared-variable/\n314 总结和更多 http://localhost:4000tutorials/machine-learning/theano/4-1-summary/\n315 print 功能 http://localhost:4000tutorials/python-basic/basic/02-1-print/\n316 merge 分支冲突 http://localhost:4000tutorials/others/git/4-2-merge-conflict/\n317 join 功能 http://localhost:4000tutorials/python-basic/threading/3-join/\n318 Why Git? http://localhost:4000tutorials/others/git/1-1-why/\n319 RNN Classifier 循环神经网络 http://localhost:4000tutorials/machine-learning/keras/2-4-RNN-classifier/\n320 def 函数 http://localhost:4000tutorials/python-basic/basic/05-1-def/\n321 安装 Ubuntu 17.10 http://localhost:4000tutorials/others/linux-basic/1-2-install/\n322 Linux 基本指令 ls 和 cd http://localhost:4000tutorials/others/linux-basic/2-01-basic-command/\n323 Sklearn 安装 http://localhost:4000tutorials/machine-learning/sklearn/1-2-install/\n324 AutoEncoder (自编码/非监督学习) http://localhost:4000tutorials/machine-learning/torch/4-04-autoencoder/\n325 DQN http://localhost:4000tutorials/machine-learning/ML-intro/4-06-DQN/\n326 为什么用 Numpy 还是慢, 你用对了吗? http://localhost:4000tutorials/data-manipulation/np-pd/4-1-speed-up-numpy/\n327 什么是过拟合 (Overfitting) http://localhost:4000tutorials/machine-learning/torch/5-03-A-overfitting/\n328 机器学习 (Machine Learning) http://localhost:4000tutorials/machine-learning/ML-intro/1-1-machine-learning/\n329 Sarsa(lambda) http://localhost:4000tutorials/machine-learning/ML-intro/4-05-sarsa-lambda/\n330 卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/ML-intro/2-2-CNN/\n331 交叉验证 3 Cross-validation http://localhost:4000tutorials/machine-learning/sklearn/3-4-cross-validation3/\n332 例子2 http://localhost:4000tutorials/machine-learning/tensorflow/2-2-example2/\n333 什么是 DQN http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-1-A-DQN/\n334 为什么选 Tensorflow? http://localhost:4000tutorials/machine-learning/tensorflow/1-1-why/\n335 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/keras/1-1-A-ANN-and-NN/\n336 设置坐标轴1 http://localhost:4000tutorials/data-manipulation/plt/2-3-axis1/\n337 Why Sklearn? http://localhost:4000tutorials/machine-learning/sklearn/1-1-why/\n338 Q Leaning http://localhost:4000tutorials/machine-learning/ML-intro/4-03-q-learning/\n339 进化策略 http://localhost:4000tutorials/machine-learning/evolutionary-algorithm/3-01-evolution-strategy/\n340 神经网络 梯度下降 http://localhost:4000tutorials/machine-learning/keras/1-1-C-gradient-descent/\n341 什么是 Multiprocessing http://localhost:4000tutorials/python-basic/multiprocessing/1-why/\n342 Listbox 列表部件 http://localhost:4000tutorials/python-basic/tkinter/2-03-listbox/\n343 Numpy 属性 http://localhost:4000tutorials/data-manipulation/np-pd/2-1-np-attributes/\n344 科普: 人工神经网络 VS 生物神经网络 http://localhost:4000tutorials/machine-learning/torch/1-1-A-ANN-and-NN/\n345 RNN LSTM (回归例子) http://localhost:4000tutorials/machine-learning/tensorflow/5-09-RNN3/\n346 CNN 卷积神经网络 2 http://localhost:4000tutorials/machine-learning/tensorflow/5-04-CNN2/\n347 figure 图像 http://localhost:4000tutorials/data-manipulation/plt/2-2-figure/\n348 用 Tensorflow 可视化梯度下降 http://localhost:4000tutorials/machine-learning/tensorflow/5-15-tf-gradient-descent/\n349 储存进程结果 Queue http://localhost:4000tutorials/python-basic/threading/4-queue/\n350 Git 安装 http://localhost:4000tutorials/others/git/1-2-install/\n351 tick 能见度 http://localhost:4000tutorials/data-manipulation/plt/2-7-tick-visibility/\n352 给你的 Ubuntu 安装软件 http://localhost:4000tutorials/others/linux-basic/1-4-install-software/\n353 基本用法 http://localhost:4000tutorials/machine-learning/theano/2-1-basic-usage/\n354 加速神经网络训练 (Speed Up Training) http://localhost:4000tutorials/machine-learning/torch/3-06-A-speed-up-learning/\n355 Contours 等高线图 http://localhost:4000tutorials/data-manipulation/plt/3-3-contours/\n356 处理不均衡数据 (Imbalanced data) http://localhost:4000tutorials/machine-learning/ML-intro/3-07-imbalanced-data/\n357 批训练 http://localhost:4000tutorials/machine-learning/torch/3-05-train-on-batch/\n358 DQN 思维决策 (Tensorflow) http://localhost:4000tutorials/machine-learning/reinforcement-learning/4-3-DQN3/\n359 快速了解 Ubuntu 17.10 基本界面 http://localhost:4000tutorials/others/linux-basic/1-3-system-overview/\n360 Why Pytorch? http://localhost:4000tutorials/machine-learning/torch/1-1-why/\n361 什么是卷积神经网络 CNN (Convolutional Neural Network) http://localhost:4000tutorials/machine-learning/torch/4-01-A-CNN/\n362 Label & Button 标签和按钮 http://localhost:4000tutorials/python-basic/tkinter/2-01-label-button/\n363 读写文件 2 http://localhost:4000tutorials/python-basic/basic/08-2-read-file2/\n364 什么是神经网络 (Neural Network) http://localhost:4000tutorials/machine-learning/torch/1-1-B-NN/\n365 什么是自编码 (Autoencoder) http://localhost:4000tutorials/machine-learning/tensorflow/5-11-A-autoencoder/\n366 Why Linux? http://localhost:4000tutorials/others/linux-basic/1-1-why/\n367 怎么样从手机 (Android安卓/IOS苹果) 通过 SSH 远程 Linux http://localhost:4000tutorials/others/linux-basic/4-03-ssh-from-phone/\n368 copy & deepcopy 浅复制 & 深复制 http://localhost:4000tutorials/python-basic/basic/13-04-copy/\n369 检验神经网络 (Evaluation) http://localhost:4000tutorials/machine-learning/ML-intro/3-01-Evaluate-NN/\n370 input 输入 http://localhost:4000tutorials/python-basic/basic/10-1-input/\n371 什么是 LSTM 循环神经网络 http://localhost:4000tutorials/machine-learning/tensorflow/5-07-B-LSTM/\n372 try 错误处理 http://localhost:4000tutorials/python-basic/basic/13-02-try/\nTotal time: 16.3 s\n"
     ]
    }
   ],
   "source": [
    "unseen = set([base_url,])\n",
    "seen = set()\n",
    "\n",
    "pool = mp.Pool(4)                       \n",
    "count, t1 = 1, time.time()\n",
    "while len(unseen) != 0:                 # still get some url to visit\n",
    "    if restricted_crawl and len(seen) > 20:\n",
    "            break\n",
    "    print('\\nDistributed Crawling...')\n",
    "    crawl_jobs = [pool.apply_async(crawl, args=(url,)) for url in unseen]\n",
    "    htmls = [j.get() for j in crawl_jobs]                                       # request connection\n",
    "\n",
    "    print('\\nDistributed Parsing...')\n",
    "    parse_jobs = [pool.apply_async(parse, args=(html,)) for html in htmls]\n",
    "    results = [j.get() for j in parse_jobs]                                     # parse html\n",
    "\n",
    "    print('\\nAnalysing...')\n",
    "    seen.update(unseen)         # seen the crawled\n",
    "    unseen.clear()              # nothing unseen\n",
    "\n",
    "    for title, page_urls, url in results:\n",
    "        print(count, title, url)\n",
    "        count += 1\n",
    "        unseen.update(page_urls - seen)     # get new url to crawl\n",
    "print('Total time: %.1f s' % (time.time()-t1, ))    # 16 s !!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}