Merge pull request #450 from yandexdataschool/spring20

Merge spring20 into master

README.md

@@ -1,5 +1,5 @@
+# Practical_RL
 
-# Practical_RL [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/yandexdataschool/practical_rl/master)
 An open course on reinforcement learning in the wild.
 Taught on-campus at [HSE](https://cs.hse.ru) and [YSDA](https://yandexdataschool.com/)  and maintained to be friendly to online students (both english and russian).
 
@@ -12,17 +12,17 @@ Taught on-campus at [HSE](https://cs.hse.ru) and [YSDA](https://yandexdataschool
 [![Github contributors](https://img.shields.io/github/contributors/yandexdataschool/Practical_RL.svg?logo=github&logoColor=white)](https://github.com/yandexdataschool/Practical_RL/graphs/contributors)
 
 # Course info
-* __Chat room__ for YSDA & HSE students is [here](https://t.me/joinchat/CDFcMVcoAQvEiI9WAo1pEQ)
+* __Chat room__ for YSDA & HSE students is [here](http://t.me/practical_rl_2020)
 * __Grading__ rules for YSDA & HSE students is [here](https://github.com/yandexdataschool/Practical_RL/wiki/Homeworks-and-grading)
 
 * __FAQ:__ [About the course](https://github.com/yandexdataschool/Practical_RL/wiki/Practical-RL), [Technical issues thread](https://github.com/yandexdataschool/Practical_RL/issues/1), [Lecture Slides](https://yadi.sk/d/loPpY45J3EAYfU), [Online Student Survival Guide](https://github.com/yandexdataschool/Practical_RL/wiki/Online-student's-survival-guide)
 
 * Anonymous [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSdurWw97Sm9xCyYwC8g3iB5EibITnoPJW2IkOVQYE_kcXPh6Q/viewform).
 
 * Virtual course environment: 
+    * [__Google Colab__](https://colab.research.google.com/) - set open -> github -> yandexdataschool/pracical_rl -> {branch name} and select any notebook you want.
     * [Installing dependencies](https://github.com/yandexdataschool/Practical_RL/issues/1) on your local machine (recommended).
-    * [__google colab__](https://colab.research.google.com/) - set open -> github -> yandexdataschool/pracical_rl -> {branch name} and select any notebook you want.
-    * Alternatives: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/yandexdataschool/practical_rl/master) and [Azure Notebooks](https://notebooks.azure.com/).
+    * Alternative: [Azure Notebooks](https://notebooks.azure.com/).
 
 
 # Additional materials
@@ -94,6 +94,7 @@ Course materials and teaching by: _[unordered]_
 - [Mikhail Konobeev](https://github.com/MichaelKonobeev) - seminars, hw checkups
 - [Ivan Kharitonov](https://github.com/neer201) - seminars, hw checkups
 - [Ravil Khisamov](https://github.com/zshrav) - seminars, hw checkups
+- [Anna Klepova](https://github.com/q0o0p) - hw checkups
 - [Fedor Ratnikov](https://github.com/justheuristic) - admin stuff
 
 # Contributions

docker/Dockerfile

@@ -1,75 +1,65 @@
-FROM ubuntu:16.04
+ARG device
+ARG username="jovyan"
+
+FROM ubuntu:18.04 AS base-cpu
+
+FROM nvidia/cuda:10.2-base-ubuntu18.04 AS base-gpu
+
+FROM base-$device AS base
+
+ARG username
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get -qq update && \
+    apt-get install -y wget unzip git cmake xvfb sudo freeglut3-dev ffmpeg
+
+RUN adduser --disabled-password --gecos "Default user" $username && \
+    adduser $username sudo && \
+    echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers && \
+    mkdir /notebooks && \
+    chown -R $username /notebooks
+
+RUN su $username -c \
+    "wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh \
+        -O /tmp/anaconda3.sh" && \
+    mkdir -p /opt/conda && \
+    chown -R $username /opt/conda && \
+    su $username -c "/bin/bash /tmp/anaconda3.sh -b -p /opt/conda -u" && \
+    rm /tmp/anaconda3.sh && \
+    ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
+    su $username -c "echo '. /opt/conda/etc/profile.d/conda.sh' >> ~/.bashrc" && \
+    su $username -c "echo 'conda activate base' >> ~/.bashrc"
+
+ADD deeplearning.yaml /tmp/deeplearning-template.yaml
+
+USER $username
+SHELL ["/bin/bash", "-i", "-c"]
+RUN conda install -y jupyter
+
+FROM base AS stage-cpu
+RUN sed -e "s/{tensorflow}/tensorflow/g" -e "s/{pytorch}/pytorch/g" /tmp/deeplearning-template.yaml > /tmp/deeplearning.yaml
+
+FROM base AS stage-gpu
+RUN sed -e "s/{tensorflow}/tensorflow-gpu/g" -e "s/{pytorch}/pytorch-gpu/g" /tmp/deeplearning-template.yaml > /tmp/deeplearning.yaml
+
+FROM stage-$device as final
+
 LABEL maintainer "Alexander Panin <justheuristic@gmail.com>, Dmitry Mittov <mittov@gmail.com>"
 
+ARG username
+USER $username
+SHELL ["/bin/bash", "-i", "-c"]
 
-RUN echo "deb http://archive.ubuntu.com/ubuntu trusty-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
-    apt-get -qq update && \
-    apt-get install -y cmake \
-                       wget \
-                       unzip \
-                       git \
-                       zlib1g-dev \
-                       libjpeg-dev \
-                       xvfb \
-                       libav-tools \
-                       xorg-dev \
-                       python-opengl \
-                       swig3.0 \
-                       python-dev \
-                       python3-dev \
-                       python-pip \
-                       python3-pip \
-                       libopenblas-dev \
-                       liblapack-dev \
-                       libsdl2-dev \
-                       libboost-all-dev \
-                       graphviz \
-                       gcc \
-                       g++ && \
-    ln -s /usr/bin/swig3.0 /usr/bin/swig
-
-RUN pip install --upgrade pip==9.0.3 && \
-    pip install --upgrade numpy scipy && \
-    pip install --upgrade sklearn \
-                           jupyter \
-                           tqdm \
-                           graphviz \
-                           gym gym[box2d] gym[atari] \
-                           matplotlib \
-                           seaborn && \
-    pip install --upgrade https://github.com/Theano/Theano/archive/master.zip \
-                           https://github.com/Lasagne/Lasagne/archive/master.zip \
-                           https://github.com/yandexdataschool/AgentNet/archive/master.zip \
-                           tensorflow \
-                           https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp27-cp27mu-linux_x86_64.whl \
-                           torchvision \
-                           keras     
-
-RUN pip install --upgrade  gym_pull ppaquette-gym-doom
-
-
-RUN pip3 install --upgrade pip==9.0.3 && \
-    pip3 install --upgrade numpy scipy && \
-    pip3 install --upgrade sklearn \
-                           jupyter \
-                           tqdm \
-                           graphviz \
-                           gym gym[box2d] gym[atari] \
-                           matplotlib \
-                           seaborn && \
-    pip3 install --upgrade https://github.com/Theano/Theano/archive/master.zip \
-                           https://github.com/Lasagne/Lasagne/archive/master.zip \
-                           https://github.com/yandexdataschool/AgentNet/archive/master.zip \
-                           https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl \
-                           torchvision \
-                           tensorflow \
-                           keras && \                           
-    python3 -m ipykernel.kernelspec
+RUN conda env create -f /tmp/deeplearning.yaml && \
+    conda activate deeplearning && \
+    python -m ipykernel install --user --name python3 --display-name "Python 3"
 
+RUN sudo rm /tmp/deeplearning*.yaml
 
 EXPOSE 8888
 VOLUME /notebooks
 WORKDIR /notebooks
+ENV PATH /opt/conda/bin:$PATH
 
 COPY run_jupyter.sh /
 CMD ["/run_jupyter.sh"]

docker/README.md

@@ -7,36 +7,47 @@ _tl;dr [dockerhub url](https://hub.docker.com/r/justheuristic/practical_rl/)_
 We recommend you to use either native docker (recommended for linux) or kitematic(recommended for windows).
 * Installing [kitematic](https://kitematic.com/), a simple interface to docker (all platforms)
 * Pure docker: Guide for [windows](https://docs.docker.com/docker-for-windows/), [linux](https://docs.docker.com/engine/installation/), or [macOS](https://docs.docker.com/docker-for-mac/).
+* If you want to use your GPU make sure you have [nvidia-docker](https://github.com/NVIDIA/nvidia-docker) and [NVidia driver](https://www.nvidia.com/en-us/drivers/unix/) + [CUDA 10.2](https://developer.nvidia.com/cuda-downloads) installed
 
 Below are the instructions for both approaches.
 
 ## Kitematic
-Find justheuristic/practical_rl in the search menu. Download and launch the container.
+Find dmittov/practical_rl in the search menu. Download and launch the container.
 
 Click on "web preview" screen in the top-right __or__ go to settings, ports and find at which port your jupyter is located, usually 32***.
 
 ## Native
-`docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 justheuristic/practical_rl`
+`docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 dmittov/practical_rl:spring2020-cpu`
 
 For example,
-```docker run --rm -it -v /home/myuser/Documents/practical_rl:/notebooks -p 8888:8888 justheuristic/practical_rl```
+```docker run --rm -it -v /home/myuser/Documents/practical_rl:/notebooks -p 8888:8888 dmittov/practical_rl:spring2020-cpu```
 
 
 Then you can access your jupyter in a browser at `localhost:<local_port>/?token=<token_you_see_in_container_logs>`, e.g. `localhost:8888/?token=ad1a5a0aab43efb47a9a805388fcf508d0b5f84a16e4542b&token=ad1a5a0aab43efb47a9a805388fcf508d0b5f84a16e4542b`
 
+#### GPU
+`docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 --gpus all dmittov/practical_rl:spring2020-cuda-10.2`
+
 ## Manual
 Build container
 
-`$ docker build -t dl .`
+`docker build -t practical_rl --build-arg device=cpu .`
+
+to build GPU version
 
+`docker build -t practical_rl --build-arg device=gpu .`
 
 Run it
 
-`$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 dl`
+`$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 practical_rl`
+
+to run GPU version
+
+`$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 --gpus all practical_rl`
 
 examples:
 
-```$ docker run --rm -it -v `pwd`:/notebooks -p 8888:8888 dl```
+```$ docker run --rm -it -v `pwd`:/notebooks -p 8888:8888 practical_rl```
 
 Copy the token from console and run
 http://localhost:8888/?token=<token>

docker/deeplearning.yaml

@@ -0,0 +1,32 @@
+name: deeplearning
+channels:
+  - defaults
+dependencies:
+  - graphviz
+  - python-graphviz
+  - ipykernel
+  - keras
+  - numpy=1.16.6
+  - pip
+  - scipy
+  - {tensorflow}=1.13.1
+  - tqdm
+  - pyopengl-accelerate
+  - {pytorch}
+  - torchvision
+  - opencv
+  - pip:
+    - atari-py
+    - sklearn
+    - matplotlib
+    - seaborn
+    - scikit-image
+    - imageio
+    - pyglet
+    - psutil
+    - gym
+    - gym[box2d]
+    - gym[atari]
+    - requests
+    - editdistance
+prefix: /opt/conda/envs/deeplearning

docker/run_jupyter.sh

@@ -1,3 +1,2 @@
 #!/usr/bin/env bash
-jupyter notebook --no-browser --allow-root --ip 0.0.0.0
-
+jupyter notebook --no-browser --ip 0.0.0.0

setup_colab.sh

@@ -1,8 +1,5 @@
-#!/bin/bash
-# a setup script for google colab. Will be updated
-pip install gym
-apt-get install -y xvfb
-wget https://raw.githubusercontent.com/yandexdataschool/Practical_DL/fall18/xvfb -O ../xvfb
-apt-get install -y python-opengl ffmpeg
-pip install pyglet==1.2.4
+#!/usr/bin/env bash
 
+apt-get -qq update
+apt-get -qq install -y xvfb
+wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/spring20/xvfb -O ../xvfb

week01_intro/README.md

@@ -1,9 +1,8 @@
 ## Materials:
 * [__Lecture slides__](https://yadi.sk/i/-EUHXUXOTC5t9Q)
 * __Russian:__
-  * Intro to RL - [video](https://yadi.sk/i/bMo0qa-x3DoqkS)
-  * Blackbox optimization - [video](https://yadi.sk/i/5yf_4oGI3EDJhJ)
-  * Seminar - [video](https://yadi.sk/i/dPsWYMK13EDJj7) _only covering crossentropy method_
+  * Intro to RL - [video](https://yadi.sk/i/HpRw6zYv3SPmFf)
+  * Seminar - [video](https://yadi.sk/i/gRTVCHfM3SPmHo)
 
 * __English:__
   * [__main__] Video-intro by David Silver (english) - [video](https://www.youtube.com/watch?v=2pWv7GOvuf0)
@@ -20,14 +19,14 @@
 * About Differential evolution (english) - [pdf](http://jvanderw.une.edu.au/DE_1.pdf)
 * Video on Ant Colony Algorithm (english) - [video](https://www.youtube.com/watch?v=D58nLNLkb0I)
 * Longer video on Ant Colony Algorithm (english) - [video](https://www.youtube.com/watch?v=xpyKmjJuqhk)
-
+* Longer video on corssentropy method (russian) - [video](https://yadi.sk/i/5yf_4oGI3EDJhJ)
+
 
 ## Practice assignment
-Instant dive in: [__seminar_gym_interface__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/seminar_gym_interface.ipynb), [__crossentropy_method__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/crossentropy_method.ipynb)
+Instant dive in: [__seminar_gym_interface__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/seminar_gym_interface.ipynb), [__crossentropy_method__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/crossentropy_method.ipynb),
+[__deep_crossentropy_method__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/deep_crossentropy_method.ipynb)
 
 * Open `gym_interface.ipynb` and follow instructions from there
 * After you're done there, proceed to `crossentropy_method.ipynb`
 * You can find homework and bonus assignment descriptions at the end of that notebook.
 * Note: so far it's enough to say `pip install gym` on top of any data-science-stuffed python, but we'd appreciate if you gradually switch to [full installation](https://github.com/openai/gym#installing-everything).
-
-