Docker and Singularity.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!mkdir -p ~/agave/funwave-tvd-docker\n",
    "\n",
    "%cd ~/agave\n",
    "\n",
    "!pip3 install setvar\n",
    "\n",
    "import re\n",
    "import os\n",
    "import sys\n",
    "from setvar import *\n",
    "from time import sleep\n",
    "\n",
    "# This cell enables inline plotting in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "loadvar()\n",
    "!auth-tokens-refresh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Creating the Docker Image</h2>\n",
    "To start, we need a Dockerfile, which has a number of simple commands.\n",
    "It starts with \"FROM\" which can specify any docker image available from DockerHub. That not only includes basic operating systems such as \"ubunto\", \"fedora\", \"centos\", etc. but specialized containers made by anyone with a dockerhub account. I've provided \"science-base\" which has OpenMPI 2.1.1 and some standard compilers, i.e. gfortran, gcc, and g++.\n",
    "\n",
    "MAINTAINER is a bit of metadata that (hopefully) will allow you to contact the container's creator, if need be.\n",
    "\n",
    "WORKDIR is the dockerfile equivalent of the \"cd\" command. Note that running \"cd\" will not change your directory.\n",
    "\n",
    "RUN simply runs the command that follows. Because the container is saved after each step, we want to avoid creating files that we don't want to keep (we want containers to be as small as possible).\n",
    "\n",
    "USER specifies the user id for running subsequent RUN commands.\n",
    "\n",
    "COPY can be used to copy files into the container from the build directory.\n",
    "\n",
    "ENTRYPOINT is a script that runs when the container starts up. What our script does is create a new user on the docker image with a user id and name that is convenient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "writefile(\"funwave-tvd-docker/Dockerfile\",\"\"\"\n",
    "FROM stevenrbrandt/science-base\n",
    "\n",
    "MAINTAINER Steven R. Brandt <sbrandt@cct.lsu.edu>\n",
    "RUN cd && \\\n",
    "    git clone https://github.com/fengyanshi/FUNWAVE-TVD && \\\n",
    "    cd FUNWAVE-TVD/src && \\\n",
    "    perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile && \\\n",
    "    make\n",
    "\n",
    "WORKDIR /home/jovyan/FUNWAVE-TVD/src\n",
    "RUN mkdir -p /home/jovyan/FUNWAVE-TVD/rundir\n",
    "WORKDIR /home/jovyan/FUNWAVE-TVD/rundir\n",
    "USER root\n",
    "COPY entrypoint.sh /usr/local/bin\n",
    "RUN chmod 755 /usr/local/bin/entrypoint.sh\n",
    "ENTRYPOINT [\"/usr/local/bin/entrypoint.sh\"]\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is our entrypoint file. It can be any bash script, and it is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "writefile(\"funwave-tvd-docker/entrypoint.sh\",\"\"\"\n",
    "#!/bin/bash\n",
    "echo ARGS: \\$@\n",
    "/home/jovyan/FUNWAVE-TVD/src/funwave_vessel \"\\$@\"\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've create our Dockerfile and entrypoint.sh, bundle them up in a tarball and send them somewhere that agave can access them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!tar -czf dockerjob.tgz -C funwave-tvd-docker Dockerfile entrypoint.sh\n",
    "!files-mkdir -S ${STORAGE_MACHINE} -N funwave-tvd-docker\n",
    "!files-upload -F dockerjob.tgz -S ${STORAGE_MACHINE} funwave-tvd-docker/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import runagavecmd as r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the docker build command. We will \"tag\" this build with the name \"funwave-tvd\" when it is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r.runagavecmd(\n",
    "    \"tar xzf dockerjob.tgz && sudo docker build --rm -t funwave-tvd .\",\n",
    "    \"agave://${STORAGE_MACHINE}/funwave-tvd-docker/dockerjob.tgz\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!jobs-output-get ${JOB_ID} fork-command-1.err\n",
    "!cat fork-command-1.err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Running the Docker Image</h2>\n",
    "It is possible to run docker interactively, but that isn't convenient inside scripts. So instead, we start it in detached mode, with the -d flag.\n",
    "\n",
    "Because your docker image has its own internal file system, it can't see files on the host machine. You can, however, transfer them using the \"docker cp\" command.\n",
    "\n",
    "Running docker is slightly tricky. When a Docker image starts up, you can execute any command you want--but when you type \"exit\" all the changes you've made to the file system vanish. Therefore it's necessary to copy them out before the docker container stops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "writefile(\"rundock.sh\",\"\"\"\n",
    "rm -fr cid.txt out.tgz\n",
    "\n",
    "# Start a docker image running in detached mode, write the container id to cid.txt\n",
    "sudo docker run -d -it --rm --cidfile cid.txt -e USER_ID=$(id -u) -e USER_NAME=$USER stevenrbrandt/funwave-tvd bash\n",
    "\n",
    "# Store the container id in CID for convenience\n",
    "CID=\\$(cat cid.txt)\n",
    "\n",
    "# Copy the input.txt file into the running image\n",
    "sudo docker cp input.txt \\$CID:/home/jovyan/FUNWAVE-TVD/rundir/\n",
    "\n",
    "# Run funwave on the image\n",
    "sudo docker exec --user jovyan \\$CID mpirun -np 4 /home/jovyan/FUNWAVE-TVD/src/funwave_vessel\n",
    "\n",
    "# Extract the output files from the running image\n",
    "# Having them in a tgz makes it more convenient to fetch them with jobs-output-get\n",
    "sudo docker exec --user jovyan \\$CID tar czf -  output > out.tgz\n",
    "\n",
    "# Stop the image\n",
    "sudo docker stop \\$CID\n",
    "\n",
    "# List the output files\n",
    "tar tzf out.tgz\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the input.txt file and the rundock.sh script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!tar czf rundock.tgz rundock.sh input.txt\n",
    "!files-upload -F rundock.tgz -S ${STORAGE_MACHINE} funwave-tvd-docker/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the rundock.sh script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r.runagavecmd(\n",
    "    \"tar xzf rundock.tgz && bash rundock.sh\",\n",
    "    \"agave://${STORAGE_MACHINE}/funwave-tvd-docker/rundock.tgz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the output of the job back to our local machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!jobs-output-list ${JOB_ID}\n",
    "!jobs-output-get ${JOB_ID} out.tgz\n",
    "!tar xzvf out.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!head output/eta_00010"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Running with Singularity</h2>\n",
    "If we have a public docker image, we can run it directly with Singularity. Singularity is desiged to be more HPC friendly than Docker. First, because it doesn't all the running user to access any user id but their own inside the container, and second, because singularity images can be run through MPI, making it easier to scale up to a distributed cluser.\n",
    "\n",
    "In this first step, we build the singularity installation. Because the result of this job is intended to be an installation for subsequent jobs, we install it to a hard-coded directory rather than using the normal Agave job directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!files-mkdir -S ${STORAGE_MACHINE} -N sing\n",
    "!files-upload -F input.txt -S ${STORAGE_MACHINE} sing/\n",
    "r.runagavecmd(\n",
    "            \"mkdir -p ~/singu && \"+\n",
    "            \"cd ~/singu && \"+\n",
    "            \"rm -f funwave-tvd.img && \"+\n",
    "            \"singularity create funwave-tvd.img --size 2000 && \"+\n",
    "            \"singularity import funwave-tvd.img docker://stevenrbrandt/funwave-tvd:latest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the Singularity image is built, we can run it with mpi. Notice that mpi executes the singularity command. The tricky part here is to make sure you've got the same version of mpi running inside and outside the container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!files-upload -F input.txt -S ${STORAGE_MACHINE} ./\n",
    "r.runagavecmd(\n",
    "    \"export LD_LIBRARY_PATH=/usr/local/lib && \"+\n",
    "    \"mpirun -np 4 singularity exec ~/singu/funwave-tvd.img /home/jovyan/FUNWAVE-TVD/src/funwave_vessel && \"+\n",
    "    \"tar cvzf singout.tgz output\",\n",
    "    \"agave://${STORAGE_MACHINE}/input.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!jobs-output-get ${JOB_ID} singout.tgz\n",
    "!rm -fr output\n",
    "!tar xzf singout.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!head output/v_00010"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}