Hawk from Scratch

(Work in progress) Installing Hawk from Scratch

This guide will show you how to set up a basic system for testing and exploring Hawk, a live learning framework for discovering rare events. We will assume your hardware setup includes an NVIDIA GPU, although it may be possible to run CPU-only or with AMD mRoc. The current test rig we use at CMU is based around Intel Xeon CPUs with 12 cores @ 3.6 GHz, 32GB RAM, and a GTX1060 or better GPU.

System setup

We will be installing Ubuntu Server 20.04 (desktop might work but we have not tested it), the NVIDIA driver for your GPU, and various utilities needed to build and run Hawk. If you are running this on a laptop, you might want to use the regular nvidia-driver-535 instead.

# On a freshly installed Ubuntu Server 20.04 system
sudo apt install nvidia-driver-535-server nvidia-cuda-toolkit

If the installation seems to have worked, reboot your system and then run nvidia-smi to make sure we can see and communicate with the loaded kernel driver and that it has identified your GPU(s) correctly.

Now we can install some minimal dependencies we need to download the Hawk source and get a development environment configured.

sudo apt install git python3-pip python3-venv
pip install --user pipx
~/.local/bin/pipx ensurepath
. ~/.bashrc
pipx install poetry

Installing Hawk

Once this is done you should be able to obtain Hawk by cloning our GitHub repository:

git clone https://github.com/cmusatyalab/hawk.git
cd hawk/
poetry install -E scout -E home

These commands install all package dependencies required for both home and scouts on the current system to run a Hawk mission.

Running the Hawk scout

At this point, Hawk has been installed and you can open a new terminal window and start a scout. Once started it will wait for a hawk_home instance to send it a new mission configuration.

cd ~/hawk/
poetry run hawk_scout

If these commands are successful, you should see a log message saying the scout is running. It is currently waiting for a mission configuration to be sent from home.

Installing or deploying Hawk scout as a Python wheel

You still need Poetry on your development host to build the wheel and requirements files, but you will not need Poetry in the environment where Hawk is deployed.

Run the following command to create the wheel and requirements files, (copy them to the deployment), and install them in a virtualenv with pip. The requirements file is not strictly needed, but it makes sure that the virtualenv is provisioned with the exact same dependencies as the development environment is using, as specified by the Poetry lock file.

cd ~/hawk/
poetry run hawk_deploy build
# copy dist/* to deployment
python3 -m venv venv
venv/bin/activate
pip install -r dist/requirements-scout.txt
pip install dist/cmuhawk-0.1.0-py3-none-any.whl[scout]

hawk_deploy has some additional commands to help automate the deployment, (re)starting, and stopping of scouts for a mission. This relies on having ssh public keys on the scout servers as well as a mission config file with a 'deploy' section to indicate which machines will run scouts and the user/port information to deploy the Hawk scout components. If a mission config has a deploy: configuration section it doesn't need to include the top-level list of scouts since that information will be extracted from the deploy config.

# example deploy config
deploy:
  scouts:
    - user@host1.fqdn
    - user@host2.fqdn
  scout_port: 6100  # possibly the non-default port to run the hawk_scout process