This project allows you to use DensePose from detectron2 together with Pupil Invisible / Neon's recordings. It generates a new visualization with denseposes overlaid on the video and gaze on top, and also generates a new CSV file with the body parts gazed and body heatmap.
You should have, Linux or MacOS and Python 3.9 or higher installed on your system. This file assumes a minimum technical knowledge of the command line and Python, if you are not familiar we recommend that you use our Google Colab notebook instead.
Follow these steps to install and use the pupil-labs-dense-pose module:
On MacOS (using the CPU)
- Open a terminal window.
- Run the following commands. For easiness, we will use the astral/uv package manager. You can install it in your system by:
curl -LsSf https://astral.sh/uv/install.sh | shIn MacOS we can only use the CPU version of detectron2, when installing it from Meta's repository there are some issues that you may find. So we will use @johnnynunez's version of detectron2 that works with the latest pytorch version.
# Optional, but recommended, run it on a virtual environment
uv venv --seed --python python3.11 --exclude-newer="2024-03-14"
source .venv/bin/activate
uv pip install -U pip setuptools wheel
# Install torch and torchvision
uv pip install torch==2.0.1 torchvision==0.15.2
# Now, we install detectron2, Meta hasn't updated it to run with the latest Pytorch version, but thanks to @johnnynunez
# we have a version that works with the latest. Grab the wheels for your version at https://github.com/johnnynunez/detectron2/actions/workflows/build-wheels.yml
# select the latest run for your system and matching Pytorch.
# Install them with pip, you will need to point to the wheel you downloaded, e.g.:
uv pip install detectron2-0.7-cp311-cp311-macosx_10_9_universal2.whl
# This will also avoid issues with poetry from Python, giving you errors with the torch module not being found even though it is installed.
# Now, we install densepose
export FORCE_CUDA="0" # as we don't have CUDA
uv pip install git+https://github.com/johnnynunez/detectron2@main#subdirectory=projects/DensePose --no-build-isolation --exclude-newer="2024-03-14"
# Now we install the module
uv pip install git+https://github.com/pupil-labs/densepose-module --exclude-newer="2024-03-14"
# And that's it!On Linux, we can run inference on either the CPU or the GPU (if we have CUDA installed). If you want to run it on the CPU, follow these steps:
# Optional, but recommended, run it on a virtual environment
uv venv --seed --python python3.11 --exclude-newer="2024-03-14"
source .venv/bin/activate
uv pip install -U pip setuptools wheel
# Install torch and torchvision
uv pip install torch==2.0.1 torchvision==0.15.2
# Now, we install detectron2, Meta hasn't update it to run with the latest pytorch version, but thanks to @johnnynunez
# we have a version that works with the latest. Grab the wheels for your version at https://github.com/johnnynunez/detectron2/actions/workflows/build-wheels.yml
# and install them with pip, you will need to point to the wheel you downloaded, e.g.:
uv pip install detectron2-3.11-pytorch2.0.1-ubuntu-latest-wheel.whl
export FORCE_CUDA="0" # as we don't have CUDA
uv pip install git+https://github.com/johnnynunez/detectron2@main#subdirectory=projects/DensePose --no-build-isolation --exclude-newer="2024-03-14"
# Now we install the module
uv pip install git+https://github.com/pupil-labs/densepose-module.git --exclude-newer="2024-03-14"
# And that's it!# Optional, but recommended, run it on a virtual environment
uv venv --seed --python python3.11 --exclude-newer="2024-03-14"
source .venv/bin/activate
uv pip install -U pip setuptools wheel
# Install torch and torchvision
uv pip install torch+cu torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
# Now, we install detectron2, Meta hasn't update to run with the latest pytorch version, but thanks to @johnnynunez
# we have a version that works with the latest. There are wheels for cuda 11.8 and pytorch 2.01 at
#(https://app.circleci.com/pipelines/github/facebookresearch/detectron2/2924/workflows/9f85ee27-173e-494c-b699-8ceb110a3398/jobs/14336/artifacts)
# if you use a different version you will need to build it yourself.
uv pip install detectron2-0.7-cp311-cp311-linux_x86_64.whl
#or to try building your own wheels:
uv pip install git+https://github.com/johnnynunez/detectron2.git --exclude-newer="2024-03-14"
export FORCE_CUDA="1" # as we want to use CUDA
# We might also need to specify the CUDA home directory
# like export CUDA_HOME="/usr/local/cuda-11.8"
uv pip install git+https://github.com/johnnynunez/detectron2@main#subdirectory=projects/DensePose --no-build-isolation --exclude-newer="2024-03-14"
# Now we install the module
uv pip install git+https://github.com/pupil-labs/densepose-module --exclude-newer="2024-03-14"
# And that's it!To run the pupil-labs-dense-pose module, execute the following command:
pl-denseposepl-densepose -hYou can also provide additional options while running the command. For example, to specify the input and output paths, use the --input_path and --output_path options. Additionally, you can use the --device option to specify the device to be used (e.g., cpu or cuda for GPU). Or the size of the gaze circle used to determine the gazed parts.
For a detailed description of available arguments and information about where inference is happening, refer to the docs or our alpha lab article
Feel free to reach out if you have any questions or need further assistance.