Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docker file for dependencies #66

Merged
merged 14 commits into from
Sep 27, 2024
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/publish-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Publish image to Github Packages

on:
push:
branches:
- main

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push-image:
runs-on: ubuntu-latest

permissions:
contents: read
packages: write
attestations: write
id-token: write

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Log in to the Container registry
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: | # The first 4 lines are default; have to include in order to specify 5th (latest)
type=schedule
type=ref,event=branch
type=ref,event=tag
type=ref,event=pr
type=raw,value=latest
russelldj marked this conversation as resolved.
Show resolved Hide resolved

- name: Build and push Docker image
id: push
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

- name: Generate artifact attestation
uses: actions/attest-build-provenance@v1
with:
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true
37 changes: 37 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Use a GPU-enabled base image
FROM nvcr.io/nvidia/cudagl:11.4.1-runtime-ubuntu20.04

USER root

# Adapted from https://github.com/jeffgillan/agisoft_metashape/blob/main/Dockerfile
LABEL authors="David Russell"
LABEL maintainer="djrussell@ucdavis"

# Create user account with password-less sudo abilities
RUN useradd -s /bin/bash -g 100 -G sudo -m user
RUN /usr/bin/printf '%s\n%s\n' 'password' 'password'| passwd user
RUN echo "user ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

ENV DEBIAN_FRONTEND=noninteractive

# Install libraries/dependencies
RUN apt-get update && \
apt-get install -y libgl1-mesa-glx libglu1 \
libcurl4 \
wget && \
rm -rf /var/lib/apt/lists/*

# Install the command line python module. Note that this does not install the GUI
RUN apt-get update -y && apt-get install -y python3-pip
RUN cd /opt && wget https://download.agisoft.com/Metashape-2.1.3-cp37.cp38.cp39.cp310.cp311-abi3-linux_x86_64.whl && \
pip3 install Metashape-2.1.3-cp37.cp38.cp39.cp310.cp311-abi3-linux_x86_64.whl && pip3 install PyYAML && \
rm -rf *.whl

# Set the container workdir
WORKDIR /app
# Copy files from current directory into /app
COPY . /app

# Set the default command and default arguments
ENTRYPOINT ["python3", "/app/python/metashape_workflow.py"]
CMD ["/data/config.yml"]
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,29 @@ For internal users working on a JS2 VM created using the OFO Dev CACAO template,

to switch to a conda environment with a current Metashape python package preinstalled and configured.

### Docker
Docker is a way to run software encapsulated with all the dependencies. You must install `docker` (instuctions [here](https://docs.docker.com/engine/install/)), and if you want to use GPUs for processing, also install the `nvidia-container-toolkit` (instructions [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)).


A docker image is available from the github container registry. It can be pulled with the following command.
```
docker pull ghcr.io/open-forest-observatory/automate-metashape:latest
```

To use the docker container, you must have a directory which contains your input data and a config file, and will have output data written to it. This will be mounted inside the container in the `/data` folder. The config file should be named `config.yml` and be at the top level of this folder. When creating this config file, all paths will be relative to the file structure of the docker container. So any paths to your data folder should be replaced with `/data`. The `-v <host path>:<container path>` flag mounts a folder from the host machine into the container.
russelldj marked this conversation as resolved.
Show resolved Hide resolved

Metashape requires a license to run. Currently, this container only supports using a floating license server, which is specified as an `<IP address>:<port number>`. Internal users can find the credentials [here](https://docs.google.com/document/d/155AP0P3jkVa-yT53a-QLp7vBAfjRa78gdST1Dfb4fls/edit?usp=sharing). On your host machine, set the `AGISOFT_FLS=<IP address>:<port number>` and then set the same environment variable in the container using `-e AGISOFT_FLS=$AGISOFT_FLS`.
russelldj marked this conversation as resolved.
Show resolved Hide resolved

Finally, Metashape is accelerated by using GPUs. If your platform has GPUs and you've installed `nvidia-container-toolkit` you can make GPUs available within the container using `--gpus all`.

The following command puts it all together. This runs `automate_metashape` on the config file named `config.yml` within the mounted `/data` directory and writes the results back out to the same folder
```
docker run -v </host/data/dir>:/data -e AGISOFT_FLS=$AGISOFT_FLS --gpus all ghcr.io/open-forest-observatory/automate-metashape
```
Note that the owner of the output data will be the `root` user. To set the ownership to your user account, you can run `sudo chown <username>:<username> <file name>` or `sudo chown <username>:<username> -R <folder name>`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add above this line (or elsewhere if you find a better spot):

If running Docker on Linux without sudo (as in this example), your user will need to be in the docker group. This can be achieved with sudo usermod -a -G docker $USER and then logging out and in, as explained here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added it to the top since it feels like another docker setup step.


You can run a different config by adding `</path/to/config/file.yml>` to the end of the command above. This can be helpful if you have multiple configs you would like to try. Note that this path is local to the container, so it will begin with `/data/` if you follow the previous configuration steps.

## Usage

**Reproducible workflow scripts:** Simply clone this repository to your machine!
Expand Down
18 changes: 10 additions & 8 deletions python/metashape_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
# 2021

import sys
import argparse

# ---- If this is a first run from the standalone python module, need to copy the license file from the full metashape install: from python import metashape_license_setup

Expand All @@ -20,16 +21,17 @@
except: # running from command line (in linux) or interactively (windows)
from metashape_workflow_functions import MetashapeWorkflow

if sys.stdin.isatty():
if len(sys.argv) < 2:
print("Usage: python <path/to/metashape_workflow.py> <path/to/config_file.yml>")
sys.exit(1)
config_file = sys.argv[1]
else:
config_file = manual_config_file
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("config_file", default=manual_config_file)

args = parser.parse_args()
return args

args = parse_args()
Comment on lines +24 to +31
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is smart and much nicer. Thanks!


# Initialize the workflow instance with the configuration file
meta = MetashapeWorkflow(config_file)
meta = MetashapeWorkflow(args.config_file)

### Run the Metashape workflow
meta.run()