-
Notifications
You must be signed in to change notification settings - Fork 535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docker compose setup #162
docker compose setup #162
Changes from 17 commits
a81fb77
8922790
8c4aa93
4ab03ab
956a4b2
d06c7b9
bfb3290
2fa001e
365f337
ae13502
32f9895
dec5d43
d2a2a96
bee15ae
b11bde3
1dc3681
4643773
a6395df
71ad0c2
77c31d1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
**/videos | ||
**/artifacts | ||
**/traces | ||
**/inputs | ||
**/har | ||
**/.git | ||
**/.github | ||
**/.mypy_cache | ||
**/.venv | ||
**/.vscode | ||
*.env* | ||
# Streamlit ignores | ||
**/secrets*.toml | ||
|
||
# Skyvern | ||
docs | ||
images | ||
|
||
.dockerignore | ||
.gitignore | ||
Dockerfile |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
name: Build Docker Image and Push to ECR | ||
|
||
on: | ||
release: | ||
types: [ published ] | ||
|
||
env: | ||
AWS_REGION: us-east-1 | ||
ECR_REPOSITORY: skyvern | ||
REGISTRY_ALIAS: t6d4b5t4 # skyvern | ||
|
||
jobs: | ||
run-ci: | ||
uses: ./.github/workflows/ci.yml | ||
|
||
build-docker-image: | ||
runs-on: ubuntu-latest | ||
needs: [ run-ci ] | ||
steps: | ||
- name: Check out Git repository | ||
uses: actions/checkout@v4 | ||
|
||
- name: Configure AWS credentials | ||
uses: aws-actions/configure-aws-credentials@v4 | ||
with: | ||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
aws-region: ${{ env.AWS_REGION }} | ||
|
||
- name: Login to Amazon ECR Public | ||
id: login-ecr-public | ||
uses: aws-actions/amazon-ecr-login@v2 | ||
with: | ||
registry-type: public | ||
|
||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v1 | ||
|
||
- name: Build, tag, and push image to Amazon Public ECR | ||
id: build-image | ||
uses: docker/build-push-action@v2 | ||
env: | ||
ECR_REGISTRY: ${{ steps.login-ecr-public.outputs.registry }} | ||
with: | ||
context: . | ||
platforms: | | ||
linux/amd64 | ||
linux/arm64 | ||
push: true | ||
tags: | | ||
${{ env.ECR_REGISTRY}}/${{ env.REGISTRY_ALIAS }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }} | ||
${{ env.ECR_REGISTRY}}/${{ env.REGISTRY_ALIAS }}/${{ env.ECR_REPOSITORY }}:${{ github.event.release.tag_name }} | ||
${{ env.ECR_REGISTRY}}/${{ env.REGISTRY_ALIAS }}/${{ env.ECR_REPOSITORY }}:latest |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -164,6 +164,7 @@ artifacts/ | |
traces/ | ||
*.pkl | ||
har/ | ||
postgres-data | ||
|
||
# Streamlit ignores | ||
**/secrets*.toml | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
FROM python:3.11 as requirements-stage | ||
|
||
WORKDIR /tmp | ||
RUN pip install poetry | ||
# Use following command, if you have network problem | ||
# RUN pip install poetry -i https://pypi.tuna.tsinghua.edu.cn/simple | ||
COPY ./pyproject.toml /tmp/pyproject.toml | ||
COPY ./poetry.lock /tmp/poetry.lock | ||
RUN poetry export -f requirements.txt --output requirements.txt --without-hashes | ||
|
||
FROM python:3.11-slim-bookworm | ||
WORKDIR /app | ||
COPY --from=requirements-stage /tmp/requirements.txt /app/requirements.txt | ||
RUN pip install --no-cache-dir --upgrade -r requirements.txt | ||
RUN pip install --no-cache-dir streamlit | ||
# Use following command, if you have network problem | ||
# RUN sed -i s@/deb.debian.org/@/mirrors.aliyun.com/@g /etc/apt/sources.list.d/debian.sources | ||
# RUN pip install --no-cache-dir --upgrade -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple | ||
# RUN pip install --no-cache-dir streamlit -i https://pypi.tuna.tsinghua.edu.cn/simple | ||
RUN playwright install-deps | ||
RUN playwright install | ||
RUN apt-get install -y xauth && apt-get clean | ||
|
||
COPY . /app | ||
|
||
ENV PYTHONPATH="/app:$PYTHONPATH" | ||
ENV VIDEO_PATH=/data/videos | ||
ENV HAR_PATH=/data/har | ||
ENV ARTIFACT_STORAGE_PATH=/data/artifacts | ||
|
||
COPY ./entrypoint-skyvern.sh /app/entrypoint-skyvern.sh | ||
RUN chmod +x /app/entrypoint-skyvern.sh | ||
|
||
COPY ./entrypoint-streamlit.sh /app/entrypoint-streamlit.sh | ||
RUN chmod +x /app/entrypoint-streamlit.sh | ||
|
||
CMD [ "/bin/bash", "/app/entrypoint-skyvern.sh" ] | ||
|
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -101,6 +101,16 @@ Note: Our setup script does these two for you, but they are here for reference. | |
``` | ||
1. Navigate to `http://localhost:8501` in your browser to start using the UI | ||
|
||
## Docker Compose setup | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should test the docker, (maybe even ask people in DC to test it out and provide feedback) and when it's all good this should be the primary way to start using Skyvern. |
||
|
||
1. Fill in the LLM provider key on the [docker-compose.yml](./docker-compose.yml) | ||
2. Run the following command: | ||
```bash | ||
docker compsoe up -d | ||
LawyZheng marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
3. Navigate to `http://localhost:8501` in your browser to start using the UI | ||
|
||
|
||
## Additional Setup for Contributors | ||
If you're looking to contribute to Skyvern, you'll need to install the pre-commit hooks to ensure code quality and consistency. You can do this by running the following command: | ||
```bash | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,57 @@ | ||||||
version: '3.8' | ||||||
|
||||||
services: | ||||||
postgres: | ||||||
image: postgres:14-alpine | ||||||
# comment out if you want to externally connect DB | ||||||
# ports: | ||||||
# - 5432:5432 | ||||||
volumes: | ||||||
- ./postgres-data:/var/lib/postgresql/data | ||||||
environment: | ||||||
- PGDATA=/var/lib/postgresql/data/pgdata | ||||||
- POSTGRES_USER=skyvern | ||||||
- POSTGRES_PASSWORD=skyvern | ||||||
- POSTGRES_POSTGRES_DB=skyvern | ||||||
|
||||||
skyvern: | ||||||
image: public.ecr.aws/t6d4b5t4/skyvern:latest | ||||||
restart: on-failure | ||||||
# comment out if you want to externally call skyvern API | ||||||
# ports: | ||||||
# - 8000:8000 | ||||||
volumes: | ||||||
- ./artifacts:/data/artifacts | ||||||
- ./videos:/data/videos | ||||||
- ./har:/data/har | ||||||
- ./.streamlit:/app/.streamlit | ||||||
environment: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we point it to the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Besides, if we change the settings in env, we are supposed to restart/recreate the container. |
||||||
- DATABASE_STRING=postgresql+psycopg://skyvern:skyvern@postgres:5432/skyvern | ||||||
- BROWSER_TYPE=chromium-headful | ||||||
- ENABLE_OPENAI=true | ||||||
- OPENAI_API_KEY=<your_openai_key> | ||||||
# If you want to use other LLM provide, like azure or anthropic | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
# - ENABLE_AZURE=true | ||||||
# - LLM_KEY=AZURE_OPENAI_GPT4V | ||||||
# - AZURE_DEPLOYMENT=<your_azure_deployment> | ||||||
# - AZURE_API_KEY=<your_azure_api_key> | ||||||
# - AZURE_API_BASE=<your_azure_api_base> | ||||||
# - AZURE_API_VERSION=<your_azure_api_version> | ||||||
depends_on: | ||||||
- postgres | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This wouldn't work well because this way it just waits for the container to be up. Instead, we should be waiting for postgres to be ready. This eventually works out but until skyvern is up, the container will restart multiple times since it'll die without a healthy DB. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great, I have noticed it already. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So I just let skyvern restart on failure. haha There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah it works but if someone checks out the logs or runs without Can you check this out and see if it works? I did something similar before, if this doesn't work we can hop on a call later? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It worked. |
||||||
|
||||||
streamlit: | ||||||
image: public.ecr.aws/t6d4b5t4/skyvern:latest | ||||||
restart: on-failure | ||||||
ports: | ||||||
- 8501:8501 | ||||||
volumes: | ||||||
- ./artifacts:/data/artifacts | ||||||
- ./videos:/data/videos | ||||||
- ./har:/data/har | ||||||
- ./.streamlit:/app/.streamlit | ||||||
command: ["/bin/bash", "entrypoint-streamlit.sh"] | ||||||
depends_on: | ||||||
- postgres | ||||||
- skyvern | ||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
|
||
#!/bin/bash | ||
|
||
set -e | ||
|
||
# check alembic | ||
alembic upgrade head | ||
alembic check | ||
|
||
if [ ! -f ".streamlit/secrets.toml" ]; then | ||
echo "Creating organization and API token..." | ||
org_output=$(python scripts/create_organization.py Skyvern-Open-Source) | ||
api_token=$(echo "$org_output" | awk '/token=/{gsub(/.*token='\''|'\''.*/, ""); print}') | ||
# Update the secrets-open-source.toml file | ||
echo -e "[skyvern]\nconfigs = [\n {\"env\" = \"local\", \"host\" = \"http://skyvern:8000/api/v1\", \"orgs\" = [{name=\"Skyvern\", cred=\"$api_token\"}]}\n]" > .streamlit/secrets.toml | ||
echo ".streamlit/secrets.toml file updated with organization details." | ||
fi | ||
|
||
# Run the command and pass in all three arguments | ||
xvfb-run python -m skyvern.forge |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
|
||
#!/bin/bash | ||
|
||
set -e | ||
|
||
# Run the command and pass in all three arguments | ||
streamlit run streamlit_app/visualizer/streamlit.py --server.port 8501 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what are these comments? can we delete these comments?