Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore #594

Closed
wants to merge 7 commits into from
Closed

Chore #594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
02ef82c
Base notebook/init (#1)
eroan-marie Nov 28, 2024
3b990df
Base notebook/init (#3)
eroan-marie Nov 28, 2024
b74fee2
Pangeo notebook/init (#5)
eroan-marie Dec 3, 2024
33dfad1
Pytorch notebook/init (#6)
eroan-marie Dec 4, 2024
9d3be28
Documentation (#8)
eroan-marie Dec 12, 2024
c92eb66
fix: novnc and CI (#7)
eroan-marie Dec 16, 2024
43356f4
chore: headers
eroan-marie Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 4 additions & 22 deletions .github/workflows/Build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
- 'update_lockfile.sh'

env:
DOCKER_ORG: pangeo
DOCKER_ORG: cnes
GITHUB_SHA: ${{ github.sha }}
GITHUB_REF: ${{ github.ref }}

Expand Down Expand Up @@ -44,31 +44,22 @@ jobs:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USER }}
password: ${{ secrets.QUAY_TOKEN }}

- name: Build and push
uses: docker/build-push-action@v6
with:
context: ${{ env.IMAGE }}
platforms: linux/amd64,linux/arm64
platforms: linux/amd64
push: true
tags: |
${{env.DOCKER_ORG}}/${{ env.IMAGE }}:master
${{env.DOCKER_ORG}}/${{ env.IMAGE }}:${{env.DOCKER_TAG}}
quay.io/${{env.DOCKER_ORG}}/${{ env.IMAGE }}:master
quay.io/${{env.DOCKER_ORG}}/${{ env.IMAGE }}:${{env.DOCKER_TAG}}

matrix-build:
needs: base-image
strategy:
fail-fast: false
matrix:
IMAGE: [base-notebook, pangeo-notebook, ml-notebook, pytorch-notebook]
IMAGE: [base-notebook, pangeo-notebook, pytorch-notebook]
name: ${{ matrix.IMAGE }}
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -102,24 +93,15 @@ jobs:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USER }}
password: ${{ secrets.QUAY_TOKEN }}

- name: Build and push
uses: docker/build-push-action@v6
with:
context: ${{ matrix.IMAGE }}
push: true
platforms: linux/amd64,linux/arm64
platforms: linux/amd64
tags: |
${{env.DOCKER_ORG}}/${{ matrix.IMAGE }}:master
${{env.DOCKER_ORG}}/${{ matrix.IMAGE }}:${{env.DOCKER_TAG}}
quay.io/${{env.DOCKER_ORG}}/${{ matrix.IMAGE }}:master
quay.io/${{env.DOCKER_ORG}}/${{ matrix.IMAGE }}:${{env.DOCKER_TAG}}

- name: Inspect Image
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ChatOpsDispatcher.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
- name: Slash Command Dispatch
uses: peter-evans/slash-command-dispatch@v4
with:
token: ${{ secrets.PANGEOBOT_TOKEN }}
token: ${{ secrets.DATALABSBOT_TOKEN }}
reaction-token: ${{ secrets.GITHUB_TOKEN }}
config: >
[
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/CondaLock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
fail-fast: false
matrix:
IMAGE: [base-notebook, pangeo-notebook, ml-notebook, pytorch-notebook]
IMAGE: [base-notebook, pangeo-notebook, pytorch-notebook]
name: ${{ matrix.IMAGE }}
runs-on: ubuntu-latest
timeout-minutes: 180
Expand All @@ -23,7 +23,7 @@ jobs:
- name: Checkout Repository
uses: actions/checkout@v4
with:
token: ${{ secrets.PANGEOBOT_TOKEN }}
token: ${{ secrets.DATALABSBOT_TOKEN }}
# These lines are critical, otherwise Pangeo-bot pushes changes directly to master from PRs!
repository: ${{ github.event.client_payload.pull_request.head.repo.full_name }}
ref: ${{ github.event.client_payload.pull_request.head.ref }}
Expand All @@ -39,9 +39,9 @@ jobs:
run: |
cd ${{ matrix.IMAGE }}
if [ ${{ matrix.IMAGE }} = "base-notebook" ]; then
conda-lock lock -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64
conda-lock lock -p linux-64
elif [ ${{ matrix.IMAGE }} = "pangeo-notebook" ]; then
conda-lock lock -f environment.yml -f ../base-notebook/environment.yml -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64
conda-lock lock -f environment.yml -f ../base-notebook/environment.yml -p linux-64
else
# Linux-64 ONLY
conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64
Expand All @@ -64,7 +64,7 @@ jobs:
- name: Checkout Repository
uses: actions/checkout@v4
with:
token: ${{ secrets.PANGEOBOT_TOKEN }}
token: ${{ secrets.DATALABSBOT_TOKEN }}
# These lines are critical, otherwise Pangeo-bot pushes changes directly to master from PRs!
repository: ${{ github.event.client_payload.pull_request.head.repo.full_name }}
ref: ${{ github.event.client_payload.pull_request.head.ref }}
Expand All @@ -74,8 +74,8 @@ jobs:

- name: Commit condalock files to PR
run: |
git config --global user.name 'pangeo-bot'
git config --global user.email 'pangeo-bot@users.noreply.github.com'
git config --global user.name 'datalabs-bot'
git config --global user.email 'datalabs-bot@users.noreply.github.com'
git status
git add */conda*.lock */conda-lock.yml */packages.txt
git commit -m "[condalock-command] autogenerated conda-lock files"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/PR.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
jobs:
DeployPangeoBot:
runs-on: ubuntu-latest
if: github.repository == 'pangeo-data/pangeo-docker-images'
if: github.repository == 'CNES/datalabs-docker-images'

steps:
- name: Checkout Repository
Expand All @@ -17,7 +17,7 @@ jobs:
- name: Add Condalock Comment
uses: peter-evans/create-or-update-comment@v4
with:
token: ${{ secrets.PANGEOBOT_TOKEN }}
token: ${{ secrets.DATALABSBOT_TOKEN }}
issue-number: ${{github.event.number}}
body: |
/condalock
Expand Down
26 changes: 13 additions & 13 deletions .github/workflows/Publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ on:
- '*'

env:
DOCKER_ORG: pangeo
DOCKER_ORG: cnes
GITHUB_REF: ${{ github.ref }}

jobs:
matrix-build:
strategy:
fail-fast: false
matrix:
IMAGE: [base-image, base-notebook, pangeo-notebook, ml-notebook, pytorch-notebook]
IMAGE: [base-image, base-notebook, pangeo-notebook, pytorch-notebook]
name: ${{ matrix.IMAGE }}
runs-on: ubuntu-latest
steps:
Expand All @@ -33,21 +33,21 @@ jobs:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USER }}
password: ${{ secrets.QUAY_TOKEN }}
# - name: Login to Quay.io
# uses: docker/login-action@v3
# with:
# registry: quay.io
# username: ${{ secrets.QUAY_USER }}
# password: ${{ secrets.QUAY_TOKEN }}

- name: Create New DockerHub Tags
run: |
docker buildx imagetools create ${DOCKER_ORG}/${{ matrix.IMAGE }}:master \
-t ${DOCKER_ORG}/${{ matrix.IMAGE }}:latest \
-t ${DOCKER_ORG}/${{ matrix.IMAGE }}:${TAG}

- name: Create New Quay.io Tags
run: |
docker buildx imagetools create quay.io/${DOCKER_ORG}/${{ matrix.IMAGE }}:master \
-t quay.io/${DOCKER_ORG}/${{ matrix.IMAGE }}:latest \
-t quay.io/${DOCKER_ORG}/${{ matrix.IMAGE }}:${TAG}
# - name: Create New Quay.io Tags
# run: |
# docker buildx imagetools create quay.io/${DOCKER_ORG}/${{ matrix.IMAGE }}:master \
# -t quay.io/${DOCKER_ORG}/${{ matrix.IMAGE }}:latest \
# -t quay.io/${DOCKER_ORG}/${{ matrix.IMAGE }}:${TAG}
12 changes: 6 additions & 6 deletions .github/workflows/Test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
types: [opened, synchronize, reopened]

env:
DOCKER_ORG: pangeo
DOCKER_ORG: cnes
GITHUB_SHA: ${{ github.sha }}
GITHUB_REF: ${{ github.ref }}

Expand All @@ -19,7 +19,7 @@ jobs:
strategy:
fail-fast: false
matrix:
IMAGE: [base-notebook, pangeo-notebook, ml-notebook, pytorch-notebook]
IMAGE: [base-notebook, pangeo-notebook, pytorch-notebook]
name: ${{ matrix.IMAGE }}
runs-on: ubuntu-latest
services:
Expand Down Expand Up @@ -54,8 +54,8 @@ jobs:
uses: docker/build-push-action@v6
with:
context: base-image
platforms: linux/amd64,linux/arm64
tags: localhost:5000/pangeo/base-image:PR
platforms: linux/amd64
tags: localhost:5000/cnes/base-image:PR
push: true
cache-from: type=gha
cache-to: type=gha,mode=max
Expand All @@ -64,12 +64,12 @@ jobs:
uses: docker/build-push-action@v6
with:
context: ${{ matrix.IMAGE }}
platforms: linux/amd64,linux/arm64
platforms: linux/amd64
push: true
tags: localhost:5000/${{env.DOCKER_ORG}}/${{ matrix.IMAGE }}:PR
build-args: PANGEO_BASE_IMAGE_TAG=PR
build-contexts: |
pangeo/base-image:PR=docker-image://localhost:5000/pangeo/base-image:PR
cnes/base-image:PR=docker-image://localhost:5000/cnes/base-image:PR

- name: List Conda Packages
run: |
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/WatchCondaForge.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
jobs:
check-version:
runs-on: ubuntu-latest
if: github.repository == 'pangeo-data/pangeo-docker-images'
if: github.repository == 'CNES/datalabs-docker-images'

steps:
- name: Checkout Repository
Expand Down Expand Up @@ -57,11 +57,11 @@ jobs:
id: cpr
uses: peter-evans/create-pull-request@v7
with:
committer: Pangeo Bot <pangeo-bot@users.noreply.github.com>
token: ${{ secrets.PANGEOBOT_TOKEN }}
committer: Datalabs Bot <datalabs-bot@users.noreply.github.com>
token: ${{ secrets.DATALABSBOT_TOKEN }}
commit-message: "Update pangeo-notebook metapackage version to ${{ steps.latest_version.outputs.version }}"
title: "Update pangeo-notebook metapackage version to ${{ steps.latest_version.outputs.version }}"
reviewers: "scottyhq"
reviewers: "eroan.marie"
branch: ${{ env.PR_BRANCH }}
body: |
A new pangeo-metapackage version has been detected.
Expand Down
0 ml-notebook/Dockerfile → .ml-notebook/Dockerfile
View file
Open in desktop
File renamed without changes.
0 ml-notebook/apt.txt → .ml-notebook/apt.txt
View file
Open in desktop
File renamed without changes.
0 ml-notebook/conda-linux-64.lock → .ml-notebook/conda-linux-64.lock
View file
Open in desktop
File renamed without changes.
0 ml-notebook/conda-lock.yml → .ml-notebook/conda-lock.yml
View file
Open in desktop
File renamed without changes.
0 ml-notebook/environment.yml → .ml-notebook/environment.yml
View file
Open in desktop
File renamed without changes.
0 ml-notebook/jupyter_notebook_config.py → .ml-notebook/jupyter_notebook_config.py
View file
Open in desktop
File renamed without changes.
0 ml-notebook/packages.txt → .ml-notebook/packages.txt
View file
Open in desktop
File renamed without changes.
0 ml-notebook/start → .ml-notebook/start
View file
Open in desktop
File renamed without changes.
0 ml-notebook/virtual-packages.yml → .ml-notebook/virtual-packages.yml
View file
Open in desktop
File renamed without changes.
33 changes: 14 additions & 19 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,40 @@
# Makefile for convenience, (doesn't look for command outputs)
.PHONY: all
all: base-image base-notebook pangeo-notebook ml-notebook pytorch-notebook
all: base-image base-notebook pangeo-notebook pytorch-notebook
TESTDIR=/srv/test

.PHONY: base-image
base-image :
cd base-image ; \
docker build -t pangeo/base-image:master --progress=plain --platform linux/amd64,linux/arm64 .
docker build -t cnes/base-image:master --progress=plain --platform linux/amd64 .

.PHONY: base-notebook
base-notebook : base-image
cd base-notebook ; \
conda-lock lock -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64; \
conda-lock lock -f environment.yml -p linux-64; \
conda-lock render -k explicit -p linux-64; \
../generate-packages-list.py conda-linux-64.lock > packages.txt; \
docker build -t pangeo/base-notebook:master . --progress=plain --platform linux/amd64,linux/arm64 ; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) pangeo/base-notebook:master ./run_tests.sh base-notebook
docker build -t cnes/base-notebook:master . --no-cache --progress=plain --platform linux/amd64; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) cnes/base-notebook:master ./run_tests.sh base-notebook

.PHONY: pangeo-notebook
pangeo-notebook : base-image
cd pangeo-notebook ; \
conda-lock lock -f environment.yml -f ../base-notebook/environment.yml -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64; \
cp -r ../base-notebook/resources . ; \
conda-lock lock -f environment.yml -f ../base-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64; \
conda-lock render -k explicit -p linux-64; \
../generate-packages-list.py conda-linux-64.lock > packages.txt; \
docker build -t pangeo/pangeo-notebook:master . --progress=plain --platform linux/amd64,linux/arm64 ; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) pangeo/pangeo-notebook:master ./run_tests.sh pangeo-notebook

.PHONY: ml-notebook
ml-notebook : base-image
cd ml-notebook ; \
conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64; \
conda-lock render -k explicit -p linux-64; \
../generate-packages-list.py conda-linux-64.lock > packages.txt; \
docker build -t pangeo/ml-notebook:master . ; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) pangeo/ml-notebook:master ./run_tests.sh ml-notebook
../merge-apt.sh ../base-notebook/apt.txt apt.txt; \
docker build -t cnes/pangeo-notebook:master . --progress=plain --platform linux/amd64; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) cnes/pangeo-notebook:master ./run_tests.sh pangeo-notebook

.PHONY: pytorch-notebook
pytorch-notebook : base-image
cd pytorch-notebook ; \
cp -r ../pangeo-notebook/resources ../base-notebook/resources . ; \
conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64; \
conda-lock render -k explicit -p linux-64; \
../generate-packages-list.py conda-linux-64.lock > packages.txt; \
docker build -t pangeo/pytorch-notebook:master . ; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) pangeo/pytorch-notebook:master ./run_tests.sh pytorch-notebook
../merge-apt.sh ../pangeo-notebook/apt.txt ../base-notebook/apt.txt apt.txt; \
docker build -t cnes/pytorch-notebook:master . ; \
docker run -w $(TESTDIR) -v $(PWD):$(TESTDIR) cnes/pytorch-notebook:master ./run_tests.sh pytorch-notebook
Loading
Loading