Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use artifacts for docker images #108

Merged
merged 49 commits into from
Jun 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
63c5d61
Only trigger for correct repo
EmilyBourne Dec 12, 2022
516e02b
Try artifacts
EmilyBourne Dec 12, 2022
77e29da
Try artifacts
EmilyBourne Dec 12, 2022
1bbe813
Trigger
EmilyBourne Dec 12, 2022
5450cc2
Correct yml error
EmilyBourne Dec 12, 2022
832e8d6
Correct ls. Correct run command
EmilyBourne Dec 12, 2022
ba4a44f
Remove debug info
EmilyBourne Dec 12, 2022
6bc476c
Look for artifact
EmilyBourne Dec 12, 2022
290663e
Revert change
EmilyBourne Dec 12, 2022
5a13646
Try better path
EmilyBourne Dec 12, 2022
a39276d
Ignore tag
EmilyBourne Dec 12, 2022
a8756e9
Try
EmilyBourne Dec 12, 2022
7ff9252
Try out export command
EmilyBourne Dec 13, 2022
655eff4
List containers
EmilyBourne Dec 13, 2022
e53a3e5
Run to create container?
EmilyBourne Dec 13, 2022
1dce8a6
Examine images
EmilyBourne Dec 14, 2022
df066a0
Remove broken line
EmilyBourne Dec 14, 2022
9ab89d5
Aim to use save
EmilyBourne Dec 14, 2022
ec12396
Change command
EmilyBourne Dec 14, 2022
2e5717e
Examine
EmilyBourne Dec 14, 2022
e1a6d80
Testing
EmilyBourne Dec 14, 2022
83a822a
Comment
EmilyBourne Dec 14, 2022
10a5b3b
Examine
EmilyBourne Dec 14, 2022
d11d17b
Load tarball
EmilyBourne Dec 14, 2022
c5a9034
Try lower case
EmilyBourne Dec 14, 2022
0ddadde
Try clearer name
EmilyBourne Dec 14, 2022
df91d34
Missing line continuation
EmilyBourne Dec 14, 2022
0a8001d
Return closer to original command
EmilyBourne Dec 14, 2022
3b9ace8
Use strategy for tests
EmilyBourne Dec 14, 2022
e9a5073
Try to be more agressive
EmilyBourne Dec 14, 2022
d2d1f6d
Disk space
EmilyBourne Dec 16, 2022
f0371b3
Add missed disk space
EmilyBourne Dec 16, 2022
5485cb0
Revert unnecessary change
EmilyBourne Mar 16, 2023
8dd358d
Merge branch 'main' into 106-docker-artifact
jbigot Mar 16, 2023
0b445b0
Merge branch 'Maison-de-la-Simulation:main' into main
EmilyBourne Apr 28, 2023
ec98c78
Merge remote-tracking branch 'origin/main' into 106-docker-artifact
EmilyBourne Apr 28, 2023
8622925
Use ifs to treat different cases
EmilyBourne Apr 28, 2023
a74ba1c
Missing quotes
EmilyBourne Apr 28, 2023
ba8cacb
Print values
EmilyBourne Apr 28, 2023
3fb00fd
Print values
EmilyBourne Apr 28, 2023
54a7884
Print names
EmilyBourne Apr 28, 2023
fc16c9d
Print full names
EmilyBourne Apr 28, 2023
734b258
Test run on push
EmilyBourne Apr 28, 2023
4a3ef0e
Simplify and reduce duplication
EmilyBourne Apr 28, 2023
d830516
Examine
EmilyBourne Apr 28, 2023
c83dbe5
Correct comparison
EmilyBourne Apr 28, 2023
6da34d9
Remove push test
EmilyBourne Apr 28, 2023
dc9833b
Revert trigger
EmilyBourne Apr 28, 2023
8090111
Merge branch 'main' into 106-docker-artifact
EmilyBourne Jun 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 47 additions & 4 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,21 @@ on:
branches: [ main ]
jbigot marked this conversation as resolved.
Show resolved Hide resolved
pull_request:
jobs:
id_repo:
runs-on: ubuntu-latest
outputs:
in_base_repo: ${{ steps.identify_repo.outputs.in_base_repo }}
steps:
- name: Identify repository
id: identify_repo
run: |
echo "in_base_repo=${{ (github.event == 'push' && github.repository == 'Maison-de-la-Simulation/ddc') || github.event.pull_request.head.repo.full_name == 'Maison-de-la-Simulation/ddc' }}" >> "$GITHUB_OUTPUT"
echo "$GITHUB_OUTPUT"
docker-build:
needs: id_repo
runs-on: ubuntu-latest
outputs:
doxygen_image: ${{ steps.image_output.outputs.doxygen_image }}
steps:
- name: Checkout built branch
uses: actions/checkout@v3
Expand All @@ -17,21 +30,51 @@ jobs:
--cache-from ghcr.io/maison-de-la-simulation/ddc/doxygen \
-t ghcr.io/maison-de-la-simulation/ddc/doxygen \
-t ghcr.io/maison-de-la-simulation/ddc/doxygen:${GITHUB_SHA:0:7} \
-t doxygen_image \
docker/doxygen
- name: Publish doxygen artifact for current SHA
if: ${{ needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
echo "doxygen_image=ghcr.io/maison-de-la-simulation/ddc/doxygen:${GITHUB_SHA:0:7}" >> $GITHUB_ENV
docker push ghcr.io/maison-de-la-simulation/ddc/doxygen:${GITHUB_SHA:0:7}
- name: Create doxygen docker artifact
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
echo "doxygen_image=doxygen_image" >> $GITHUB_ENV
docker save doxygen_image > doxygen.tar
- name: Publish
if: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
if: ${{ github.event_name == 'push' && github.ref_name == 'main' && needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
docker push ghcr.io/maison-de-la-simulation/ddc/doxygen
- name: Save doxygen docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/upload-artifact@v3
with:
name: doxygen-artifact
path: doxygen.tar
retention-days: 1
- name: Output image
id: image_output
run: |
echo "doxygen_image=${doxygen_image}" >> "$GITHUB_OUTPUT"
pages:
runs-on: ubuntu-latest
needs: docker-build
needs: [docker-build, id_repo]
steps:
- name: Checkout built branch
uses: actions/checkout@v3
with:
fetch-depth: 0
submodules: true
- name: Collect doxygen docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/download-artifact@v3
with:
name: doxygen-artifact
- name: Load doxygen docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
docker load < doxygen.tar
- name: Build site
run: |
cat<<-EOF > run.sh
Expand All @@ -43,9 +86,9 @@ jobs:
EOF
mkdir docs_out
chmod a+rwx docs_out
docker run -v ${PWD}:/src ghcr.io/maison-de-la-simulation/ddc/doxygen:${GITHUB_SHA:0:7} bash /src/run.sh
docker run -v ${PWD}:/src ${{ needs.docker-build.outputs.doxygen_image }} bash /src/run.sh
- name: Publish site
if: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
if: ${{ github.event_name == 'push' && github.ref_name == 'main' && needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
git worktree add -B gh-pages public remotes/origin/gh-pages
find public -mindepth 1 -maxdepth 1 '!' -name .git -exec rm -rf '{}' '+'
Expand Down
171 changes: 160 additions & 11 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,22 @@ on:
branches: [ main ]
pull_request:
jobs:
id_repo:
runs-on: ubuntu-latest
outputs:
in_base_repo: ${{ steps.identify_repo.outputs.in_base_repo }}
steps:
- name: Identify repository
id: identify_repo
run: |
echo "in_base_repo=${{ (github.event == 'push' && github.repository == 'Maison-de-la-Simulation/ddc') || github.event.pull_request.head.repo.full_name == 'Maison-de-la-Simulation/ddc' }}" >> "$GITHUB_OUTPUT"
echo "$GITHUB_OUTPUT"

docker-build-test_env:
needs: id_repo
runs-on: ubuntu-latest
outputs:
test_env_image: ${{ steps.image_output.outputs.test_env_image }}
steps:
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
Expand All @@ -17,7 +31,7 @@ jobs:
android: true
dotnet: true
haskell: true
large-packages: false
large-packages: true
swap-storage: true
- name: Checkout built branch
uses: actions/checkout@v3
Expand All @@ -29,14 +43,39 @@ jobs:
--cache-from ghcr.io/maison-de-la-simulation/ddc/test_env \
-t ghcr.io/maison-de-la-simulation/ddc/test_env \
-t ghcr.io/maison-de-la-simulation/ddc/test_env:${GITHUB_SHA:0:7} \
-t test_env_image \
docker/test_env
- name: Publish test_env artifact for current SHA
if: ${{ needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
echo "test_env_image=ghcr.io/maison-de-la-simulation/ddc/test_env:${GITHUB_SHA:0:7}" >> $GITHUB_ENV
docker push ghcr.io/maison-de-la-simulation/ddc/test_env:${GITHUB_SHA:0:7}
- name: Create test_env docker artifact
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
echo "test_env_image=test_env_image" >> $GITHUB_ENV
docker save test_env_image > test_env.tar
- name: Publish
if: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
if: ${{ github.event_name == 'push' && github.ref_name == 'main' && needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
docker push ghcr.io/maison-de-la-simulation/ddc/test_env
- name: Save test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/upload-artifact@v3
with:
name: test_env-artifact
path: test_env.tar
retention-days: 1
- name: Output image
id: image_output
run: |
echo "test_env_image=${test_env_image}" >> "$GITHUB_OUTPUT"

docker-build-jammy_env:
needs: id_repo
runs-on: ubuntu-latest
outputs:
jammy_image: ${{ steps.image_output.outputs.jammy_image }}
steps:
- name: Checkout built branch
uses: actions/checkout@v3
Expand All @@ -48,12 +87,34 @@ jobs:
--cache-from ghcr.io/maison-de-la-simulation/ddc/jammy_env \
-t ghcr.io/maison-de-la-simulation/ddc/jammy_env \
-t ghcr.io/maison-de-la-simulation/ddc/jammy_env:${GITHUB_SHA:0:7} \
-t jammy_env_image \
docker/jammy_env
- name: Publish jammy_env artifact for current SHA
if: ${{ needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
echo "jammy_image=ghcr.io/maison-de-la-simulation/ddc/jammy_env:${GITHUB_SHA:0:7}" >> $GITHUB_ENV
docker push ghcr.io/maison-de-la-simulation/ddc/jammy_env:${GITHUB_SHA:0:7}
- name: Create jammy_env artifact
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
echo "jammy_image=jammy_env_image" >> $GITHUB_ENV
docker save jammy_env_image > jammy_env.tar
- name: Publish
if: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
if: ${{ github.event_name == 'push' && github.ref_name == 'main' && needs.id_repo.outputs.in_base_repo == 'true' }}
run: |
docker push ghcr.io/maison-de-la-simulation/ddc/jammy_env
- name: Save jammy_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/upload-artifact@v3
with:
name: jammy_env-artifact
path: jammy_env.tar
retention-days: 1
- name: Output image
id: image_output
run: |
echo "jammy_image=${jammy_image}" >> "$GITHUB_OUTPUT"

lint:
runs-on: ubuntu-latest
steps:
Expand All @@ -64,14 +125,36 @@ jobs:
exclude: ''
extensions: 'hpp,cpp'
clangFormatVersion: 12

test-Debug:
runs-on: ubuntu-latest
needs: docker-build-test_env
needs: [docker-build-test_env, id_repo]
steps:
- name: Checkout built branch
uses: actions/checkout@v3
with:
submodules: recursive
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false

# all of these default to true, but feel free to set to
# "false" if necessary for your workflow
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: Collect test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/download-artifact@v3
with:
name: test_env-artifact
- name: Load test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
docker load < test_env.tar
- name: Build
run: |
cat<<-EOF > run.sh
Expand All @@ -81,15 +164,37 @@ jobs:
make -j 2 VERBOSE=1
ctest -j 2 --output-on-failure --timeout 5 --output-junit tests.xml
EOF
docker run -v ${PWD}:/src:ro ghcr.io/maison-de-la-simulation/ddc/test_env:${GITHUB_SHA:0:7} bash /src/run.sh
docker run -v ${PWD}:/src:ro ${{ needs.docker-build-test_env.outputs.test_env_image }} bash /src/run.sh

test-Release:
runs-on: ubuntu-latest
needs: docker-build-test_env
needs: [docker-build-test_env, id_repo]
steps:
- name: Checkout built branch
uses: actions/checkout@v3
with:
submodules: recursive
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false

# all of these default to true, but feel free to set to
# "false" if necessary for your workflow
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: Collect test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/download-artifact@v3
with:
name: test_env-artifact
- name: Load test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
docker load < test_env.tar
- name: Build
run: |
cat<<-EOF > run.sh
Expand All @@ -99,15 +204,37 @@ jobs:
make -j 2 VERBOSE=1
ctest -j 2 --output-on-failure --timeout 5 --output-junit tests.xml
EOF
docker run -v ${PWD}:/src:ro ghcr.io/maison-de-la-simulation/ddc/test_env:${GITHUB_SHA:0:7} bash /src/run.sh
docker run -v ${PWD}:/src:ro ${{ needs.docker-build-test_env.outputs.test_env_image }} bash /src/run.sh

test-cuda:
runs-on: ubuntu-latest
needs: docker-build-jammy_env
needs: [docker-build-jammy_env, id_repo]
steps:
- name: Checkout built branch
uses: actions/checkout@v3
with:
submodules: recursive
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false

# all of these default to true, but feel free to set to
# "false" if necessary for your workflow
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: Collect jammy_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/download-artifact@v3
with:
name: jammy_env-artifact
- name: Load jammy_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
docker load < jammy_env.tar
- name: Build
run: |
cat<<-EOF > run.sh
Expand All @@ -119,15 +246,37 @@ jobs:
cmake -DCMAKE_CXX_FLAGS="-Wall -Wextra -pedantic -Werror=vla" -DBUILD_BENCHMARKS=ON -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_AMPERE80=ON -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_DEPRECATED_CODE_3=OFF -DKokkos_ENABLE_DEPRECATED_CODE_4=OFF -DKokkos_ENABLE_DEPRECATION_WARNINGS=OFF /src
make -j 2 VERBOSE=1
EOF
docker run -v ${PWD}:/src:ro ghcr.io/maison-de-la-simulation/ddc/jammy_env:${GITHUB_SHA:0:7} bash /src/run.sh
docker run -v ${PWD}:/src:ro ${{ needs.docker-build-jammy_env.outputs.jammy_image }} bash /src/run.sh

test-hip:
runs-on: ubuntu-latest
needs: docker-build-test_env
needs: [docker-build-test_env, id_repo]
steps:
- name: Checkout built branch
uses: actions/checkout@v3
with:
submodules: recursive
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false

# all of these default to true, but feel free to set to
# "false" if necessary for your workflow
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: Collect test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
uses: actions/download-artifact@v3
with:
name: test_env-artifact
- name: Load test_env docker
if: ${{ needs.id_repo.outputs.in_base_repo == 'false' }}
run: |
docker load < test_env.tar
- name: Build
run: |
cat<<-EOF > run.sh
Expand All @@ -138,4 +287,4 @@ jobs:
cmake -DCMAKE_CXX_FLAGS="-Wall -Wextra -pedantic -Werror=vla" -DBUILD_BENCHMARKS=ON -DKokkos_ENABLE_HIP=ON -DKokkos_ARCH_VEGA90A=ON -DCMAKE_PREFIX_PATH=/opt/rocm -DKokkos_ENABLE_DEPRECATED_CODE_3=OFF -DKokkos_ENABLE_DEPRECATED_CODE_4=OFF -DKokkos_ENABLE_DEPRECATION_WARNINGS=OFF /src
make -j 2 VERBOSE=1
EOF
docker run -v ${PWD}:/src:ro ghcr.io/maison-de-la-simulation/ddc/test_env:${GITHUB_SHA:0:7} bash /src/run.sh
docker run -v ${PWD}:/src:ro ${{ needs.docker-build-test_env.outputs.test_env_image }} bash /src/run.sh