Auto merge of rust-lang#92214 - ehuss:submodule-bg-exit, r=Mark-Simul…

…acrum

Error if submodule fetch fails.

In CI, if fetching a submodule fails, the script would exit successfully. Later parts of the build will fail due to the missing files, but it is a bit confusing, and I think it would be better to error out earlier.

The reason is that in bash, `wait` without arguments will exit 0 even if a background job exits with an error. The solution here is to wait on each individual job, which will return the exit code of the job.

This was encountered in rust-lang#92177.

.github/workflows/ci.yml

@@ -104,6 +104,9 @@ jobs:
       - name: disable git crlf conversion
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         if: success() && !env.SKIP_JOB
+      - name: checkout submodules
+        run: src/ci/scripts/checkout-submodules.sh
+        if: success() && !env.SKIP_JOB
       - name: install MSYS2
         run: src/ci/scripts/install-msys2.sh
         if: success() && !env.SKIP_JOB
@@ -119,9 +122,6 @@ jobs:
       - name: disable git crlf conversion
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         if: success() && !env.SKIP_JOB
-      - name: checkout submodules
-        run: src/ci/scripts/checkout-submodules.sh
-        if: success() && !env.SKIP_JOB
       - name: ensure line endings are correct
         run: src/ci/scripts/verify-line-endings.sh
         if: success() && !env.SKIP_JOB
@@ -502,6 +502,9 @@ jobs:
       - name: disable git crlf conversion
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         if: success() && !env.SKIP_JOB
+      - name: checkout submodules
+        run: src/ci/scripts/checkout-submodules.sh
+        if: success() && !env.SKIP_JOB
       - name: install MSYS2
         run: src/ci/scripts/install-msys2.sh
         if: success() && !env.SKIP_JOB
@@ -517,9 +520,6 @@ jobs:
       - name: disable git crlf conversion
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         if: success() && !env.SKIP_JOB
-      - name: checkout submodules
-        run: src/ci/scripts/checkout-submodules.sh
-        if: success() && !env.SKIP_JOB
       - name: ensure line endings are correct
         run: src/ci/scripts/verify-line-endings.sh
         if: success() && !env.SKIP_JOB
@@ -615,6 +615,9 @@ jobs:
       - name: disable git crlf conversion
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         if: success() && !env.SKIP_JOB
+      - name: checkout submodules
+        run: src/ci/scripts/checkout-submodules.sh
+        if: success() && !env.SKIP_JOB
       - name: install MSYS2
         run: src/ci/scripts/install-msys2.sh
         if: success() && !env.SKIP_JOB
@@ -630,9 +633,6 @@ jobs:
       - name: disable git crlf conversion
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         if: success() && !env.SKIP_JOB
-      - name: checkout submodules
-        run: src/ci/scripts/checkout-submodules.sh
-        if: success() && !env.SKIP_JOB
       - name: ensure line endings are correct
         run: src/ci/scripts/verify-line-endings.sh
         if: success() && !env.SKIP_JOB

src/ci/github-actions/ci.yml

@@ -169,6 +169,10 @@ x--expand-yaml-anchors--remove:
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         <<: *step
 
+      - name: checkout submodules
+        run: src/ci/scripts/checkout-submodules.sh
+        <<: *step
+
       - name: install MSYS2
         run: src/ci/scripts/install-msys2.sh
         <<: *step
@@ -194,10 +198,6 @@ x--expand-yaml-anchors--remove:
         run: src/ci/scripts/disable-git-crlf-conversion.sh
         <<: *step
 
-      - name: checkout submodules
-        run: src/ci/scripts/checkout-submodules.sh
-        <<: *step
-
       - name: ensure line endings are correct
         run: src/ci/scripts/verify-line-endings.sh
         <<: *step

src/ci/init_repo.sh

@@ -43,6 +43,11 @@ function fetch_github_commit_archive {
         curl -f -sSL -o $cached $2"
     mkdir $module
     touch "$module/.git"
+    # On Windows, the default behavior is to emulate symlinks by copying
+    # files. However, that ends up being order-dependent while extracting,
+    # which can cause a failure if the symlink comes first. This env var
+    # causes tar to use real symlinks instead, which are allowed to dangle.
+    export MSYS=winsymlinks:nativestrict
     tar -C $module --strip-components=1 -xf $cached
     rm $cached
 }
@@ -62,6 +67,7 @@ for i in ${!modules[@]}; do
         url=${urls[$i]}
         url=${url/\.git/}
         fetch_github_commit_archive $module "$url/archive/$commit.tar.gz" &
+        bg_pids[${i}]=$!
         continue
     else
         use_git="$use_git $module"
@@ -70,4 +76,9 @@ done
 retry sh -c "git submodule deinit -f $use_git && \
     git submodule sync && \
     git submodule update -j 16 --init --recursive $use_git"
-wait
+STATUS=0
+for pid in ${bg_pids[*]}
+do
+    wait $pid || STATUS=1
+done
+exit ${STATUS}