Skip to content

Commit

Permalink
Fix release pipeline (#851)
Browse files Browse the repository at this point in the history
**Issue:**
The most recent release failed for 2 different reasons:
1) Many cross-compilation jobs failed, they all [use builder like this](https://github.com/awslabs/aws-crt-java/blob/ff164f25866afbcea64d61226904eb2ab420b348/codebuild/cd/generic-unix-build.sh#L31-L33)
    - failed while doing: `mv target/cmake-build/aws-crt-java/* target/cmake-build/`
    - error message: "target/cmake-build/deps already present"
2) The `aws-crt-java-manylinux-x64-fips` job failed with error "Could not find Go"

**Description of changes:**
1) Fix 1st issue by doing `mv` on the 1 directory we care about, instead of doing `mv` on a wildcard where lots of stuff we don't care about can collide
2) Fix 2nd issue by installing golang via package manager
    - Previously, it was manually unpacking golang to a folder, and adding a relative path (`.go/go/bin`) to `$PATH` (which is weird usually you add absolute paths). And broke when the new "prebuild aws-lc" logic changed the working directory
3) Other small fixes
  • Loading branch information
graebm authored Dec 2, 2024
1 parent ff164f2 commit 91f9370
Show file tree
Hide file tree
Showing 5 changed files with 38 additions and 9 deletions.
3 changes: 3 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ if (BUILD_DEPS)
list(APPEND AWSLC_CMAKE_ARGUMENTS -DFIPS=ON)
list(APPEND AWSLC_CMAKE_ARGUMENTS -DDISABLE_PERL=OFF)

# Pick up GO_PATH env-var, set by aws-crt-builder when cross-compiling, see:
# https://github.com/awslabs/aws-crt-builder/blob/31307c808ed9f2ea1eb16503b25a9b582f886481/builder/imports/golang.py#L84
# https://github.com/awslabs/aws-crt-builder/blob/31307c808ed9f2ea1eb16503b25a9b582f886481/builder/actions/cmake.py#L110
if (DEFINED ENV{GO_PATH})
list(APPEND AWSLC_CMAKE_ARGUMENTS -DGO_EXECUTABLE=$ENV{GO_PATH}/go)
message(STATUS "Overriding GO_EXECUTABLE to ${GO_EXECUTABLE}")
Expand Down
16 changes: 13 additions & 3 deletions codebuild/cd/generic-unix-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,20 @@ chmod a+x builder
GIT_TAG=$(git describe --tags)

./builder build -p aws-crt-java --target=$AWS_CRT_TARGET run_tests=false
# Builder corss-compiles the shared lib to `target/cmake-build/aws-crt-java/`, move it to the expected path for mvn to generate the jar.
mv target/cmake-build/aws-crt-java/* target/cmake-build/

# When cross-compiling with builder, the shared lib gets an extra "/aws-crt-java/" in its path.
# Move it to expected location.
if [ -d target/cmake-build/aws-crt-java/lib ]; then
mv target/cmake-build/aws-crt-java/lib target/cmake-build/lib
fi

# Double check that shared lib is where we expect
if ! find target/cmake-build/lib -type f -name "*.so" | grep -q .; then
echo "No .so files found"
exit 1
fi

JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn -B package -DskipTests -Dshared-lib.skip=true -Dcrt.classifier=$CLASSIFIER

aws s3 cp --recursive --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
aws s3 cp --recursive --exclude "*" --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
aws s3 cp target/ s3://aws-crt-java-pipeline/${GIT_TAG}/jar/ --recursive --exclude "*" --include "aws-crt*.jar"
15 changes: 13 additions & 2 deletions codebuild/cd/linux-aarch64-fips-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,19 @@ chmod a+x builder
GIT_TAG=$(git describe --tags)

./builder build -p aws-crt-java run_tests=false --target=linux-arm64 --cmake-extra=-DCRT_FIPS=ON
mv target/cmake-build/aws-crt-java/* target/cmake-build/

# When cross-compiling with builder, the shared lib gets an extra "/aws-crt-java/" in its path.
# Move it to expected location.
if [ -d target/cmake-build/aws-crt-java/lib ]; then
mv target/cmake-build/aws-crt-java/lib target/cmake-build/lib
fi

# Double check that shared lib is where we expect
if ! find target/cmake-build/lib -type f -name "*.so" | grep -q .; then
echo "No .so files found"
exit 1
fi

JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn -B package -DskipTests -Dshared-lib.skip=true -Dcrt.classifier=linux-aarch_64-fips

aws s3 cp --recursive --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/fips_lib
aws s3 cp --recursive --exclude "*" --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/fips_lib
5 changes: 2 additions & 3 deletions codebuild/cd/manylinux-x64-fips-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@ phases:
- git submodule update --init
# double check aws-lc is the FIPS approved branch.
- bash ./codebuild/cd/test-fips-branch.sh
- curl -OL https://go.dev/dl/go1.21.6.linux-amd64.tar.gz && mkdir ./go
- tar -C ./go -xvf go1.21.6.linux-amd64.tar.gz
- export PATH=$PATH:./go/go/bin
# aws-lc FIPS build requires golang for codegen
- yum install -y golang
- mvn -B package -DskipTests -Dcrt.classifier=linux-x86_64-fips -Dcmake.crt_fips=ON

post_build:
Expand Down
8 changes: 7 additions & 1 deletion codebuild/cd/musl-linux-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,11 @@ docker container prune -f
# Upload the artifacts to S3
export GIT_TAG=$(git describe --tags)

aws s3 cp --recursive --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
# Double check that shared lib is where we expect
if ! find target/cmake-build/lib -type f -name "*.so" | grep -q .; then
echo "No .so files found"
exit 1
fi

aws s3 cp --recursive --exclude "*" --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
aws s3 cp target/ s3://aws-crt-java-pipeline/${GIT_TAG}/jar/ --recursive --exclude "*" --include "aws-crt*.jar"

0 comments on commit 91f9370

Please sign in to comment.