Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] Share Generated Assets Across All Platforms #1074

Merged
merged 6 commits into from
Aug 8, 2022
Merged

[ci] Share Generated Assets Across All Platforms #1074

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
49cb9c9
Generate assets first on Jenkins
dcvz Aug 6, 2022
33c576f
Remove file copy
dcvz Aug 6, 2022
8de8ae4
Share assets with Windows
dcvz Aug 6, 2022
2e5089f
Merge branch 'develop-zhora' into ci/asset-generation
dcvz Aug 8, 2022
0402151
Run extraction in Release mode (fast)
dcvz Aug 8, 2022
7c439db
Update timeouts
dcvz Aug 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 33 additions & 14 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,32 @@ pipeline {
}

stages {
stage('Generate Assets') {
options {
timeout(time: 20)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we are extracting assets once and stashing them, we can probably remove this timeout from the other stages. That was added due to a rare bug where extraction would hang up (haven't seen it for a while, fixed now?) or get stuck in an infinite loop (saw once). It should help with stages timing out when multiple jobs are queued, as well.

Alternatively, we could just set the timeouts on the other stages to a much higher value, like 60 min or something.

Copy link
Contributor Author

@dcvz dcvz Aug 8, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sholdee I can remove the timeouts. What does it use for default when there's no timeout?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will not timeout unless explicitly set.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll set it to 60 for now as a safeguard and we can keep an eye on it for a bit..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just saw macOS doesn't have a timeout and that's been ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated and removed

}
agent {
label "SoH-Mac-Builders"
}
steps {
checkout([
$class: 'GitSCM',
branches: scm.branches,
doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
extensions: scm.extensions,
userRemoteConfigs: scm.userRemoteConfigs
])
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh '''
cp ../../ZELOOTD.z64 OTRExporter/baserom_non_mq.z64

cmake --no-warn-unused-cli -H. -Bbuild-cmake -GNinja -DCMAKE_BUILD_TYPE:STRING=Release
cmake --build build-cmake --target ExtractAssets --config Release
'''
stash includes: 'soh/assets/**/*', name: 'assets'
}
}
}
stage('Build SoH') {
parallel {
stage ('Build Windows') {
Expand All @@ -34,12 +60,9 @@ pipeline {
])

catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
bat """

xcopy "..\\..\\ZELOOTD.z64" "OTRExporter\\"

"${env.CMAKE}" -S . -B "build\\${env.PLATFORM}" -G "Visual Studio 17 2022" -T ${env.TOOLSET} -A ${env.PLATFORM} -D Python3_EXECUTABLE=${env.PYTHON} -D CMAKE_BUILD_TYPE:STRING=Release
"${env.CMAKE}" --build ".\\build\\${env.PLATFORM}" --target ExtractAssets --config Release
unstash 'assets'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i know it shouldn't be an issue, but just want to make sure that there's nothing funky with using assets generated on linux for windows

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't due to the comment below :)

bat """
"${env.CMAKE}" -S . -B "build\\${env.PLATFORM}" -G "Visual Studio 17 2022" -T ${env.TOOLSET} -A ${env.PLATFORM} -D Python_EXECUTABLE=${env.PYTHON} -D CMAKE_BUILD_TYPE:STRING=Release
"${env.CMAKE}" --build ".\\build\\${env.PLATFORM}" --config Release
cd ".\\build\\${env.PLATFORM}"
"${env.CPACK}" -G ZIP
Expand Down Expand Up @@ -72,9 +95,8 @@ pipeline {
userRemoteConfigs: scm.userRemoteConfigs
])
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
unstash 'assets'
sh '''

cp ../../ZELOOTD.z64 OTRExporter/baserom_non_mq.z64
docker build . -t soh
docker run --name sohcont -dit --rm -v $(pwd):/soh soh /bin/bash
docker exec sohcont scripts/linux/appimage/build.sh
Expand Down Expand Up @@ -108,11 +130,9 @@ pipeline {
userRemoteConfigs: scm.userRemoteConfigs
])
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
unstash 'assets'
sh '''
cp ../../ZELOOTD.z64 OTRExporter/baserom_non_mq.z64

cmake --no-warn-unused-cli -H. -Bbuild-cmake -GNinja -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"
cmake --build build-cmake --target ExtractAssets --
cmake --build build-cmake --config Release --
(cd build-cmake && cpack)

Expand Down Expand Up @@ -146,9 +166,8 @@ pipeline {
userRemoteConfigs: scm.userRemoteConfigs
])
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh '''

cp ../../ZELOOTD.z64 OTRExporter/baserom_non_mq.z64
unstash 'assets'
sh '''
docker build . -t sohswitch
docker run --name sohcont -dit --rm -v $(pwd):/soh sohswitch /bin/bash
docker exec sohcont scripts/switch/build.sh
Expand Down
7 changes: 5 additions & 2 deletions ZAPDTR/ZAPD/ZResource.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,11 +348,14 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s\"", name.c_str(), outName.c_str(), nameStr.c_str());

if (nameSet && nameSet->find(name) == nameSet->end()) {
str += StringHelper::Sprintf(R"(
#ifdef _WIN32
str += StringHelper::Sprintf("\nstatic const __declspec(align(2)) char %s[] = d%s;", name.c_str(), name.c_str());
static const __declspec(align(2)) char %s[] = d%s;
#else
str += StringHelper::Sprintf("\nstatic const char %s[] __attribute__((aligned (2))) = d%s;", name.c_str(), name.c_str());
static const char %s[] __attribute__((aligned (2))) = d%s;
#endif
)", name.c_str(), name.c_str(), name.c_str(), name.c_str());

Comment on lines +351 to +358
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels a little harder to read than what was in there before, but i can't think of a cleaner way to do it off the top of my head

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, before we just generated the final outcome for the compiling platform.. now because its shared between both platforms, I need to generate an output that works on MSVC + gnu/clang

if (nameSet) {
nameSet->insert(name);
}
Expand Down
1 change: 0 additions & 1 deletion scripts/linux/appimage/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
#!/bin/bash

cmake --no-warn-unused-cli -H. -Bbuild-cmake -GNinja -DCMAKE_BUILD_TYPE:STRING=Release
cmake --build build-cmake --target ExtractAssets --
cmake --build build-cmake --config Release --

(cd build-cmake && cpack -G External)
3 changes: 0 additions & 3 deletions scripts/switch/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
#!/bin/bash

cmake --no-warn-unused-cli -H. -Bbuild-linux -GNinja
cmake --build build-linux --target ExtractAssets

cmake -H. -Bbuild-switch -GNinja -DCMAKE_TOOLCHAIN_FILE=/opt/devkitpro/cmake/Switch.cmake
cmake --build build-switch --target soh_nro