Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Jenkins] Add NOMLIR stage. [Workaround] Limit usage of gfx908 nodes in non-nightly builds #2622

Merged
merged 2 commits into from
Dec 20, 2023
Merged

[Jenkins] Add NOMLIR stage. [Workaround] Limit usage of gfx908 nodes in non-nightly builds #2622

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e709acf
add nomlir stage and make gfx908 specific stages nightly
junliume Dec 20, 2023
968621f
Tiny doc fix
junliume Dec 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ def CheckPerfDbValid(Map conf=[:]){
/// * The default compiler is usually not specified.
/// BuildType := { Release* | Debug | Install } [ BuildTypeModifier ]
/// * BuildTypeModifier := { NOCOMGR | Embedded | Static | Normal-Find | Fast-Find
/// CK | MLIR | Tensile | Tensile-Latest | Package | ... }
/// CK | NOMLIR | Tensile | Tensile-Latest | Package | ... }
/// TestSet := { All | Smoke* | Performance Dataset } [ Codecov ]
/// * "All" corresponds to "cmake -DMIOPEN_TEST_ALL=On".
/// * "Smoke" (-DMIOPEN_TEST_ALL=Off) is the default and usually not specified.
Expand Down Expand Up @@ -466,7 +466,7 @@ pipeline {
description: "")
booleanParam(
name: "TARGET_GFX908",
defaultValue: true,
defaultValue: env.BRANCH_NAME == env.NIGHTLY_BRANCH ? true : false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Notice] AnyGPU stages ignore this. AnyGPU stages of non-nightly builds will occupy gfx908 nodes.

[R] Mention that this is a W/A for the MI100 node shortage problem.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atamazov yes, and it is left on purpose. We want all stages not to be restricted to gfx908, but sometimes anyGPU stages may occupy (but not limited by) gfx908 nodes and if issues found it is still legit.

description: "")
booleanParam(
name: "TARGET_GFX90A",
Expand Down Expand Up @@ -530,8 +530,9 @@ pipeline {
Bf16_flags = " -DMIOPEN_TEST_BFLOAT16=On"
Int8_flags = " -DMIOPEN_TEST_INT8=On"
Full_test = " -DMIOPEN_TEST_ALL=On"
Smoke_targets = "check MIOpenDriver"
Smoke_targets = " check MIOpenDriver"
NOCOMGR_flags = " -DMIOPEN_USE_COMGR=Off"
NOMLIR_flags = " -DMIOPEN_USE_MLIR=Off"
}
triggers{

Expand Down Expand Up @@ -711,6 +712,23 @@ pipeline {
buildHipClangJobAndReboot( build_type: 'debug', setup_flags: NOCOMGR_flags, build_cmd: NOCOMGR_build_cmd, test_flags: ' --verbose ')
}
}
stage('Fp32 Hip Debug NOMLIR AnyGPU') {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[R] Please update

MIOpen/Jenkinsfile

Lines 407 to 409 in 10250d6

/// BuildType := { Release* | Debug | Install } [ BuildTypeModifier ]
/// * BuildTypeModifier := { NOCOMGR | Embedded | Static | Normal-Find | Fast-Find
/// CK | MLIR | Tensile | Tensile-Latest | Package | ... }

MLIR -> NOMLIR

when {
beforeAgent true
expression { params.TARGET_VEGA20 || params.TARGET_VEGA10 || params.TARGET_GFX908 || params.TARGET_GFX90A }
}
options {
retry(2)
}
agent{ label rocmnode("vega || gfx908 || gfx90a") }
environment{
// Can be removed altogether with when WORKAROUND_SWDEV_290754.
NOMLIR_build_cmd = "CTEST_PARALLEL_LEVEL=4 MIOPEN_CONV_PRECISE_ROCBLAS_TIMING=0 MIOPEN_LOG_LEVEL=5 make -j\$(nproc) check"
}
steps{
buildHipClangJobAndReboot( build_type: 'debug', setup_flags: NOMLIR_flags, build_cmd: NOMLIR_build_cmd, test_flags: ' --verbose ')
}
}
stage('Fp32 Hip Debug Embedded Vega20') {
when {
beforeAgent true
Expand Down