Added system_lib for system prestage jars #5239

annaibm · 2024-04-18T13:01:57Z

Prestage system test prequisities to system_lib

related: #4912

annaibm · 2024-04-18T13:02:38Z

Grinder links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/39879/consoleFull
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/39882/console
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/39906/consoleFull

llxia · 2024-04-18T17:28:52Z

pre-stage jars are not used. asm.jar still got downloaded to /home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs/asm/:

00:08:53.766  configure-asm:
00:08:53.766       [echo] Executing macro download-file
00:08:53.766       [echo] File to download: https://repository.ow2.org/nexus/content/repositories/releases/org/ow2/asm/asm/9.0/asm-9.0.jar
00:08:53.766       [echo] Destination: /home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs/asm/asm.jar
00:08:53.766       [echo] Download tool: curl
00:08:53.766      [mkdir] Created dir: /home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs/asm
00:08:53.766       [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
00:08:53.766       [exec]                                  Dload  Upload   Total   Spent    Left  Speed
00:08:53.766       [exec] 
00:08:54.639       [exec]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0100  118k  100  118k    0     0   114k      0  0:00:01  0:00:01 --:--:--  114k
00:08:54.639

I suspect that we need to update the reference in system test repo.

llxia · 2024-04-18T18:12:27Z

Try change this line

aqa-tests/system/system.mk

Line 77 in 3ae4045

-systemtest-prereqs=$(Q)$(SYSTEMTEST_RESROOT)$(D)systemtest_prereqs$(Q) \

llxia · 2024-05-10T19:30:44Z

system/system.mk

@@ -71,15 +71,21 @@ ifdef JVM_OPTIONS
  APPLICATION_OPTIONS := $(APPLICATION_OPTIONS) -jvmArgs $(Q)$(JVM_OPTIONS)$(Q)
 endif

+ifndef SYSTEM_LIB_DIR
+  SYSTEM_LIB_DIR := "/home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs"


Try to set SYSTEM_LIB_DIR :=$(SYSTEMTEST_RESROOT)/systemtest_prereqs

llxia · 2024-05-10T19:33:03Z

system/common.xml

@@ -337,7 +343,7 @@
 				</then>
 		</if>
 		<copy todir="${SYSTEMTEST_DEST}/systemtest_prereqs/">
-			<fileset dir="${TEST_ROOT}/systemtest_prereqs/" includes="**" />
+			<fileset dir="${system_lib_dir}" includes="**"/>


Why do we still need to copy to systemtest_prereqs dir?

No, we don't need to copy to the systemtest_prereqs directory. I had kept it to check if local tests might need it; apparently, it's not needed. I will remove it.

llxia · 2024-05-10T19:33:41Z

buildenv/jenkins/JenkinsfileBase

+					sh "perl ./aqa-tests/TKG/scripts/getDependencies.pl -path ${env.LIB_DIR} -task default -customUrl ${customUrl}"
+				} else {
+					sh "perl ./aqa-tests/TKG/scripts/getDependencies.pl -path ${env.LIB_DIR} -task default"
+				}


for system tests, we do not need customUrl?

Passing the customUrl for system tests was duplicating the process of downloading the jar files.

why having customUrl will duplicate the process? When running at openj9 Jenkins, we should try to use systemtest.getDependency from openj9 Jenkins, not Adoptium.

@llxia I have used customUrl and replaced test.getDependency to systemtest.getDependency for system test libs in TKG.

Grinder links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40841/

llxia · 2024-05-10T19:38:13Z

buildenv/jenkins/JenkinsfileBase

+				def defaultUrl = "https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/"
+				def systemUrl = "https://ci.adoptium.net/job/systemtest.getDependency/lastSuccessfulBuild/artifact/"
+				env.LIB_DIR = (env.BUILD_LIST == 'system') ? env.SYSTEM_LIB_DIR : env.LIB_DIR
+				def customUrl = (env.BUILD_LIST == 'system') ? systemUrl : defaultUrl


I would prefer to have if statement to make the logic clear:

if (env.BUILD_LIST == 'system') { env.LIB_DIR = env.SYSTEM_LIB_DIR customUrl = "https://ci.adoptium.net/job/systemtest.getDependency/lastSuccessfulBuild/artifact/" } else { customUrl = "https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/" }

Please note this change will conflict with #5289. Depends on which PR gets merged first, the other one will need to be updated.

annaibm · 2024-05-21T19:48:47Z

Grinder links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40745/ -linux
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40746/ -windows
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40768/ -mac

annaibm · 2024-05-24T20:43:53Z

Grinder links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40841/

llxia · 2024-05-27T15:49:49Z

aqa-tests/buildenv/jenkins/JenkinsfileBase

Lines 679 to 689 in a6dbf7c

    
           try { 
        
           	if (env.BUILD_LIST.contains('system')) { 
        
           		//get pre-staged test jars from systemtest.getDependency build before system test compilation 
        
           		timeout(time: 60, unit: 'MINUTES') { 
        
           			copyArtifacts fingerprintArtifacts: true, projectName: "systemtest.getDependency", selector: lastSuccessful(), target: 'aqa-tests' 
        
           		} 
        
           	} 
        
           } catch (Exception e) { 
        
           	echo 'Exception: ' + e.toString() 
        
           	echo 'Cannot run copyArtifacts from test.getDependency/systemtest.getDependency. Skipping copyArtifacts...' 
        
           }

should be removed.

buildenv/jenkins/JenkinsfileBase

system/common.xml

- Prestage system test prequisities to system_lib related: adoptium#4912 Signed-off-by: Anna Babu Palathingal <anna.bp@ibm.com>

annaibm · 2024-05-28T15:36:11Z

Grinder links:
Sytem test
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40892/
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40894/ (windows)
non -system test:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40893/
Local test passed:

LT  08:34:18.459 - Test stopped due to reaching runtime limit
LT  08:34:18.459 - Load test completed
LT  08:34:18.461 -   Ran     : 57047
LT  08:34:18.461 -   Passed  : 57047
LT  08:34:18.461 -   Failed  : 0
LT  08:34:18.461 -   Result  : PASSED
STF 08:34:19.009 - Monitoring Report Summary:
STF 08:34:19.009 -   o Process LT  ended with the expected exit code (0)
STF 08:34:19.009 - EXECUTE stage completed
STF 08:34:19.509 - 
STF 08:34:19.509 - ====================   T E A R D O W N   ====================
STF 08:34:19.509 - Running teardown: perl /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17169097929139/MiniMix_5m_1/20240528-082852-MixedLoadTest/tearDown.pl
STF 08:34:19.601 - TEARDOWN stage completed
STF 08:34:19.609 - 
STF 08:34:19.609 - =====================   R E S U L T S   =====================
STF 08:34:19.609 - Stage results:
STF 08:34:19.609 -   setUp:     pass
STF 08:34:19.609 -   execute:   pass
STF 08:34:19.609 -   teardown:  pass
STF 08:34:19.609 - 
STF 08:34:19.609 - Overall result: PASSED
-----------------------------------
MiniMix_5m_1_PASSED
-----------------------------------

TEST TEARDOWN:

Attempting to destroy all caches in cacheDir /home/jenkins/.cache/javasharedresources/

JVMSHRC806I Compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
JVMSHRC807I Non-compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xcompressedrefs if you want to destroy a compressed references cache.
JVMSHRC005I No shared class caches available
cache cleanup done
MiniMix_5m_1 Finish Time: Tue May 28 08:34:19 2024 Epoch Time (ms): 1716910459719

===============================================
Running test MiniMix_5m_2 ...
===============================================
MiniMix_5m_2 Start Time: Tue May 28 08:34:19 2024 Epoch Time (ms): 1716910459781
Skipped due to jvm options ( ) => MiniMix_5m_2_SKIPPED
MiniMix_5m_2 Finish Time: Tue May 28 08:34:19 2024 Epoch Time (ms): 1716910459809
make[4]: Leaving directory '/home/jenkins/workspace/Grinder/aqa-tests/system/otherLoadTest'
make[3]: Leaving directory '/home/jenkins/workspace/Grinder/aqa-tests/system'
make[2]: Leaving directory '/home/jenkins/workspace/Grinder/aqa-tests'


All tests finished, run result summary:
cd "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/scripts"; \
perl "resultsSum.pl" --failuremk="/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/failedtargets.mk" --resultFile="/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17169097929139/TestTargetResult" --platFile="/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/resources/buildPlatformMap.properties" --diagnostic=failure --jdkVersion=17 --jdkImpl=openj9 --jdkVendor="eclipse" --spec=linux_x86-64 --buildList=system --customTarget="" --testTarget=MiniMix_5m --tapPath=/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17169097929139/ --tapName= --comment=""


TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
	MiniMix_5m_0
	MiniMix_5m_1

TOTAL: 3   EXECUTED: 2   PASSED: 2   FAILED: 0   DISABLED: 0   SKIPPED: 1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ALL TESTS PASSED

_MiniMix_5m done

llxia · 2024-05-29T18:40:54Z

This PR has to be merged with adoptium/TKG#533 and adoptium/STF#135

llxia

LGTM

LongyuZhang

LGTM

This reverts commit a4ef8f2.

)" (adoptium#5361)" This reverts commit 071c84b.

)" (adoptium#5361)" Changes for 1443, to check if vendor test repo is SVTrepo and make changes accordingly. This reverts commit 071c84b.

- Revert "Revert "Added system_lib for system prestage jars (adoptium#5239)" (adoptium#5361)" This reverts commit 071c84b. - Check if vendor test repo is SVTTestRepo and if so create directories `systemtest_prereqs` under respective test directories. - Copy files from `system_lib` to the created directories so they can levarage the use of the system prestage jars. related: adoptium#4912 related: https://github.ibm.com/runtimes/backlog/issues/1443 Signed-off-by: Anna Babu Palathingal <anna.bp@ibm.com>

annaibm force-pushed the test_systemtest branch from c9768f6 to ff8692e Compare April 18, 2024 15:43

annaibm force-pushed the test_systemtest branch 3 times, most recently from ed08e77 to 961dd88 Compare April 24, 2024 16:56

annaibm changed the title ~~Added system_prereq_lib for system prestage jars~~ Added system_lib for system prestage jars Apr 24, 2024

annaibm force-pushed the test_systemtest branch 7 times, most recently from 86bdafd to 8d35f5b Compare May 1, 2024 14:24

annaibm force-pushed the test_systemtest branch 2 times, most recently from cdb93d2 to 6876b4f Compare May 9, 2024 14:36

annaibm force-pushed the test_systemtest branch from 07284af to ee2d527 Compare May 9, 2024 18:29

llxia reviewed May 10, 2024

View reviewed changes

annaibm force-pushed the test_systemtest branch 6 times, most recently from 2149e1d to fb539fd Compare May 17, 2024 20:34

annaibm force-pushed the test_systemtest branch 3 times, most recently from 327b8e2 to 7722d60 Compare May 21, 2024 19:12

llxia requested a review from LongyuZhang May 27, 2024 15:46

annaibm force-pushed the test_systemtest branch 2 times, most recently from 44d500c to 6ed29ac Compare May 27, 2024 17:33

LongyuZhang reviewed May 27, 2024

View reviewed changes

buildenv/jenkins/JenkinsfileBase Show resolved Hide resolved

system/common.xml Outdated Show resolved Hide resolved

annaibm force-pushed the test_systemtest branch 3 times, most recently from 407420e to 719afe5 Compare May 28, 2024 13:41

Added system_lib for system prestage jars

bdbbfab

- Prestage system test prequisities to system_lib related: adoptium#4912 Signed-off-by: Anna Babu Palathingal <anna.bp@ibm.com>

annaibm force-pushed the test_systemtest branch from 719afe5 to bdbbfab Compare May 28, 2024 13:53

annaibm requested a review from LongyuZhang May 28, 2024 15:36

llxia approved these changes May 29, 2024

View reviewed changes

annaibm marked this pull request as ready for review May 29, 2024 19:10

LongyuZhang approved these changes May 29, 2024

View reviewed changes

LongyuZhang merged commit a4ef8f2 into adoptium:master May 29, 2024
2 checks passed

llxia added a commit that referenced this pull request May 31, 2024

Revert "Added system_lib for system prestage jars (#5239)"

0ee113d

This reverts commit a4ef8f2.

llxia mentioned this pull request May 31, 2024

Revert "Added system_lib for system prestage jars" #5361

Merged

JasonFengJ9 pushed a commit that referenced this pull request May 31, 2024

Revert "Added system_lib for system prestage jars (#5239)" (#5361)

071c84b

This reverts commit a4ef8f2.

This was referenced Jun 10, 2024

Include Java version on generating classes jar file to resolve UnsupportedClassVersion adoptium/openj9-systemtest#155

Closed

Add classes.jar for sharedClassesTestData into specific jdk version dir. adoptium/openj9-systemtest#156

Merged

annaibm added a commit to annaibm/aqa-tests that referenced this pull request Jun 18, 2024

Revert "Revert "Added system_lib for system prestage jars (adoptium#5239

80659cc

)" (adoptium#5361)" This reverts commit 071c84b.

annaibm mentioned this pull request Jun 20, 2024

Added system_lib for system prestage jars #5400

Merged

annaibm deleted the test_systemtest branch July 25, 2024 15:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added system_lib for system prestage jars #5239

Added system_lib for system prestage jars #5239

annaibm commented Apr 18, 2024 •

edited

Loading

annaibm commented Apr 18, 2024

llxia commented Apr 18, 2024

llxia commented Apr 18, 2024

llxia May 10, 2024

llxia May 10, 2024

annaibm May 13, 2024

llxia May 10, 2024

annaibm May 13, 2024

llxia May 22, 2024

annaibm May 24, 2024

annaibm May 24, 2024

llxia May 10, 2024

annaibm commented May 21, 2024 •

edited

Loading

annaibm commented May 24, 2024

llxia commented May 27, 2024

annaibm commented May 28, 2024 •

edited

Loading

llxia commented May 29, 2024

llxia left a comment

LongyuZhang left a comment

Added system_lib for system prestage jars #5239

Added system_lib for system prestage jars #5239

Conversation

annaibm commented Apr 18, 2024 • edited Loading

annaibm commented Apr 18, 2024

llxia commented Apr 18, 2024

llxia commented Apr 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annaibm commented May 21, 2024 • edited Loading

annaibm commented May 24, 2024

llxia commented May 27, 2024

annaibm commented May 28, 2024 • edited Loading

llxia commented May 29, 2024

llxia left a comment

Choose a reason for hiding this comment

LongyuZhang left a comment

Choose a reason for hiding this comment

annaibm commented Apr 18, 2024 •

edited

Loading

annaibm commented May 21, 2024 •

edited

Loading

annaibm commented May 28, 2024 •

edited

Loading