pre-stage test libs on the machine #4902

llxia · 2023-12-05T16:18:29Z

test libs will be staged under ${WORKSPACE}/../../testDependency/lib on the machine if it does not exist

resolves: #4500

llxia · 2023-12-05T16:57:39Z

first run on a machine that does not pre-staged test libs (Grinder)
a. stage all test libs (30secs)

00:00:56.011  + perl ./aqa-tests/TKG/scripts/getDependencies.pl -path /home/jenkins/workspace/Grinder/../../testDependency/lib -task default -customUrl https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/
00:00:56.011  /home/jenkins/workspace/Grinder/../../testDependency/lib does not exist, creating one.
00:00:56.011  mkdir /home/jenkins/workspace/Grinder/../../testDependency
00:00:56.011  mkdir /home/jenkins/workspace/Grinder/../../testDependency/lib
00:00:56.011  --------------------------------------------
00:00:56.011  path is set to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:00:56.011  task is set to default
00:00:56.011  dependencyList is set to all
00:00:56.011  --------------------------------------------
00:00:56.011  Starting download third party dependent jars
00:00:56.011  --------------------------------------------
00:00:56.011  downloading dependent third party jars to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:00:56.011  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:56.430  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:56.430  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//openj9jtregtimeouthandler.jar
00:00:56.803  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar
00:00:56.803  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:57.610  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar.sha256sum.txt
00:00:57.610  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//jaxb-api.jar
00:00:58.414  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/jaxb-api.jar
00:00:58.414  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//asm-all.jar
00:00:59.485  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/asm-all.jar
...
00:01:24.801  downloading https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact//commons-exec.jar
00:01:25.621  --> file downloaded to /home/jenkins/workspace/Grinder/../../testDependency/lib/commons-exec.jar
00:01:25.621  downloaded dependent third party jars successfully

b. during test compilation, only re-download the jar if SHA does not match. Otherwise, do nothing.

00:01:41.188  getDependentLibs:
00:01:41.188       [exec] --------------------------------------------
00:01:41.188       [exec] path is set to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:01:41.188       [exec] task is set to default
00:01:41.188       [exec] dependencyList is set to junit4
00:01:41.188       [exec] --------------------------------------------
00:01:41.188       [exec] Starting download third party dependent jars
00:01:41.188       [exec] --------------------------------------------
00:01:41.188       [exec] downloading dependent third party jars to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:01:41.188       [exec] /home/jenkins/workspace/Grinder/../../testDependency/lib/junit4.jar exists with correct hash, not downloading
00:01:41.188       [exec] downloaded dependent third party jars successfully

second run on the same machine (pre-stage test libs exist, no downloads) (Grinder) (less than 1sec)

00:07:22.073  + perl ./aqa-tests/TKG/scripts/getDependencies.pl -path /home/jenkins/workspace/Grinder/../../testDependency/lib -task default -customUrl https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/
00:07:22.880  --------------------------------------------
00:07:22.880  path is set to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:07:22.880  task is set to default
00:07:22.880  dependencyList is set to all
00:07:22.880  --------------------------------------------
00:07:22.880  Starting download third party dependent jars
00:07:22.880  --------------------------------------------
00:07:22.880  downloading dependent third party jars to /home/jenkins/workspace/Grinder/../../testDependency/lib
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/openj9jtregtimeouthandler.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jaxb-api.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/asm-all.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jcommander.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jtreg_6_1.tar.gz exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/org.eclipse.osgi-3.16.100.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/junit4.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jtreg_7_3_1_1.tar.gz exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/json-simple.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jcstress-tests-all-20220908.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jtreg_5_1_b01.tar.gz exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/testng.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/asm.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/asmtools.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/commons-cli.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/jython-standalone.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/javassist.jar exists with correct hash, not downloading
00:07:22.880  /home/jenkins/workspace/Grinder/../../testDependency/lib/commons-exec.jar exists with correct hash, not downloading
00:07:22.880  downloaded dependent third party jars successfully

llxia · 2023-12-05T20:36:43Z

jdk_math xlinux: Grinder
floatSanityTests windows: Grinder
floatSanityTests zos: Grinder

llxia · 2023-12-06T15:54:12Z

FYI @AdamBrousseau @pshipton

LongyuZhang

LGTM

smlambert

Minor typo in comment, but otherwise LGTM.

smlambert · 2023-12-06T18:54:06Z

buildenv/jenkins/JenkinsfileBase

-					}
+			// check pre-stage test libs on the machine
+			// check for each lib. If lib does not exist, donwload it.
+			// If lib exists, SHA will be check. Redlownload if SHA does not match.


Suggested change

// If lib exists, SHA will be check. Redlownload if SHA does not match.

// If lib exists, SHA will be checked. Re-download if SHA does not match.

updated. Thanks

Signed-off-by: Lan Xia <Lan_Xia@ca.ibm.com>

llxia mentioned this pull request Dec 5, 2023

Allow LIB_DIR, customUrl and curlOpts to be set for get dependency logic adoptium/TKG#476

Merged

llxia force-pushed the mytest branch 7 times, most recently from 29cc92e to 667d7d2 Compare December 6, 2023 14:49

llxia requested review from smlambert and LongyuZhang December 6, 2023 15:53

llxia marked this pull request as ready for review December 6, 2023 15:53

LongyuZhang approved these changes Dec 6, 2023

View reviewed changes

smlambert approved these changes Dec 6, 2023

View reviewed changes

pre-stage test libs on the machine

0e5d734

Signed-off-by: Lan Xia <Lan_Xia@ca.ibm.com>

llxia force-pushed the mytest branch from 667d7d2 to 0e5d734 Compare December 6, 2023 19:04

smlambert merged commit 0f75b36 into adoptium:master Dec 6, 2023
1 check passed

This was referenced Dec 7, 2023

TKG not compiling on Windows - error: package org.json.simple adoptium/TKG#478

Closed

pre-stage test libs on the machine for parallel case #4908

Merged

smlambert mentioned this pull request Mar 5, 2024

AQAvit Community Call (March 5, 2024 edition) #5090

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pre-stage test libs on the machine #4902

pre-stage test libs on the machine #4902

llxia commented Dec 5, 2023 •

edited

Loading

llxia commented Dec 5, 2023 •

edited

Loading

llxia commented Dec 5, 2023 •

edited

Loading

llxia commented Dec 6, 2023

LongyuZhang left a comment

smlambert left a comment

smlambert Dec 6, 2023

llxia Dec 6, 2023

	// If lib exists, SHA will be check. Redlownload if SHA does not match.
	// If lib exists, SHA will be checked. Re-download if SHA does not match.

pre-stage test libs on the machine #4902

pre-stage test libs on the machine #4902

Conversation

llxia commented Dec 5, 2023 • edited Loading

llxia commented Dec 5, 2023 • edited Loading

llxia commented Dec 5, 2023 • edited Loading

llxia commented Dec 6, 2023

LongyuZhang left a comment

Choose a reason for hiding this comment

smlambert left a comment

Choose a reason for hiding this comment

smlambert Dec 6, 2023

Choose a reason for hiding this comment

llxia Dec 6, 2023

Choose a reason for hiding this comment

llxia commented Dec 5, 2023 •

edited

Loading

llxia commented Dec 5, 2023 •

edited

Loading

llxia commented Dec 5, 2023 •

edited

Loading