Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FISH-1018] Out of memory redeploy leaks #5081

Merged
merged 7 commits into from
Jan 26, 2021

Conversation

sgflt
Copy link
Contributor

@sgflt sgflt commented Dec 31, 2020

Fixes #4098

Description

There were multiple causes:

  1. fields in services
  2. nested classes created in services that holds implicit reference to them

Important Info

Blockers

Testing

New tests

Testing Performed

Testing Environment

Documentation

Notes for Reviewers

Leak in Globals I left untouched.

- this could be the first cause of leak
…lasses

- anonymous and inner holds implicit reference to parent
- this was probably the second cause of leak
@lprimak
Copy link
Contributor

lprimak commented Jan 1, 2021

jenkins test

@lprimak lprimak self-requested a review January 1, 2021 19:15
@lprimak lprimak self-assigned this Jan 1, 2021
@lprimak lprimak added PR: CLA CLA submitted on PR by the contributor Type: Bug Label issue as a bug defect Type: Community Contribution labels Jan 1, 2021
@dmatej dmatej self-requested a review January 14, 2021 21:26
@AlanRoth AlanRoth removed the Type: Bug Label issue as a bug defect label Jan 18, 2021
@lprimak
Copy link
Contributor

lprimak commented Jan 20, 2021

Looks good. Will test it with class loader leak tester (in appserver/tests/payara-samples) to see if it works

@lprimak
Copy link
Contributor

lprimak commented Jan 20, 2021

Unfortunately, leak tester does not pass, more work needs to be done to determine additional classloader leaks

@lprimak
Copy link
Contributor

lprimak commented Jan 20, 2021

lprimak@Lennys-MacBook-Pro classloader-data-api % curl http://localhost:8080/ClassloaderDataAPI/api/instance-count/webapp/
Instances Before GC: 7
Instances Remaining: 7
Instances: 
WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/jsp/ex/loader_1996132605/; Object: 76fa90fd Created: Jan 20, 2021 1:06:16 PM

WebappClassLoader (delegate=true; repositoryURLs=file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/__default-web-module/) JarNames: []; canonicalLoaderDir: null; Object: 740d468e Created: Jan 20, 2021 1:05:36 PM

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/jsp/ex/loader_754200651/; Object: 2cf4304b Created: Jan 20, 2021 1:06:53 PM

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/jsp/ex/loader_1601764582/; Object: 5f78fce6 Created: Jan 20, 2021 1:07:28 PM

WebappClassLoader (delegate=true; repositoryURLs=file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/commons-lang3-3.11.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/ex/,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/primefaces-8.0.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/slf4j-jdk14-1.7.30.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/flowlogix-jee-4.0.1-SNAPSHOT.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/slf4j-api-1.7.30.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/classes/,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/omniutils-0.11.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/omnifaces-3.9.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/omnipersistence-0.15.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/optimusfaces-0.13.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ex/WEB-INF/lib/flowlogix-datamodel-4.0.1-SNAPSHOT.jar,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/ejb/ex/) JarNames: []; canonicalLoaderDir: /Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/jsp/ex/loader_832313690/; Object: 319c195a Created: Jan 20, 2021 1:08:56 PM

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/jsp/ex/loader_1514302499/; Object: 5a426c23 Created: Jan 20, 2021 1:06:40 PM

WebappClassLoader (delegate=true; repositoryURLs=file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/ClassloaderDataAPI/,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/applications/ClassloaderDataAPI/WEB-INF/classes/,file:/Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/ejb/ClassloaderDataAPI/) JarNames: []; canonicalLoaderDir: /Users/lprimak/Documents/dev/Payara/Payara/appserver/distributions/payara/target/stage/payara5/glassfish/domains/domain1/generated/jsp/ClassloaderDataAPI/loader_630168213/; Object: 258f9a95 Created: Jan 20, 2021 1:05:37 PM

@lprimak lprimak added the PR: DO NOT MERGE Don't merge PR until further notice label Jan 20, 2021
@sgflt sgflt marked this pull request as draft January 21, 2021 15:24
@sgflt sgflt force-pushed the 4098-out-of-memory-error-redeploy-leaks branch from ed74d0e to 99ed6c8 Compare January 21, 2021 16:08
@sgflt
Copy link
Contributor Author

sgflt commented Jan 21, 2021

Tested and before fix count is rising very rapidly. After fix count of classloaders is flipping between 7 and 10.

Before:

Instances Before GC: 16
Instances Remaining: 16
Instances: 
WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/classes/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-io-2.6.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/lib/console-core-5.2020.8-SNAPSHOT.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-suntheme-4.0.2.10.payara-p21.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-fileupload-1.3.3.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-4.0.2.10.payara-p17.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/json-1.0.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/__admingui//,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/dojo-ajax-nodemo-0.4.4.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/__admingui/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/prototype-1.7.3.jar) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_1733152244/; Object: 674dcdf4 Created: 21.1.2021 17:10:49

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_864480257/; Object: 3386ec01 Created: 21.1.2021 17:11:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_321440822/; Object: 1328cc36 Created: 21.1.2021 17:11:26

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/ClassloaderDataAPI/,file:/tmp/payara5/glassfish/domains/domain1/applications/ClassloaderDataAPI/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/ClassloaderDataAPI/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/ClassloaderDataAPI/loader_1956231135/; Object: 7499b7df Created: 21.1.2021 17:11:05

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_488364938/; Object: 1d1bdb8a Created: 21.1.2021 17:11:24

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/login_war/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/login_war/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1020822604/; Object: 3cd8844c Created: 21.1.2021 17:11:29

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/bug_war/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/bug_war/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_580776718/; Object: 229df30e Created: 21.1.2021 17:11:29

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_2009317906/; Object: 77c3c212 Created: 21.1.2021 17:11:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_436409390/; Object: 1a03142e Created: 21.1.2021 17:11:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1823407660/; Object: 6caefe2c Created: 21.1.2021 17:11:16

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/__default-web-module/) JarNames: []; canonicalLoaderDir: null; Object: 34990e44 Created: 21.1.2021 17:11:28

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_237453130/; Object: e273f4a Created: 21.1.2021 17:11:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_384894071/; Object: 16f10477 Created: 21.1.2021 17:10:47

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/static_war/,file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/static_war/WEB-INF/classes) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1980162286/; Object: 7606e0ee Created: 21.1.2021 17:11:29

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1651138589/; Object: 626a601d Created: 21.1.2021 17:11:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1741147451/; Object: 67c7cd3b Created: 21.1.2021 17:11:26
Instances Before GC: 103
Instances Remaining: 139
Instances: 
WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1816093920/; Object: 6c3f64e0 Created: 21.1.2021 17:11:44

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_944980672/; Object: 385342c0 Created: 21.1.2021 17:12:51

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1603744317/; Object: 5f97323d Created: 21.1.2021 17:12:22

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_976443198/; Object: 3a33573e Created: 21.1.2021 17:11:38

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/ClassloaderDataAPI/,file:/tmp/payara5/glassfish/domains/domain1/applications/ClassloaderDataAPI/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/ClassloaderDataAPI/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/ClassloaderDataAPI/loader_1956231135/; Object: 7499b7df Created: 21.1.2021 17:11:05

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_2009317906/; Object: 77c3c212 Created: 21.1.2021 17:11:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_479627349/; Object: 1c968855 Created: 21.1.2021 17:11:58

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_311620837/; Object: 1292f4e5 Created: 21.1.2021 17:11:48

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1489942636/; Object: 58ceb86c Created: 21.1.2021 17:12:18

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1730566004/; Object: 67265774 Created: 21.1.2021 17:11:29

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_237453130/; Object: e273f4a Created: 21.1.2021 17:11:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_384894071/; Object: 16f10477 Created: 21.1.2021 17:10:47

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1205967797/; Object: 47e19bb5 Created: 21.1.2021 17:11:58

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_473903865/; Object: 1c3f32f9 Created: 21.1.2021 17:12:32

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1779651926/; Object: 6a135556 Created: 21.1.2021 17:12:40

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1351601930/; Object: 508fcf0a Created: 21.1.2021 17:12:12

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_991123946/; Object: 3b1359ea Created: 21.1.2021 17:12:50

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_2015182400/; Object: 781d3e40 Created: 21.1.2021 17:12:22

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1580515796/; Object: 5e34c1d4 Created: 21.1.2021 17:12:51

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_66123364/; Object: 3f0f664 Created: 21.1.2021 17:12:43

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_369235488/; Object: 16021620 Created: 21.1.2021 17:12:47

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1165378232/; Object: 457642b8 Created: 21.1.2021 17:12:45

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1135155195/; Object: 43a917fb Created: 21.1.2021 17:12:06

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_737341595/; Object: 2bf2f09b Created: 21.1.2021 17:11:48

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_399343315/; Object: 17cd7ed3 Created: 21.1.2021 17:12:32

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1823658695/; Object: 6cb2d2c7 Created: 21.1.2021 17:11:54

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1183745413/; Object: 468e8585 Created: 21.1.2021 17:12:36

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1260760567/; Object: 4b25adf7 Created: 21.1.2021 17:12:08

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_371367980/; Object: 1622a02c Created: 21.1.2021 17:11:56

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1749695413/; Object: 684a3bb5 Created: 21.1.2021 17:12:43

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_721359236/; Object: 2aff1184 Created: 21.1.2021 17:12:22

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_136481902/; Object: 8228c6e Created: 21.1.2021 17:12:04

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1531327885/; Object: 5b46358d Created: 21.1.2021 17:12:47

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_826323778/; Object: 3140b342 Created: 21.1.2021 17:11:56

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1225542472/; Object: 490c4b48 Created: 21.1.2021 17:12:28

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_347418571/; Object: 14b52fcb Created: 21.1.2021 17:11:36

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1900599692/; Object: 7148d98c Created: 21.1.2021 17:11:38

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_819019074/; Object: 30d13d42 Created: 21.1.2021 17:11:54

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1019306601/; Object: 3cc16269 Created: 21.1.2021 17:12:36

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_197240168/; Object: bc1a568 Created: 21.1.2021 17:12:12

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_847463091/; Object: 328342b3 Created: 21.1.2021 17:12:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_4039209/; Object: 3da229 Created: 21.1.2021 17:12:02

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_62411647/; Object: 3b8537f Created: 21.1.2021 17:11:50

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1932092086/; Object: 732962b6 Created: 21.1.2021 17:12:42

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1872911289/; Object: 6fa25bb9 Created: 21.1.2021 17:12:42

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1969252779/; Object: 756069ab Created: 21.1.2021 17:11:50

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1204776757/; Object: 47cf6f35 Created: 21.1.2021 17:11:42

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_765446205/; Object: 2d9fc83d Created: 21.1.2021 17:11:40

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1088710120/; Object: 40e465e8 Created: 21.1.2021 17:12:45

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_421985029/; Object: 1926fb05 Created: 21.1.2021 17:12:08

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_233055256/; Object: de42418 Created: 21.1.2021 17:12:10

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1813999199/; Object: 6c1f6e5f Created: 21.1.2021 17:11:52

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_2017024730/; Object: 78395ada Created: 21.1.2021 17:12:30

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1747490219/; Object: 682895ab Created: 21.1.2021 17:12:50

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1062448860/; Object: 3f53aedc Created: 21.1.2021 17:12:12

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_391640633/; Object: 1757f639 Created: 21.1.2021 17:11:29

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1922620574/; Object: 7298dc9e Created: 21.1.2021 17:11:29

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_924893871/; Object: 3720c2af Created: 21.1.2021 17:12:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1616979791/; Object: 6061274f Created: 21.1.2021 17:12:14

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1840318496/; Object: 6db10820 Created: 21.1.2021 17:12:00

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_2140381901/; Object: 7f93a2cd Created: 21.1.2021 17:12:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1512791267/; Object: 5a2b5ce3 Created: 21.1.2021 17:11:52

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_545725692/; Object: 20871cfc Created: 21.1.2021 17:12:04

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1100838359/; Object: 419d75d7 Created: 21.1.2021 17:12:32

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_380254624/; Object: 16aa39a0 Created: 21.1.2021 17:11:44

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/__default-web-module/) JarNames: []; canonicalLoaderDir: null; Object: 70ec8e92 Created: 21.1.2021 17:12:53

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_864480257/; Object: 3386ec01 Created: 21.1.2021 17:11:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_321440822/; Object: 1328cc36 Created: 21.1.2021 17:11:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_488364938/; Object: 1d1bdb8a Created: 21.1.2021 17:11:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_280831438/; Object: 10bd25ce Created: 21.1.2021 17:12:30

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1828378912/; Object: 6cfad920 Created: 21.1.2021 17:12:28

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1473260212/; Object: 57d02ab4 Created: 21.1.2021 17:11:54

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_212137005/; Object: ca4f42d Created: 21.1.2021 17:11:44

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_370414824/; Object: 161414e8 Created: 21.1.2021 17:11:48

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1713988140/; Object: 6629622c Created: 21.1.2021 17:12:06

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_764132210/; Object: 2d8bbb72 Created: 21.1.2021 17:12:18

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1651138589/; Object: 626a601d Created: 21.1.2021 17:11:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_53717700/; Object: 333aac4 Created: 21.1.2021 17:11:36

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1603995254/; Object: 5f9b0676 Created: 21.1.2021 17:12:20

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_705689437/; Object: 2a0ff75d Created: 21.1.2021 17:12:00

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_950684617/; Object: 38aa4bc9 Created: 21.1.2021 17:12:42

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_435050648/; Object: 19ee5898 Created: 21.1.2021 17:12:43

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_254779784/; Object: f2fa188 Created: 21.1.2021 17:12:40

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1059757507/; Object: 3f2a9dc3 Created: 21.1.2021 17:12:38

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_200557242/; Object: bf442ba Created: 21.1.2021 17:12:38

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_669955481/; Object: 27eeb599 Created: 21.1.2021 17:12:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1823407660/; Object: 6caefe2c Created: 21.1.2021 17:11:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_680965460/; Object: 2896b554 Created: 21.1.2021 17:12:28

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1378725833/; Object: 522dafc9 Created: 21.1.2021 17:11:31

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1911475108/; Object: 71eecba4 Created: 21.1.2021 17:12:36

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1179032960/; Object: 46469d80 Created: 21.1.2021 17:11:40

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1767245743/; Object: 695607af Created: 21.1.2021 17:11:56

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_626897248/; Object: 255db160 Created: 21.1.2021 17:12:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_852486318/; Object: 32cfe8ae Created: 21.1.2021 17:12:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_142090626/; Object: 8782182 Created: 21.1.2021 17:12:38

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1106112379/; Object: 41edef7b Created: 21.1.2021 17:12:18

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_932590042/; Object: 379631da Created: 21.1.2021 17:12:02

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1955985154/; Object: 7495f702 Created: 21.1.2021 17:11:31

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1741147451/; Object: 67c7cd3b Created: 21.1.2021 17:11:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1195744567/; Object: 47459d37 Created: 21.1.2021 17:12:08

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_248083058/; Object: ec97272 Created: 21.1.2021 17:11:52

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_45466761/; Object: 2b5c489 Created: 21.1.2021 17:12:30

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1955047059/; Object: 7487a693 Created: 21.1.2021 17:11:31

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/classes/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-io-2.6.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/lib/console-core-5.2020.8-SNAPSHOT.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-suntheme-4.0.2.10.payara-p21.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-fileupload-1.3.3.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-4.0.2.10.payara-p17.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/json-1.0.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/__admingui//,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/dojo-ajax-nodemo-0.4.4.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/__admingui/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/prototype-1.7.3.jar) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_1733152244/; Object: 674dcdf4 Created: 21.1.2021 17:10:49

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_12861201/; Object: c43f11 Created: 21.1.2021 17:11:36

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_144582091/; Object: 89e25cb Created: 21.1.2021 17:11:38

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_2062358744/; Object: 7aed18d8 Created: 21.1.2021 17:11:46

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1184090270/; Object: 4693c89e Created: 21.1.2021 17:12:14

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_749107814/; Object: 2ca67a66 Created: 21.1.2021 17:12:50

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_936044478/; Object: 37cae7be Created: 21.1.2021 17:12:02

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1701614533/; Object: 656c93c5 Created: 21.1.2021 17:12:47

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_275825319/; Object: 1070c2a7 Created: 21.1.2021 17:12:16

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1652344980/; Object: 627cc894 Created: 21.1.2021 17:12:04

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_436409390/; Object: 1a03142e Created: 21.1.2021 17:11:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1316176315/; Object: 4e7341bb Created: 21.1.2021 17:11:58

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_2003802433/; Object: 776f9941 Created: 21.1.2021 17:12:20

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1245858136/; Object: 4a424958 Created: 21.1.2021 17:11:50

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_34794923/; Object: 212edab Created: 21.1.2021 17:12:10

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1100750390/; Object: 419c1e36 Created: 21.1.2021 17:12:10

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_97865183/; Object: 5d54ddf Created: 21.1.2021 17:11:34

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_594583471/; Object: 23709faf Created: 21.1.2021 17:12:26

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1383767951/; Object: 527a9f8f Created: 21.1.2021 17:12:34

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_505749059/; Object: 1e251e43 Created: 21.1.2021 17:11:42

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1261552839/; Object: 4b31c4c7 Created: 21.1.2021 17:11:34

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1601283506/; Object: 5f71a5b2 Created: 21.1.2021 17:12:51

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_547383058/; Object: 20a06712 Created: 21.1.2021 17:12:24

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_192706105/; Object: b7c7639 Created: 21.1.2021 17:12:06

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1057308288/; Object: 3f053e80 Created: 21.1.2021 17:12:34

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_280728547/; Object: 10bb93e3 Created: 21.1.2021 17:11:46

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1704678692/; Object: 659b5524 Created: 21.1.2021 17:11:42

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1115072197/; Object: 4276a6c5 Created: 21.1.2021 17:12:20

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_435411827/; Object: 19f3db73 Created: 21.1.2021 17:12:34

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_975106637/; Object: 3a1ef24d Created: 21.1.2021 17:11:46

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_939671939/; Object: 38024183 Created: 21.1.2021 17:12:40

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_391588543/; Object: 17572abf Created: 21.1.2021 17:12:14

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_921854904/; Object: 36f263b8 Created: 21.1.2021 17:12:00

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1501227342/; Object: 597ae94e Created: 21.1.2021 17:11:34

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_38010877/; Object: 243fffd Created: 21.1.2021 17:12:45

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1796913560/; Object: 6b1ab998 Created: 21.1.2021 17:11:40

After:

Instances Before GC: 10
Instances Remaining: 7
Instances: 
WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/ClassloaderDataAPI/,file:/tmp/payara5/glassfish/domains/domain1/applications/ClassloaderDataAPI/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/ClassloaderDataAPI/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/ClassloaderDataAPI/loader_1944705915/; Object: 73e9db7b Created: 21.1.2021 16:58:11

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/__default-web-module/) JarNames: []; canonicalLoaderDir: null; Object: 71a1afc2 Created: 21.1.2021 16:59:07

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/login_war/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/test-case-0/login_war/,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/login_war/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_963195331/; Object: 396931c3 Created: 21.1.2021 16:59:07

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/classes/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-io-2.6.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/lib/console-core-5.2020.8-SNAPSHOT.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-suntheme-4.0.2.10.payara-p21.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-fileupload-1.3.3.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-4.0.2.10.payara-p17.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/json-1.0.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/__admingui//,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/dojo-ajax-nodemo-0.4.4.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/__admingui/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/prototype-1.7.3.jar) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_2123957121/; Object: 7e990381 Created: 21.1.2021 16:55:12

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/bug_war/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/test-case-0/bug_war/,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/bug_war/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_615725453/; Object: 24b3398d Created: 21.1.2021 16:59:07

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/static_war/,file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/static_war/WEB-INF/classes,file:/tmp/payara5/glassfish/domains/domain1/generated/test-case-0/static_war/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_1104489754/; Object: 41d52d1a Created: 21.1.2021 16:59:07

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_1145296226/; Object: 4443d562 Created: 21.1.2021 16:55:11
Instances Before GC: 7
Instances Remaining: 10
Instances: 
WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1742230310/; Object: 67d85326 Created: 21.1.2021 17:01:33

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/test-case-0/static_war/,file:/tmp/payara5/glassfish/domains/domain1/applications/test-case-0/static_war/WEB-INF/classes,file:/tmp/payara5/glassfish/domains/domain1/generated/test-case-0/static_war/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_966787083/; Object: 39a0000b Created: 21.1.2021 17:01:33

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/static_war/loader_826612631/; Object: 31451b97 Created: 21.1.2021 17:01:31

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_1899660529/; Object: 713a84f1 Created: 21.1.2021 17:01:31

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/login_war/loader_1632037478/; Object: 6146ea66 Created: 21.1.2021 17:01:31

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/test-case-0/bug_war/loader_2011136483/; Object: 77df81e3 Created: 21.1.2021 17:01:33

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/ClassloaderDataAPI/,file:/tmp/payara5/glassfish/domains/domain1/applications/ClassloaderDataAPI/WEB-INF/classes/,file:/tmp/payara5/glassfish/domains/domain1/generated/ClassloaderDataAPI/) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/ClassloaderDataAPI/loader_1944705915/; Object: 73e9db7b Created: 21.1.2021 16:58:11

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/domains/domain1/generated/__default-web-module/) JarNames: []; canonicalLoaderDir: null; Object: 3734aee Created: 21.1.2021 17:01:32

WebappClassLoader (delegate=true; repositoryURLs=file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/classes/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-io-2.6.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/lib/console-core-5.2020.8-SNAPSHOT.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-suntheme-4.0.2.10.payara-p21.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/commons-fileupload-1.3.3.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/webui-jsf-4.0.2.10.payara-p17.jar,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/json-1.0.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/ejb/__admingui//,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/dojo-ajax-nodemo-0.4.4.jar,file:/tmp/payara5/glassfish/domains/domain1/generated/__admingui/,file:/tmp/payara5/glassfish/lib/install/applications/__admingui/WEB-INF/extra/prototype-1.7.3.jar) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_2123957121/; Object: 7e990381 Created: 21.1.2021 16:55:12

WebappClassLoader (delegate=true) JarNames: []; canonicalLoaderDir: /tmp/payara5/glassfish/domains/domain1/generated/jsp/__admingui/loader_1145296226/; Object: 4443d562 Created: 21.1.2021 16:55:11

@sgflt sgflt marked this pull request as ready for review January 21, 2021 16:13
@sgflt sgflt marked this pull request as draft January 21, 2021 16:21
@sgflt sgflt marked this pull request as ready for review January 21, 2021 17:43
@sgflt
Copy link
Contributor Author

sgflt commented Jan 21, 2021

I have double checked results and I can't see any additional classloader leak.

@lprimak
Copy link
Contributor

lprimak commented Jan 21, 2021

I think the issue is in anonymous inner classes that are part of the WebappClassLoader, I have not had a chance to look any deeper in to this but I think that's the main issue

@lprimak lprimak changed the title Fixes 4098 out of memory redeploy leaks [FISH-1018] Out of memory redeploy leaks Jan 24, 2021
- making sure Server Threads and Timers do not inherit app's context class loaders
- making sure app's security contexts don't get propagated to server threads and timers

Added correct ear classes to class loader leak tests
@lprimak lprimak removed the PR: DO NOT MERGE Don't merge PR until further notice label Jan 26, 2021
@lprimak
Copy link
Contributor

lprimak commented Jan 26, 2021

jenkins test

@lprimak lprimak merged commit bb96e1d into payara:master Jan 26, 2021
@sgflt sgflt deleted the 4098-out-of-memory-error-redeploy-leaks branch January 26, 2021 06:07
lprimak added a commit that referenced this pull request Feb 5, 2021
* [FISH-1018] Out of memory redeploy leaks (#5081)

* #4098 Reduced too broad scope of variable

* #4098 Moved field to local variable

- this could be the first cause of leak

* #4098 Refactored inner and anonymous classes to nested static classes

- anonymous and inner holds implicit reference to parent
- this was probably the second cause of leak

* #4098 Fixed code consistency

* Fix more class loader leaks by:
- making sure Server Threads and Timers do not inherit app's context class loaders
- making sure app's security contexts don't get propagated to server threads and timers

Added correct ear classes to class loader leak tests

Co-authored-by: lprimak <lenny@flowlogix.com>

* [FISH-1018] found more leaks and more reliable leak test (#5102)

* found more leaks and more reliable leak test

* bump jakarta.el to -p3 patch

* tyrus patched update

Co-authored-by: Lukáš Kvídera <kvideral@qwsome.eu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: CLA CLA submitted on PR by the contributor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Out of memory after number of redeployments
3 participants