java.lang.OutOfMemoryError: GC overhead limit exceeded

[lmaylein]

For the second time within 24 hours our dataverse application has stopped with the following log messages. Does this mean I have to adjust the jvm-options?

[2018-07-26T22:15:05.502+0200] [glassfish 4.1] [WARNING] [jsf.application.resource.unable_to_serve_from_library] [javax.enterprise.resource.webcontainer.jsf.application] [tid: _ThreadID=50 _ThreadName=jk-connector(4)] [timeMillis: 1532636105502] [levelValue: 900] [[
  JSF1064: Unable to find or serve resource, primefaces.js, from library, primefaces.]]

[2018-07-26T22:15:05.503+0200] [glassfish 4.1] [WARNING] [] [javax.enterprise.resource.webcontainer.jsf.application] [tid: _ThreadID=50 _ThreadName=jk-connector(4)] [timeMillis: 1532636105503] [levelValue: 900] [[
  
java.io.IOException
]]

[2018-07-26T22:15:13.567+0200] [glassfish 4.1] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=50 _ThreadName=jk-connector(4)] [timeMillis: 1532636113567] [levelValue: 900] [[
  StandardWrapperValve[Faces Servlet]: Servlet.service() for servlet Faces Servlet threw exception
java.io.IOException
]]

[2018-07-26T22:15:39.402+0200] [glassfish 4.1] [SEVERE] [jsf.cannot_instantiate_component_error] [javax.enterprise.resource.webcontainer.jsf.application] [tid: _ThreadID=51 _ThreadName=jk-connector(5)] [timeMillis: 1532636139402] [levelValue: 1000] [[
  JSF1068: Cannot instantiate component with component-type javax.faces.Parameter]]

[2018-07-26T22:15:39.404+0200] [glassfish 4.1] [WARNING] [] [javax.enterprise.web.core] [tid: _ThreadID=51 _ThreadName=jk-connector(5)] [timeMillis: 1532636139404] [levelValue: 900] [[
  Servlet.service() for servlet Faces Servlet threw exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.HashSet.<init>(HashSet.java:106)
        at org.jboss.weld.injection.producer.InterceptionModelInitializer.mergeMethodInterceptorBindings(InterceptionModelInitializer.java:303)
        at org.jboss.weld.injection.producer.InterceptionModelInitializer.getMemberBindingAnnotations(InterceptionModelInitializer.java:209)
        at org.jboss.weld.injection.producer.InterceptionModelInitializer.initCdiBusinessMethodInterceptors(InterceptionModelInitializer.java:172)
        at org.jboss.weld.injection.producer.InterceptionModelInitializer.initCdiInterceptors(InterceptionModelInitializer.java:137)
        at org.jboss.weld.injection.producer.InterceptionModelInitializer.init(InterceptionModelInitializer.java:104)
        at org.jboss.weld.injection.producer.BeanInjectionTarget.buildInterceptionModel(BeanInjectionTarget.java:93)
        at org.jboss.weld.injection.producer.BeanInjectionTarget.initializeInterceptionModel(BeanInjectionTarget.java:88)
        at org.jboss.weld.injection.producer.BeanInjectionTarget.initializeAfterBeanDiscovery(BeanInjectionTarget.java:98)
        at org.jboss.weld.injection.producer.InjectionTargetInitializationContext.initialize(InjectionTargetInitializationContext.java:42)
        at org.jboss.weld.injection.producer.InjectionTargetService.addInjectionTargetToBeInitialized(InjectionTargetService.java:55)
        at org.jboss.weld.injection.producer.InjectionTargetService.addInjectionTargetToBeInitialized(InjectionTargetService.java:49)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.initialize(InjectionTargetFactoryImpl.java:145)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:92)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:80)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:70)
        at org.jboss.weld.manager.BeanManagerImpl.createInjectionTarget(BeanManagerImpl.java:1137)
        at org.glassfish.weld.services.JCDIServiceImpl.injectManagedObject(JCDIServiceImpl.java:283)
        at org.glassfish.faces.integration.GlassFishInjectionProvider.inject(GlassFishInjectionProvider.java:189)
        at com.sun.faces.application.ApplicationImpl.newThing(ApplicationImpl.java:1759)
        at com.sun.faces.application.ApplicationImpl.createComponentApplyAnnotations(ApplicationImpl.java:1917)
        at com.sun.faces.application.ApplicationImpl.createComponent(ApplicationImpl.java:1168)
        at javax.faces.application.ApplicationWrapper.createComponent(ApplicationWrapper.java:637)
        at javax.faces.application.ApplicationWrapper.createComponent(ApplicationWrapper.java:637)
        at com.sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.createComponent(ComponentTagHandlerDelegateImpl.java:584)
        at com.sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:176)
        at javax.faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
        at javax.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:95)
        at javax.faces.view.facelets.DelegatingMetaTagHandler.applyNextHandler(DelegatingMetaTagHandler.java:137)
        at com.sun.faces.facelets.tag.jsf.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:203)
        at javax.faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:120)
        at javax.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:95)
]]

[pdurbin]

Anyone trying to troubleshoot this should look at the Glassfish logs provided by @lmaylein in https://help.hmdc.harvard.edu/Ticket/Display.html?id=264966

@donsizemore my first thought is that you and @akio-sone fixed some memory leaks recently. Do you think @lmaylein is running an affected version? #4903 indicates he is running 4.8.6.

[donsizemore]

@pdurbin yes, file handles could cause glassfish to stop responding, but restarting the service should clear the open file descriptors. BTW, we have a patched warfile if anybody wants that particular fix without upgrading to 4.9.

[pdurbin]

@lmaylein are you still having these problems? I hope not!

[lmaylein]

I'm going to test this when our migration to 4.9.4 is finished...

[lmaylein]

blocked by #5142

[lmaylein]

We are now on 4.9.4.
I will watch it.

[lmaylein]

It seems to me that we have at least a similar problem.
After about 24 hours the glassfish process runs with 400% CPU and is not responding for several minutes.

Unlike the problems with 4.8.6 there are no specific error messages in server.log.

Hardware: Virtual machine, 8 GByte RAM
OS: CentOS
Glassfish-Konfiguration:

        <jvm-options>-XX:MaxPermSize=512m</jvm-options>
        <jvm-options>-XX:PermSize=256m</jvm-options>
        <jvm-options>-Xmx4096m</jvm-options>
        <jvm-options>-Xms4096m</jvm-options>

[pdurbin]

@lmaylein that doesn't sound great. And it's hard to troubleshoot without logs. I wonder what Glassfish is doing. It's fine to post here, of course, but I would suggest opening a ticket as well by emailing support@dataverse.org

[pdurbin]

I'm seeing "java.lang.OutOfMemoryError: GC overhead limit exceeded" in server.log in the poor Dataverse server I'm hitting every 2 seconds with curl over at #6035 (comment)

[mheppler]

Found this issue while looking for a home for the comments below on a closed issue. Wanted to reference them somewhere, so I can link to them in Upgrade to PrimeFaces 8 #6634. Happy to move them somewhere more appropriate or open a new issue. (The use case that @pdurbin describes in his last comment directly above this one, is very, very similar to what is described by @landreev below.)

Based on the comment in issue 6035 from @PaulBoon, we should investigate if this issue is improved when upgrading to PrimeFaces 8, as a fix for primefaces/primefaces#4849 was included in the PrimeFaces 7.0.5 release.

Related comments about autoupdate and testing from @landreev on that issue as well:

The autoupdate issue (referenced above) is an interesting lead. (And it is something our command line tests are NOT testing at all.)

Hmm. There is a growing number of javax.faces.component.StateHolderSaver instances (mentioned in the autoupdate issue) that are not garbage-collected, even with testing page loads on the command line.

To be clear, that leak in the PrimeFaces autoUpdate is not our biggest problem. It's a tiny object (24 bytes!); it's just that the person who reported the bug has a pretty unique application case - thousands of users, all keeping active sessions open for hours, with pages refreshing continuously via autoupdate; so in the end they get enough of the copies of these 24 byte classes to eat up all their memory.

[djbrooke]

@lmaylein - I'm going to close this out, as I hope that it was fixed by #6035 (or will be fixed by already-merged #6634). If you confirm it's still an issue, feel free to reopen.