Update to payara-5.2021.8 with upgrade instructions

[qqmyers]

At some point we'll want to update to payara 5.2021.8 (or later). This issue is a placeholder for that and a place to document the following so we don't forget when a payara update happens. I think this will require some release notes/upgrade instructions for whatever Dataverse release it is associated with:

It looks like 5.2021.8 includes and update of the H2 database to v 1.4.200 which can expose an issue with older H2 databases, specifically h2database/h2database#2078 . In making updates at QDR, I ran into this and was unable to get Dataverse to restart after an update due to this, seeing stack traces about the EJBTimers that include org.h2.message.DbException 'unable to read at position ...' errors and 'java.lang.IllegalStateException: Unsupported type 17'.

The solution was to remove the contents of <domain>/lib/databases prior to restarting, along with deleting the contents of the <domain>/generated and <domain>/osgi-cache directories. A few notes:

• I did not undeploy Dataverse, just stopped the service/stopped the domain. It's possible that undeploy/redeploy would allow some other way to handle things
• Deleting just the lib/databases dir contents caused errors in stopping the domain as the EJBTimers and org.glassfish.flashlight.impl.provider.FlashlightProbeProviderFactory.unregisterProbeProvider failed to shutdown properly (since their db info was gone). This resulted in failure to restart the domain and left the server with Dataverse not running. Removing the generated and/or osgi-cache directory content and restarting fixed this (presumably the rest of the state of the EJBTimers/Flashlight is cached in those locations.) Restarting after all of those were cleared allowed a good restart.
• I also experimented with removing the <jvm-options>-javaagent:/usr/local/payara5/glassfish/lib/monitor/flashlight-agent.jar</jvm-options> line in domain.xml . As far as I can tell this didn't help but I mention it in case a restart without that line did something important (i.e. along with clearing the dirs).
• At QDR, I noticed a lib/databases/ejbtimer subdirectory along with ejbtimer.lock.db ejbtimer.mv.db and ejbtimer.trace.db files associated with H2 (the last only if there's been an error). Looking a bit, the subdirectory is from the old Derby instance - our instructions to copy the domain forward leave the old unused Derby db on the disk. The fix here to clear lib/databases gets rid of it. (However, new installations may not have the Derby cruft at all).

In terms of instructions, I think we've had releases where clearing the generated/osgi-cache dirs has been mentioned. I think this just adds one more (lib/databases).

[poikilotherm]

Hey @qqmyers maybe you would like to include your valuable thoughts and findings in #8064? 😉

[pdurbin]

Closing. Consolidating. I copied the content above into #8064 as discussed in #ig-devops in Slack.