Whitelists are not loaded correctly in driver based tests #5914
Comments
|
Filed as https://r3-cev.atlassian.net/browse/CORDA-3566. Thanks for the excellent investigation / analysis. |
|
Thanks for taking a look at this! From Jira (which I can't seem to add comments to):
If there are good arguments for why in-process mode does not model reflect the environment of a real deployment, then I guess it can be removed. What are the main reasons that you think they don't do that? I think that even if they don't provide a good way to run integration tests on CI, in-process tests are still valuable for us for the following reasons:
If the in-process mode is removed, then I think the documentation should at least be updated with more details on how to debug failures. |
As you have observed, when nodes are run "in-process" then classes do not always end up being sourced from the classloaders that you expect. But having said that, we do appear to have regressed Corda from 4.0 here, and that isn't good either. You mentioned that your issue was at least fixed for "out-of-process" nodes in Corda 4.4. Can you try applying this diff for 4 4 to @@ -142,11 +142,7 @@ class DriverDSLImpl(
private lateinit var _notaries: CordaFuture<List<NotaryHandle>>
override val notaryHandles: List<NotaryHandle> get() = _notaries.getOrThrow()
- override val cordappsClassLoader: ClassLoader? = if (!startNodesInProcess) {
- createCordappsClassLoader(cordappsForAllNodes)
- } else {
- null
- }
+ override val cordappsClassLoader = createCordappsClassLoader(cordappsForAllNodes)
interface Waitable {
@Throws(InterruptedException::class) |
|
@chrisr3 Before applying the patch, I see, as expected, that the Whitelist is instantiated several times, but each time from the wrong location (the build output rather than the node directory): But after applying the pathc, I get the same result: The reason is that even though a class loader is now created via private fun createCordappsClassLoader(cordapps: Collection<TestCordappInternal>?): ClassLoader? {
if (cordapps == null || cordapps.isEmpty()) {
return null
}
return URLClassLoader(cordapps.map { it.jarFile.toUri().toURL() }.toTypedArray())
}each object in |
|
@ulrikrasmussen To be fair, I didn't expect that my diff would change where "in process" nodes would instantiate their whitelists from. Such a feat would probably require such a radical replumbing of how the driver uses ClassLoaders that it would ultimately be equivalent to using "out of process" nodes anyway. What I was actually wondering was whether my diff would allow the failing driver-based tests from your original report to pass again. |
|
Ah, sorry, I think I misunderstood you then. I'll try it out and will get back to you beginning of next week. |
|
This change has just been merged into the |
|
Sorry for taking so long to get back to you---I was suddenly sidetracked by something else at work. I wanted to test out your patch on my original problem, but I think we have been talking past each other. My original problem was that tests failed after an upgrade to Corda 4.3, but I can't apply your patch to that version of Corda (there is no |
|
Yes, but you also reported that the problem was "partially fixed" in Corda 4.4. I was therefore wondering if the problem in Corda 4.4 is now fixed completely. |
|
Yes, but that conclusion was based on just detecting the symptom of Whitelists not being instantiated from the correct jar, which I did by patching an existing test in the Corda repo instead of spending time upgrading my code to Corda 4.4. I would have to upgrade my own project to a release candidate of 4.4 to test if that makes my actual test pass. Is your patch included in the latest RC? |
|
Yes, this change has been included in tag |
|
@chrisr3 I now finally got around to upgrading our code to Corda 4.4. It seems to be working for driver based tests on 4.4, but there is now a similar issue for |
|
Hi @ulrikrasmussen, I think I fixed this in CORDA-3643, which should be included in Corda 4.5 😃 . |
The problem
After an upgrade to Corda 4.3 from 4.0, my driver based tests started to fail because the nodes were unable to find classes to be serialized on the whitelist. However, they are present on a whitelist published as a
META-INF.servicein the CorDapp.An investigation into the problem revealed that, when running a node via the driver based test framework, all CorDapps are loaded with an empty list of whitelists. The problem is in the function
findWhitelists()in the fileJarScanningCordappLoader.kt. It extends the classpath with the jar of the CorDapp that has been deployed on the node and then proceeds to create a list of instances of all classes implementingSerializationWhitelistthat can be found on the extended classpath, filtering out the instances whose class does not come from the deployed CorDapp jar.The code seems to hinge on the assumption that all
SerializationWhitelistservices on the classpath will actually be loaded, but this is not true. The documentation ofServiceLoaderincludes the following paragraph:Since the driver framework starts nodes with a classpath that includes the build paths of the project, and because parent classloaders are visited first,
ServiceLoaderwill instantiate the whitelists from the wrong location, and the instances in the deployed jar will never be created because they have the same package and class names.This happens regardless of the value of
startNodesInProcessin the driver parameters.To reproduce in Corda 4.3
The bug is reproduced here: https://github.com/ulrikrasmussen/reproduce-whitelist-bug
Run the driver based test and observe that it fails. However, the whitelist class is actually instantiated, and I have inserted logging code in the constructor which tells you the class location of the instantiated whitelist and the stack trace. This is printed to stdout on the nodes:
Note that the location of the instantiated class is from the build output, and not the CorDapp jar. Only this instance is created.
Further digging: Why did it work in Corda 4.0?
AMQPSerializationSchemeused to also invoke theServiceLoaderto find whitelists, which would skip verification of the class location and load the whitelist from the build directory anyway. This has probably masked the issue, and is why the problem did not occur in Corda 4.0. The behavior was removed in #5112, although the description in this PR does not indicate that this change of behavior was intentional.Is it fixed in Corda 4.4?
Only partially. Tested at version 1380779.
The easiest way to test this was to apply the patch in the bottom of this issue to the simm-valuation-demo and observe the location of the instantiated whitelist in
stdoutwhen running the driver based test. I ran the driver based test withstartNodesInProcessequal totrueandfalse, respectively.:startNodesInProcess = false: The loaded whitelist is the correct one from the deployed jar, so this case seems to have been fixed.startNodesInProcess = true: The incorrect whitelist from the build output is loaded. Also, custom serializers cannot be found, so the test fails for other reasons before it can get to the whitelist issue. This is probably a related bug that also has to be fixed.Debugging Patch:
The text was updated successfully, but these errors were encountered: