Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring back ability to disable ClassLoader isolation #3654

Closed
wants to merge 2 commits into from

Conversation

C2H6O
Copy link
Contributor

@C2H6O C2H6O commented Nov 8, 2022

This PR brings back functionality that was introduced in #2082 but was recently removed 077837a#diff-202cb8a4767ef0496a1551b59461db6625ca9aad2ddf92fefc467a07887cfe01. We rely on shared ClassLoaders for setting up a Sqlite Driver (with a tokenizer) prior to running the migrations.

I took a slightly different approach from #2082 by introducing the disableClassLoaderIsolation property on the VerifyMigrationTask, instead of SqlDelightWorkerTask. Not sure which approach is preferred, let me know!

Related issue: #3618

@C2H6O C2H6O force-pushed the lx/disableClassLoaderIsolation branch from fb48006 to b003a24 Compare November 11, 2022 19:04
@hfhbd
Copy link
Collaborator

hfhbd commented Nov 13, 2022

I think we should use classloader isolation only and provide a configuration: https://docs.gradle.org/current/userguide/worker_api.html#changing_the_isolation_mode

See also: #3644

@C2H6O
Copy link
Contributor Author

C2H6O commented Nov 14, 2022

@hfhbd our use case is the following:

afterEvaluate {
  tasks.withType<VerifyMigrationTask>().configureEach {
      doFirst {
          TokenizingDriver(this.project).replaceJDBCDriver()
      }
  }
}

TokenizingDriver:

public fun replaceJDBCDriver() {
    // JDBC installs itself on initialization. Let's... not.
    val driverList = DriverManager.getDrivers().toList()
    DriverManager.deregisterDriver(driverList[0])
    DriverManager.registerDriver(this)
}

DriverManager contains a static list of drivers and if we use ClassLoader isolation the shared static state that we've setup prior to running the VerifyMigrationTask is not accessible to the ClassLoader running the VerifyMigrationTask. What's unclear to me is whether the approach outlined in #3644 would resolve this issue. Would this approach allow us to share the same static instance?

@C2H6O
Copy link
Contributor Author

C2H6O commented Nov 21, 2022

@hfhbd @AlecStrong is merging this PR something that you are willing to entertain? I'm trying to figure out if we should setup a fork of SqlDelight with these changes or not. Without these changes we are unable to run the migration verification tasks with our custom Sqlite driver 🙏

@hfhbd
Copy link
Collaborator

hfhbd commented Nov 24, 2022

So the real problem is the autoregistration of the jdbc driver? If we use the classpath and configuration gradle option, then you could simple override this dependency, I guess. I recently tried this approach in another project and it works nicely, but I didn't test it with default dependencies.

@AlecKazakova
Copy link
Collaborator

do you mind running spotlessApply on this branch? In general I'm not reviewing PRs that don't go green (I check to see if it was a flake since those do happen sometimes) since the tests 9 out of 10 times do a better job reviewing code than i do

@AlecKazakova
Copy link
Collaborator

I'm not opposed to this, but I'm pretty sure we just cant operate with classloaderIsolation disabled anymore because of how dialects are loaded now. However if all the tests pass, it might be fine

@C2H6O C2H6O force-pushed the lx/disableClassLoaderIsolation branch from 467e7bb to f353f53 Compare November 27, 2022 02:49
@C2H6O C2H6O force-pushed the lx/disableClassLoaderIsolation branch from f353f53 to 1b5cc7f Compare November 27, 2022 02:49
@C2H6O
Copy link
Contributor Author

C2H6O commented Nov 27, 2022

spotlessApply ... Looks like workflows (GH Actions) don't run on this PR since I'm a first time contributor

@C2H6O
Copy link
Contributor Author

C2H6O commented Nov 27, 2022

Looks like the new test in this PR is failing...

Caused by: java.util.NoSuchElementException: No value present
	at app.cash.sqldelight.gradle.VerifyMigrationTask$VerifyMigrationAction$environment$2.invoke(VerifyMigrationTask.kt:113)
	at app.cash.sqldelight.gradle.VerifyMigrationTask$VerifyMigrationAction$environment$2.invoke(VerifyMigrationTask.kt:105)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at app.cash.sqldelight.gradle.VerifyMigrationTask$VerifyMigrationAction.getEnvironment(VerifyMigrationTask.kt:105)
	at app.cash.sqldelight.gradle.VerifyMigrationTask$VerifyMigrationAction.execute(VerifyMigrationTask.kt:118)
	at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:63)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1$1.create(NoIsolationWorkerFactory.java:66)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1$1.create(NoIsolationWorkerFactory.java:62)
	at org.gradle.internal.classloader.ClassLoaderUtils.executeInClassloader(ClassLoaderUtils.java:100)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1.lambda$execute$0(NoIsolationWorkerFactory.java:62)
	at org.gradle.workers.internal.AbstractWorker$1.call(AbstractWorker.java:44)
	at org.gradle.workers.internal.AbstractWorker$1.call(AbstractWorker.java:41)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.workers.internal.AbstractWorker.executeWrappedInBuildOperation(AbstractWorker.java:41)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1.execute(NoIsolationWorkerFactory.java:59)
	at org.gradle.workers.internal.DefaultWorkerExecutor.lambda$submitWork$2(DefaultWorkerExecutor.java:205)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runExecution(DefaultConditionalExecutionQueue.java:187)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.access$700(DefaultConditionalExecutionQueue.java:120)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner$1.run(DefaultConditionalExecutionQueue.java:162)
	at org.gradle.internal.Factories$1.create(Factories.java:31)
	at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:114)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runBatch(DefaultConditionalExecutionQueue.java:157)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.run(DefaultConditionalExecutionQueue.java:126)
	... 2 more

I'll have to look into this on Monday, but does that look like a failure that you were expecting @AlecStrong? If yes, then I'm not really sure what options there are for registering our own driver before migrations get run. Do you have any suggestions?

@hfhbd
Copy link
Collaborator

hfhbd commented Nov 28, 2022

@C2H6O Like I said, I would try to use defaultDependencies and a configuration and more Gradle standards instead so that you could add a dependency to your JDBC driver as a regular dependency without any hacks.

I use this in a similar project:

val kobolFlowGraphPlugin = project.configurations.register("kobolFlowGraphPlugin") {
  it.isCanBeResolved = true
  it.isCanBeConsumed = false
}
project.tasks.register("flowGraph", KobolFlowGraph::class.java) {
  it.classpath.from(kobolFirPlugin, kobolFlowGraphPlugin.map {
    it.defaultDependencies {
      it.add(project.dependencies.create("app.softwork:kobol-plugins-flow-graph-plantuml:$version"))
    }
  })
}

public abstract class CreateFlowGraph : WorkAction<CreateFlowGraph.Parameters> {
    public interface Parameters : WorkParameters {
        public val inputFiles: ConfigurableFileCollection
        public val outputFolder: DirectoryProperty
    }

    override fun execute() {
        val inputs: Set<File> = parameters.inputFiles.files
        val outputFolder = parameters.outputFolder.get().asFile

        val firPlugins = ServiceLoader.load(FirPluginBeforePhase::class.java) + ServiceLoader.load(
            FirPluginAfterPhase::class.java
        )
        for (flowGraph in ServiceLoader.load(FlowGraphFactory::class.java)) {
            inputs.toTree(firPlugins + flowGraph(outputFolder))
        }
    }
}

// Fetched with the ServiceLoader
public fun interface FlowGraphFactory {
    public operator fun invoke(outputFolder: File): FlowGraph
}

plugins {
  id("app.softwork.kobol")
}

dependencies {
// optional overwrite this dependency or use app.softwork:kobol-plugins-flow-graph-plantuml by default  
kobolFlowGraphPlugin(...)
}

tasks.flowGraph {
  sources.from("foo.cbl")
}

The JDBC Drivers are also fetched with the ServiceLoader mechanism, so I guess it will work.

@AlecStrong If we want to use more Gradle configurations to setup the dependencies, I will create a PR (and copy this test :)

@hfhbd
Copy link
Collaborator

hfhbd commented Nov 28, 2022

@C2H6O How do you setup your Driver at the moment? Could you share your code?

@C2H6O
Copy link
Contributor Author

C2H6O commented Nov 28, 2022

@hfhbd here's all the code relevant to the driver (and the driver itself, just for completeness):

class TokenizingDriver(project: Project) : Driver {
    private val wrappedDriver: JDBC = JDBC()
    private val projectRoot = project.rootDir

    public fun replaceJDBCDriver() {
        // JDBC installs itself on initialization. Let's... not.
        val driverList = DriverManager.getDrivers().toList()

        // If we have other drivers installed, houston, we have a problem!
        assert(driverList.size == 1)
        DriverManager.deregisterDriver(driverList[0])

        DriverManager.registerDriver(this)
    }

    override fun acceptsURL(url: String?): Boolean =
        wrappedDriver.acceptsURL(url)

    override fun connect(url: String?, info: Properties?): Connection {
        val props = info ?: Properties()
        props.setProperty("enable_load_extension", "true")

        val connection = wrappedDriver.connect(url, props)
        val ext = File(projectRoot, "build/sqlitetokenizer/libsqlite-fts5-synonym-tokenizer.dylib")
        connection.prepareStatement("SELECT load_extension('${ext.absolutePath}', 'sqlite3_fts5_synonym_tokenizer_init');").execute()

        return connection
    }

    override fun getMajorVersion(): Int =
        wrappedDriver.majorVersion

    override fun getMinorVersion(): Int =
        wrappedDriver.minorVersion

    override fun getPropertyInfo(url: String?, info: Properties?): Array<DriverPropertyInfo> =
        wrappedDriver.getPropertyInfo(url, info)

    override fun jdbcCompliant(): Boolean =
        wrappedDriver.jdbcCompliant()

    override fun getParentLogger(): Logger =
        wrappedDriver.parentLogger
}

In one of the build.gradle.kts files we have the following code:

afterEvaluate {
  tasks.withType<VerifyMigrationTask>().configureEach {
      doFirst {
          TokenizingDriver(this.project).replaceJDBCDriver()
      }
  }
}

It does 2 things:

  1. Unregisters the default JDBC driver that gets registered automatically (https://github.com/xerial/sqlite-jdbc/blob/master/src/main/java/org/sqlite/JDBC.java#L29):
  2. Registers our own TokenizingDriver

The main issue is that the JDBC driver gets automatically registered and we don't have a hook into the VerifyMigrationTask's ClassLoader to perform those 2 steps when JDBC driver gets loaded by a different ClassLoader.

Let me know if I've missed anything and can provide more info!

@C2H6O
Copy link
Contributor Author

C2H6O commented Mar 16, 2023

@hfhbd after spending some time on this, my guess is that I would add this logic into SqlDelightPlugin and then the consumer would be able to swap the driver like a regular dependency. Is that the idea?

@hfhbd
Copy link
Collaborator

hfhbd commented Mar 16, 2023

@C2H6O Yes, this is the idea.

@C2H6O
Copy link
Contributor Author

C2H6O commented Mar 24, 2023

Closing this in favor of #3986

@C2H6O C2H6O closed this Mar 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants