Gradle workers

[mikehalmamoj]

v10.0.0 introduced Gradle workers to perform the linting - this causes us a slight issue with memory management on our CI builds as we need to keep the number of threads to a minimum due to blowing max heap size.

Please can you make the workers configurable? i.e. allow us to turn them off.

[Tapchicoma]

Workers count is configured via Gradle org.gradle.workers.max property (or disable parallel task execution).

Workers memory you could set via workerMaxHeapSize, though I would advice you not to decrease it further.

Generally if you are using on your CI some docker executor with cgroup memory limit, I would advice you as well to set JAVA_TOOL_OPTIONS environment variable to "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:-PreferContainerQuotaForCPUCount".

i.e. allow us to turn them off.

This will not happen.

[mikehalmamoj]

Ha! Fair enough.

You wouldn't believe how much pain we've suffered trying to tweak the Java options for memory management. Another couple of options to throw into the mix - maybe the magic spell will work this time. :)

[abrooksv]

I ran into this too but on Windows (so the UseContainerSupport doesn't work)

We have 6 gradle modules with 2-3 source sets per.

Our logs are filled with:

:core:runKtlintCheckOverKotlinScripts (Thread[Execution worker for ':',5,main]) started.
:core:runKtlintCheckOverMainSourceSet (Thread[Execution worker for ':' Thread 3,5,main]) started.
core:runKtlintCheckOverTestSourceSet (Thread[Daemon worker,5,main]) started.
Starting process 'Gradle Worker Daemon 11'. Working directory: C:\Users\ContainerAdministrator\.gradle\workers Command: C:\Program Files\Amazon Corretto\jdk11.0.9_12\bin\java.exe @C:\Users\ContainerAdministrator\AppData\Local\Temp\gradle-worker-classpath17185580907803460576txt -Xmx256m -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 11'
Successfully started process 'Gradle Worker Daemon 11'
Started Gradle worker daemon (0.735 secs) with fork options DaemonForkOptions{executable=C:\Program Files\Amazon Corretto\jdk11.0.9_12\bin\java.exe, minHeapSize=null, maxHeapSize=256m, jvmArgs=[], keepAliveMode=DAEMON}.
Started Gradle worker daemon (0.865 secs) with fork options DaemonForkOptions{executable=C:\Program Files\Amazon Corretto\jdk11.0.9_12\bin\java.exe, minHeapSize=null, maxHeapSize=256m, jvmArgs=[], keepAliveMode=DAEMON}.
Started Gradle worker daemon (1.079 secs) with fork options DaemonForkOptions{executable=C:\Program Files\Amazon Corretto\jdk11.0.9_12\bin\java.exe, minHeapSize=null, maxHeapSize=256m, jvmArgs=[], keepAliveMode=DAEMON}.

This has doubled our memory usage and causing us to run out of paging file on windows machines.

It seems the workers are keepAlive DAEMON so the memory is never returned back, when ktlint is done running.

Downgrading back to 9.4.1, we are not seeing this memory growth curve.

[jzbrooks]

I run into this pretty regularly for local builds on macOS without containers. I'll capture a dump the next time it happens, if that would be helpful. Since it only happens once or twice a day to me, I usually just kill the daemons with ./gradlew --stop, which works around the issue.