Mill clean should stop/close workers

[lefou]

Discussed in #3272

^{Originally posted by roman-mibex-2 July 18, 2024}
TLDR: mill clean should stop workers / or discard them.

I have a mill build with some webpack workers, which stay up. I sometimes had a puzzling state
where these workers seem complete out of sync. I've noticed that a ./mill clean breaks them.

Here is a small demostration, without webpack ;)

build.sc:

import mill._
import os.Path

object `mill-demo` extends RootModule  {
  def resources = T.source(millSourcePath/"barely-changing.txt")
  def source = T.source(millSourcePath/"frequently-changing.js")

  def webpackWorker = T.worker{
    // When the worker starts, we copy stuff from other targets
    // These targets rarely change, and we want to keep them 'hot' in the worker
    os.copy(resources().path,T.dest/"resource.txt",replaceExisting = true)
    new MyWebpackWorker(T.dest)
  }

  def webpack = T{
    webpackWorker().compile(source().path)
  }

  class MyWebpackWorker(myWorkingDir:Path){
    println("Worker is starting up")
      def compile(latest:Path)={
        os.copy(latest,myWorkingDir/"code.js", replaceExisting = true)
        val state = PathRef(myWorkingDir)
        println(s"State of working dir is ${state}")
        state
      }
  }
}

Demostration of the issue:

$ date > barely-changing.txt
$ date > frequently-changing.js

# Initial build, all fine
$ ./mill show webpack
[1/1] show > [2/4] webpackWorker 
Worker is starting up, worker:1930565333
[1/1] show > [4/4] webpack 
worker: 1930565333
"ref:v0:51849a16:/home/roman/dev/private-dev/mill-demo/out/webpackWorker.dest"

# I change the source code, all works as expected
$ date > frequently-changing.js
$ ./mill show webpack
[1/1] show > [4/4] webpack
worker: 1930565333
"ref:v0:800319cb:/home/roman/dev/private-dev/mill-demo/out/webpackWorker.dest"

# In the rare occasion the resources change, the worker is restarted, works as expected
$ ./mill show webpack
[1/1] show > [2/4] webpackWorker 
Worker is starting up, worker:2000339927
[1/1] show > [4/4] webpack 
worker: 2000339927
"ref:v0:9faaa5de:/home/roman/dev/private-dev/mill-demo/out/webpackWorker.dest"

# However, as soon as mill clean ran, hell breaks loose
# The worker is still alive, but the underlying workind directory was wiped
[1/1] show > [4/4] webpack 
worker: 2000339927
1 targets failed
show 1 targets failed
webpack java.nio.file.NoSuchFileException: /home/roman/dev/private-dev/mill-demo/out/webpackWorker.dest/code.js
    java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    java.base/sun.nio.fs.UnixFileSystem.copyFile(UnixFileSystem.java:673)
    java.base/sun.nio.fs.UnixFileSystem.copy(UnixFileSystem.java:1084)
    java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:300)
    java.base/java.nio.file.Files.copy(Files.java:1304)
    os.copy$.copyOne$1(FileOps.scala:200)
    os.copy$.apply(FileOps.scala:204)
    millbuild.build$mill$minusdemo$MyWebpackWorker.compile(build.sc:28)
    millbuild.build$mill$minusdemo$.$anonfun$webpack$3(build.sc:19)
roman@roman-lenovo24 ~/d/p/mill-demo (master) [1]> 

As demostrated, the mill clean cleans the out directory, but does not stop the worker.
Then, when the worker is invoked again, it's working directory is wiped.
In this demostration, it simply crashes.

In the real work this ends up with a webpack worker with a broken working directory,
producting garbage results.

Therefore, I think that mill clean should kill/restart workers.
Better solutions are of course also welcome.

[lefou]

As a work around, you can run mill shutdown to stop the server, which will also stop all workers.

[lihaoyi]

@alexarchambault this is a good task to pick up to get familiar with the internals of the evaluator