Manage Scala 3 instances and fetch compiler bridges

[adpi2]

Part of #6080
Depends on sbt/librarymanagement#350

Manage Scala 3 instances

When Scala version is Scala 3:

• the scala3-compiler artifact is added to the list of tool dependencies. scala3-compiler transitively depends on scala3-library and scala-library
• if autoScalaLibrary := true, the scala3-library is added to the list of library dependencies

Coursier does not handle autoScalaLibrary properly with Scala 3. But we don't really need to set autoScalaLibrary = true in the Coursier configuration because sbt already add the proper scala library to the list of dependencies.

Fetch the Scala 3 compiler bridge binaries

Contrary to Scala 2, the Scala 3 compiler bridges are tied to the compiler versions. There is a single compiler bridge for each compiler version. Also they are pure java so binary compatible with all Scala versions.

Consequently, we can fetch the binary module directly without using the ZincCompilerBridgeProvider.

Manage the .tasty files

This PR introduces an auxiliaryClassFiles setting that is passed to Zinc via incOptions.
When the Scala version is Scala 3, the auxiliaryClassFiles setting contains the TastyFiles singleton. Otherwise it is empty.

One can add more auxiliaryClassFiles with, for example, auxiliaryClassFiles += ScalaJSFiles.instance

[adpi2]

@eed3si9n How can I get the CI to run against the latest librarymanagement?
(without releasing librarymanagement because I may have some changes do to around IvyDependencyResoltuion about overrideScalaVersion)

[eed3si9n]

@adpi2 Not possible now, but it shouldn't be too hard to tweak https://github.com/sbt/sbt/blob/develop/.github/workflows/nightly.yml or copy-paste it and adding a workflow_dispatch similar to https://github.com/sbt/sbt-dist/blob/master/.github/workflows/distro.yml

[eed3si9n]

@adpi2 I just released https://github.com/sbt/librarymanagement/releases/tag/v1.5.0-M1

[adpi2]

@eed3si9n Thanks for publishing librarymanagement. I will also need latest zinc before merging this PR.

I changed the CI of this PR to make it run on fresh io, zinc and librarymanagement 1.5.0-SNAPSHOT, published locally.
But the scripted tests cannot resolve the compiler-bridge 1.5.0-SNAPSHOT because they don't use the user ivy-cache:

• The compiler-bridge is published to /home/runner/.ivy2/local/org.scala-sbt/compiler-bridge_2.12/1.5.0-SNAPSHOT/poms/compiler-bridge_2.12.pom
• But the scripted test try to resolve it from:

[info] [warn] 	module not found: org.scala-sbt#compiler-bridge_2.12;1.5.0-SNAPSHOT
[info] [warn] ==== local: tried
[info] [warn]   /tmp/sbt_9611947f/target/ivy-cache/local/org.scala-sbt/compiler-bridge_2.12/1.5.0-SNAPSHOT/ivys/ivy.xml

Do you know how I can force the scripted tests to use the user ivy cache?

[adpi2]

In fact, it seems that my ivy cache problem is due to the settings of some particular scripted test and not to scripted tests in general:

For instance in dependency-management/artifact/build.sbt

sbt/sbt/src/sbt-test/dependency-management/artifact/build.sbt

Line 20 in 21fc19a

ivyPaths := IvyPaths(baseDirectory.value, Some(target.value / "ivy-cache")),

overrides the user global cache.

[adpi2]

I adapted those tests with

scalaCompilerBridgeResolvers += userLocalFileResolver(appConfiguration.value)

// use the user local resolver to fetch the SNAPSHOT version of the compiler-bridge
def userLocalFileResolver(appConfig: AppConfiguration): Resolver = {
  val ivyHome = appConfig.provider.scalaProvider.launcher.ivyHome
  Resolver.file("User Local", ivyHome / "local")(Resolver.defaultIvyPatterns)
}

and it works!

[adpi2]

This is ready for review!

The autoScalaLib = false, in LMCoursier configuration may not be ideal but it doesn't seem to cause much trouble. Hopefully I'll be able to adapt Coursier to Scala 3 artifacts but it does not seem so simple: autoScalaLib is not an lmcoursier only param, it goes deep into Coursier code itself.

[adpi2]

For now I have just created an issue in the Coursier repository about autoScalaLib and forceScalaVersion: coursier/coursier#1933

[smarter]

Coursier does not handle autoScalaLibrary properly with Scala 3. But we don't really need to set autoScalaLibrary = true in the Coursier configuration because sbt already add the proper scala library to the list of dependencies.

I'm not sure I understand what that means, will the sbt setting autoScalaLibrary := false still work correctly in scala 3 projects? If sbt deals with adding the library itself, then could the parameter passed to coursier always be autoScalaLibrary = false or would that do the wrong thing in Scala 2? In any case it sounds like we should open an issue against coursier /cc @alexarchambault

[smarter]

Oh oops, I read the description but missed the comment where you mentioned coursier/coursier#1933 which I guess is the follow-up.

[adpi2]

I'm not sure I understand what that means, will the sbt setting autoScalaLibrary := false still work correctly in scala 3 projects? If sbt deals with adding the library itself, then could the parameter passed to coursier always be autoScalaLibrary = false or would that do the wrong thing in Scala 2?

So yes the autoScalaLibrary := flase will work correctly in Scala 3 and Scala 2 projects. Also autoScalaLibrary := true still works perfectly with Scala 2.

The only thing that is different with autoScalaLibrary := true and Scala 3 is that Coursier will not try to override the Scala version of the dependencies to Scala artifacts. I guess that it is completely harmless in the very large majority of cases.

Also I think that choosing a custom scalaOrganization does not work with Coursier. Is it even supported with Scala 2? I don't think so. But clearly in Scala 3 we don't need that yet, and it works with Ivy.

[smarter]

The only thing that is different with autoScalaLibrary := true and Scala 3 is that Coursier will not try to override the Scala version of the dependencies to Scala artifacts. I guess that it is completely harmless in the very large majority of cases.

It's something we'll have to figure out at some point, there's an issue open on that subject: scala/scala3#10146

[eed3si9n]

Do we know why things are failing on AppVeyor?

[error] C:\projects\sbt\main\src\main\scala\sbt\Defaults.scala:91:8: object TastyFiles is not a member of package xsbti.compile
[error] import xsbti.compile.TastyFiles
[error]        ^
[error] C:\projects\sbt\main\src\main\scala\sbt\Keys.scala:251:41: not found: type AuxiliaryClassFiles
[error]   val auxiliaryClassFiles = taskKey[Seq[AuxiliaryClassFiles]]("The auxiliary class files that must be managed by Zinc (for instance the TASTy files)")
[error]                                         ^
[error] C:\projects\sbt\main\src\main\scala\sbt\Defaults.scala:889:10: value withAuxiliaryClassFiles is not a member of xsbti.compile.IncOptions
[error] possible cause: maybe a semicolon is missing before `value withAuxiliaryClassFiles'?
[error]         .withAuxiliaryClassFiles(auxiliaryClassFiles.value.toArray)
[error]          ^
[error] three errors found
[error] (mainProj / Compile / compileIncremental) Compilation failed
[error] Total time: 147 s (02:27), completed Dec 23, 2020 10:17:18 AM

[adpi2]

@eed3si9n I need a release of zinc that contains sbt/zinc#956

[eed3si9n]

@adpi2 Let me get on that.