Lots of Builds in the community-build are silently skipping tests and even test-compiles

[deusaquilus]

It looks like lots of Builds not running or even compiling their tests. When they get to testing this happens:

2022-01-30T08:16:22.4896099Z [error] Error loading test framework (java.lang.NoClassDefFoundError: sbt/testing/Framework). This usually means that you are using a layered class loader that cannot reach the sbt.testing.Framework class. The most likely cause is that your project has a runtime dependency on your test framework, e.g. scalatest. To fix this, you can try to set
2022-01-30T08:16:22.4899342Z [error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
2022-01-30T08:16:22.4900029Z [error] or
2022-01-30T08:16:22.4900718Z [error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
2022-01-30T08:16:22.5477827Z [success] Total time: 23 s, completed Jan 30, 2022 9:16:22 AM

As similar issues happens for some builds trying to use ZTestFramework:

2022-01-30T08:21:26.8954542Z [error] Error loading test framework (scala.MatchError: zio.test.sbt.ZTestFramework@31fd0360 (of class zio.test.sbt.ZTestFramework)). This usually means that you are using a layered class loader that cannot reach the sbt.testing.Framework class. The most likely cause is that your project has a runtime dependency on your test framework, e.g. scalatest. To fix this, you can try to set

When this error occurs in a project, none of the tests will be run or even compiled. It is effectively a total disabling of testing code for the given project.

I first saw these in the my quill PR builds:

Some quick grepping (attached) shows me this is happening for the following projects:

in community-build-a

• scalatest
• scala-stm
• scalatestplus-testng
• izumi-reflect
• Lucre

in community-build-c

• protoquill (I got this one...)
• akka
• jackson-module-scala
• libretto
• xml-interpolator
• scalatestplus-scalacheck
• scala-parallel-collections
• sconfig
• onnx-scala

I solved this problem by adding something slightly different to my settings:

Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat

When I added this, the error went away.

This could all be totally wrong and my grepping could be incorrect too but I've tried this with the quill build. I tried running the community build with and actual compile error in my scalatests and without the classLoaderLayeringStrategy parameter. The build passes as though nothing is wrong.

I think that project owners should maybe have a look at their builds and see if this is happening for them.

grepping_results.txt

[smarter]

Huh, it's weird that sbt logs an error but doesn't fail the task. If ClassLoaderLayeringStrategy.Flat fixes it then I guess we should do this by default but I wonder when this issue started. Does this ring any bell @griggt ?

[ekrich]

I know for sconfig that the pulled version is a bit old causing errors with the newer sbt. Not sure how that gets updated.

[griggt]

I noticed it this weekend but ran out of time to investigate. It's on my todo list for tonight.

Thanks for opening the ticket @deusaquilus

It's a bit concerning that sbt seems to return a success exit code despite the error.

[pshirshov]

Well, I'm not sure what is this about, though in case you need us to make any changes in izumi-reflect we are happy to help. I believe just adding that setting would be enough?

[griggt]

This issue first started with the upgrade to sbt 1.6.1 in #14202

https://github.com/lampepfl/dotty/actions/runs/1657330713

The preceding CI run on master was clean

https://github.com/lampepfl/dotty/actions/runs/1656594683

[ekrich]

Older versions used withDottyCompat so maybe updating to new versions. I tried this locally.
git submodule update --remote Don't know if that is correct but it did update a bunch of projects.

[deusaquilus]

@pshirshov izumi-reflect community-build appears to not be running or even compiling tests. It has the following error:

[error] Error loading test framework (java.lang.NoClassDefFoundError: sbt/testing/Framework). This usually means that you are using a layered class loader that cannot reach the sbt.testing.Framework class. The most likely cause is that your project has a runtime dependency on your test framework, e.g. scalatest. To fix this, you can try to set
[error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
[error] or
[error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
[error] Error loading test framework (java.lang.NoClassDefFoundError: sbt/testing/Framework). This usually means that you are using a layered class loader that cannot reach the sbt.testing.Framework class. The most likely cause is that your project has a runtime dependency on your test framework, e.g. scalatest. To fix this, you can try to set
[error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
[error] or
[error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat

You can see it here:
https://github.com/lampepfl/dotty/runs/5001357175?check_suite_focus=true#step:8:3950

There are a couple ways to fix this. I fixed it by adding the following setting to SBT build.

Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat

[smarter]

This issue first started with the upgrade to sbt 1.6.1 in #14202

Thanks, I assume something in 1.6.0 is the culprit here but it's not clear what (sbt/sbt#6480 is the only related PR I could find). @eed3si9n: does sbt publish nightly builds these days?

[smarter]

The fact that our community build apparently still injects the obsolete sbt-dotty plugin might be related since sbt/sbt#6480 removed support for sbt-dotty: https://github.com/lampepfl/dotty/blob/1776d812cfaec31813d5e0865ebdb8654aaf3de0/project/Build.scala#L1591

[griggt]

our community build apparently still injects the obsolete sbt-dotty plugin

Any objection to discontinuing this practice and manually fixing up in the dotty-staging forks any projects that have not updated upstream?

[deusaquilus]

@griggt @smarter So I still use dottyLatestNightlyBuild().get from sbt-dotty to get latest nightly version so that I can compile with it locally. Has this functionality been reproduced by something in SBT proper?

[smarter]

@griggt That'd be awesome, go for it!
@deusaquilus Nothing built in, you'd have to copy the code in your own build: https://github.com/lampepfl/dotty/blob/2f407470d2fca0f57b46bfde37acf6d79915f1f4/sbt-dotty/src/dotty/tools/sbtplugin/DottyPlugin.scala#L30-L83

[deusaquilus]

@smarter Works for me. Thanks!

[griggt]

Some local testing shows the issue also manifests with sbt 1.6.0 but not 1.5.8 (we jumped from 1.5.7 to 1.6.1 in #14202)

It also appears that not injecting sbt-dotty when using sbt 1.6.x fixes things (at least for shapeless)

Will have more time later tonight to investigate further and PR a fix. It still seems to me that sbt should return a non-zero exit code if the tests aren't being run.

[eed3si9n]

does sbt publish nightly builds these days?

Sorry I haven't gotten around to resurrecting nightlies.

I wonder if this is a side effect of no longer trapping the exit code or waiting on all threads in tests. If so forking might fix it, but it's somewhat controversial - sbt/sbt#6413

[eed3si9n]

I have a PR up so it won't silently succeed the build - sbt/sbt#6806