Add support for Scala Native

[djneades]

It is early days for Scala Native (https://github.com/scala-native/scala-native), but support would be be very helpful.

[jcranky]

Perhaps it is a naive question, but would supporting scala native means in practice?

[djneades]

That’s a good question. I have only just begun to look at Scala Native myself, but immediately ran into problems because none of my project’s dependencies had Scala Native support (hence this issue). I see that the scalaz folks have recently started to support Scala Native, so their work may give a few useful pointers as to what could be required: https://github.com/scalaz/scalaz/wiki/7.3.0-M10

I suspect that some of the Cats dependencies (e.g. Simulacrum) will need themselves to acquire Scala Native support before Cats itself can do so.

[mscharley]

1. So, compiler plugins should continue to work and not need any changes. (I believe)
2. Any JVM dependencies however will need to be ported across.
3. This project would need to be cross-compiled.

For a taste, here's a PR for a simple library I manage: mscharley/scala-xoroshiro128#4

[aoiroaoino]

FYI
scalaz - scalaz/scalaz#1322
Monocle - optics-dev/Monocle#508

[mkotsbak]

@mscharley 2. JVM dependencies should not be a problem as Cats is already supporting Scala.Js which also does not support any JVM dependencies.

[4lex1v]

I'd to try Cats with some experimental Scala Native code, any chance that this issue will get some attention? How hard to add support for Scala Native?

[johnynek]

I don't know of anyone working on it. Someone interested should just try. It is probably just some sbt surgery to get it to work. I very much doubt dependencies or jdk usage will be a barrier.

[ChetanBhasin]

I would love to try out my hand at this, but I think a bigger bottleneck is Simulacrum at this point.

According to Scaladex, Simulacrum doesn't yet support native, but I'm not sure how substantial changes would be required there to support native.

[denisrosset]

Adding support for scala native to simulacrum was not a big deal; see typelevel/simulacrum#92

However, it looks like scalacheck/scalatest will not be supported any time soon!

[schrepfler]

Just to highlight, if we get cats-kernel for SN, we can probably pull spire fore SN which would be really interesting to benchmark.

[cb372]

I managed to cross-build cats-kernel for scala-native. Here's the commit with my changes. Happy to submit a PR if it's deemed useful.

I made a toy project that uses cats-kernel and actually works in scala-native, which is pretty cool!

Getting any other modules to cross-build is a lot more complicated, as others have discovered in the comments above.

Here is the list of things that I think are still blocking cats-core from being cross-built:

• simulacrum - PR in progress
• machinist - PR raised but ignored - is this project dead? Hasn't seen any maintenance for months. PR is now slightly out of date.
• scalatest - PR has been going on for months but is still being actively worked on

And these are blocking the various testing/laws modules from being cross-built:

• scalacheck - PR in progress
• discipline - no issue/PR
• catalysts - issue

[kubukoz]

Update for Scalacheck - typelevel/scalacheck#368 is merged, but the artifacts seem not to be published yet. Progress in typelevel/scalacheck#396

[kubukoz]

Aaand I added an issue for Discipline: typelevel/discipline#57

[s5bug]

Status update:

Here is the list of things that I think are still blocking cats-core from being cross-built:

• simulacrum - PR in progress
• machinist - PR raised but ignored - is this project dead? Hasn't seen any maintenance for months. PR is now slightly out of date.
• scalatest - PR has been going on for months but is still being actively worked on

I checked all of these PRs and either them or their replacements have been merged. 🎉

And these are blocking the various testing/laws modules from being cross-built:

• scalacheck - PR in progress
• discipline - no issue/PR
• catalysts - issue

Scalacheck artifact publishing seems to be paused/abandoned? Last comment is past one month. rickynils/scalacheck#396

Discipline has a new issue:

Aaand I added an issue for Discipline: typelevel/discipline#57

The catalysts issue typelevel/catalysts#11 hasn't had any activity. The last commit to master is from 2 months ago.

Update:

For catalysts, I am helping with maintenance, but could use some help on adding native support (once unblocked with scalacheck)

Summary:

• Simulacrum
• Machinist
• Scalatest
• Scalacheck (Dead?)
• Discipline (Blocked by Scalacheck)
• Catalysts (Blocked by Scalacheck)

[kailuowang]

For catalysts, I am helping with maintenance, but could use some help on adding native support (once unblocked with scalacheck)

[Alistair-Johnson]

re catalysts... see typelevel/catalysts#11

[Alistair-Johnson]

OTOH... I wondering if it's useful to add scala-native support wherever we can in the build, even if that means code cannot be tested. And what about publishing untested code?

Just some thoughts...

[schrepfler]

This would be great to happen.

[lolgab]

Seeing no interest from Scalacheck maintainer on Scala native artifact publishing, I published it under my groupid.
You can use it: "com.github.lolgab" %%% "scalacheck" % "0.4.1".

[schrepfler]

So I guess now all we need is Discipline and Catalyst? Will anyone reach out to the project leads?

[lolgab]

I'm working on porting Catalysts.

[denisrosset]

Ohoh, it seems that cats-kernel already works. If algebra is ported (I don't see any obstacle), we'll see which parts of Spire cooperate well!

[lolgab]

I tried to port Catalyst. It uses a custom sbt plugin with all typelevel libraries versions.. And not all of them crosscompile. The plugin doesn't allow to use different groupids for the same artifact ( org.scalacheck for jvm and js and com.github.lolgab for native).
There is another problem, scalatest 3.1.x is not compatible with scalacheck, but uses its Generators. I ported upickle's scalacheck tests to scalatest but with cats:

• there are a lot more tests
• I don't think you want to use SNAP version of the testing library.

[schrepfler]

We need some help from @kailuowang

[lolgab]

Discipline depends on specs2 but I think the specs2 side is irrelevant for cats (it has 2 optional dependencies, specs2 or scalatest. How can we work around it? Port specs2 is hard..

[kailuowang]

@lolgab one way I can think of is to break out discipline's specs2 dependent code into a new module and thus rid of cats dependency on specs2 - maybe in that light, we should do the same with scalatest dependent code as well.
IIRC, it shouldn't take too much time. I can support you on the discipline side.

[lolgab]

@kailuowang I made a PR to discipline to remove specs2 dependency from core:
typelevel/discipline#105
There were only a Test I rewrote in scalacheck (because it is already a dependency of discipline).
I changed a little the .travis.yml to test both Scala Native 0.3.9 and 0.4.0-M2.
Hope you'll publish the artifact for the two after merging :)

[lolgab]

In the meantime I published some artifacts for Scala Native 0.3 and 0.4.0-M2 under my organization:

"com.github.lolgab" %%% "cats-core" % "2.0.0-M4"

All cats library is supported..

[kailuowang]

thanks @lolgab , let's see if we can get them to add a new active maintainer.

[s5bug]

Artifacts for Scalacheck/Native have been published! typelevel/scalacheck#396 (comment)

[kubukoz]

We should get back to this! It would be lovely to have cats & cats-effect published for Native, esp. for things like decline and CLI tools written with it.

[s5bug]

We're kind of blocked by Scala Native itself now, with it not supporting 2.12 nor 2.13 yet. It also hasn't seen a commit to master in about half a year.

We'll have to wait for 0.4.0-M3 at least of SN, and then start the process of getting dependencies published again.

I don't remember if Threads are planned to be available in 0.4.0 or not. Would that be a blocker to CE or would it just hook in to something like libuv?

[johnynek]

I just want to mention, graal native image works great with cats and decline and I use it to make instantly starting and fast native builds of scala code. I’d recommend it to anyone.

[rtar]

I just want to mention, graal native image works great with cats and decline and I use it to make instantly starting and fast native builds of scala code. I’d recommend it to anyone.

Scala Native is a different and very interesting beast. I.e. it allows you to just go and use any C library with no or close to no boilerplate.

I ended up doing Scala Native apps whenever I need to use C library because it is... easier and more convenient than setting up Makefile etc. 😆

[rtar]

We're kind of blocked by Scala Native itself now, with it not supporting 2.12 nor 2.13 yet. It also hasn't seen a commit to master in about half a year.

We'll have to wait for 0.4.0-M3 at least of SN, and then start the process of getting dependencies published again.

I don't remember if Threads are planned to be available in 0.4.0 or not. Would that be a blocker to CE or would it just hook in to something like libuv?

I think if it works for Scala.js where only one thread is available, it may, just as well, work for Scala Native.

[s5bug]

Scala.js uses an event loop, so it would work for Scala Native under libuv. The thing is that libuv really defeats the point of going fast when you're bound to one thread with it.

[rtar]

Scala.js uses an event loop, so it would work for Scala Native under libuv. The thing is that libuv really defeats the point of going fast when you're bound to one thread with it.

I think it would be a temporary issue. And, as soon as multiple threads are implemented natively, the one would be able to get a free speed-up.

I think the secret weapon of Scala Native is awesome interoperability with existing C libraries and very little dependencies / bloat, which becomes very important in Kubernetes dominated world.

[lolgab]

Completely agree @rtar!

[ruurtjan]

Scala native 0.4.0 has been released, so I guess that removes a blocker from this issue :)
https://scala-native.readthedocs.io/en/v0.4.0/changelog/0.4.0.html

I don't remember if Threads are planned to be available in 0.4.0 or not. Would that be a blocker to CE or would it just hook in to something like libuv?

From the release notes:

While the GC itself will use multiple threads, Scala Native still does not support multi-threading in the application code. Commix GC was written in C and uses pthread to work. In case your application needs concurrency support, you may try the experimental library scala-native-loop

[kubukoz]

I guess we'll worry about concurrency when we get Cats published and it's time to work on cats-effect 😅 AFAIK ZIO made it, so we might be able to as well.

[lolgab]

I'm interested in contributing to port the Cats and other Typelevel libraries to Scala Native.
Read here about opening an issue on Github. Where am I supposed to open it?
Anyway, I openen a PR to add Scala Native to discipline-munit here: typelevel/discipline-munit#73
It is already green on CI ✅

[arashi01]

Okay so all cross module currently building and testing for me locally. See #3750

discipline-munit is the only think holding everything up. If someone can have a look at typelevel/discipline-munit#73 mentioned above, and cut a discipline-munit release then we can resolve this soon!

[lolgab]

@arashi01 There is discipline-scalatest too. If it's fine for you to depend on Scalatest 2.3.4-M1 I can send a PR there too, otherwise we can wait for Scalatest 2.3.4

[arashi01]

@lolgab I think easiest is to just wait for a discipline-munit release including your PR to be cut. I think the general direction is to move from Scalatest to MUnit, since MUnit has better support for current Dotty versions.

c.c @larsrh !

[larsrh]

I will take a look.