| layout | post-type | by | title |
|---|---|---|---|
blog-detail |
blog |
Paweł Marks, VirtusLab |
Long-term compatibility plans for Scala 3 |
Scala 3 has stronger compatibility guarantees than Scala 2 ever had. That allows us to improve the language at a much faster pace than it was possible before 3.0. However, with the current scheme, when you bump the dependencies of your project, you may also have to bump the version of Scala used by the project. After an extended evaluation, we have decided to solve them by creating a new kind of Scala release - Scala LTS, that will be supported and receive updates for at least three years after the initial release.
This post aims to explain what were the problems we are trying to solve and how we envision Scala LTS to work.
Compilers, as all software really, are constantly evolving. Since the initial release of Scala 3, there have been six stable versions of the compiler. Each of them has brought performance and stability enhancements, fixed bugs, improved reporting or introduced some new experimental features to the language. Even though we still treat all those versions as an implementation of the same language. What allows us to think in such a way is the compatibility guarantees.
The term “compatibility” is usually used quite vaguely. In the Scala world, the first intuition for this term could be phrased as follows: a new version of Scala is compatible with a previous version of Scala if a program that used to work with the previous version still works with the new version. Let me examine the term “compatibility” closer right now. For Scala 3 compiler, there are actually two things that are named “compatibility”: source compatibility and output compatibility.
We say that two versions of the compiler are source compatible if every single project that can be compiled with one version can also be compiled with the other, and the resulting programs behave in the same way. We put a lot of effort into assuring that source compatibility is preserved between the patch releases, and every infringement of it between minor versions is easy to fix. All in all, breaking it is usually not a big deal, as it can be easily detected at compile time. The worst consequence it can cause is temporarily locking the affected codebase on one version of the compiler.
Output compatibility is much more tricky and essential for our long-term plans for Scala. In the rest of this post, the term “compatibility” will always refer to output compatibility. We can say that compiler A is output compatible with compiler B when compiler B can use the output (binaries and TASTy files) generated by compiler A and it understands it correctly.
We can further subdivide compatibility to forward and backward. We say that the two versions are forward compatible when the older compiler can depend on the output of the newer one. Conversely, backward compatibility means that the newer compiler can use the output of the older version. Scala 3 guarantees backward compatibility between all releases and forward compatibility between patch releases in the same minor line. That means that Scala 3.a.x can consume the output of Scala 3.b.y only if a is greater or equal to b. For example, the output of the Scala 3.0.2 compiler can be used by Scala 3.0.1 or 3.1.0. On the other hand, code compiled with Scala 3.1.0 cannot be a dependency of any project compiled with 3.0.2.
While this sounds good, it can be problematic for an intertwined ecosystem. Even though we do what we can to make the transition between minor versions of the compiler as smooth as possible, there will always be projects stuck on older versions for various reasons. That means that they may not be able to update some of their dependencies if said libraries updated themselves to the newer compiler. That, in order, encourages libraries to stay on the oldest possible version of the compiler, which bars them from potentially beneficial improvements in the newer compilers.
Let’s examine the simple example. We have a project that for some reason, is stuck on Scala 3.1. We depend on version 1.3.5 of some library that was compiled with Scala 3.1. The library publishes version 1.4.0 with the compiler updated to 3.2. We cannot update our dependency because of output incompatibility. That also means that all patches for security issues won’t be available unless the library author decides to backport them and release them as version 1.3.6. Knowing all this, the library author would never migrate from Scala 3.1 to 3.2.
We have been discussing and testing various potential solutions in the past few months. Today we want to tell more about the attempts that failed and present the solution that we believe is the best for the future of Scala. The three main criteria we were using to examine potential solutions were:
- The burden on library authors - we don’t want library maintainers and authors to feel overwhelmed by making frequent changes to the compiler that require actions on their side
- The burden on the compiler maintainers - the compiler team has limited resources, so if maintaining compatibility required a considerable amount of work, we would have much less time to spend on improvements and bug fixes in the compiler
- Innovativeness - some features require updating the minor version of the compiler (native image-friendly lazy vals being the latest example), and we want them delivered to the users as soon as possible. Also, we want important bug fixes to be delivered to users regardless of their update policy.
For a short time, we have considered releasing a patch version for each supported past minor line every time a new minor release of the compiler appears. That would allow every minor version of the Scala 3 compiler to accept the output of any other minor version (assuming the latest patch is used). This is something similar to what is happening in Scala 2 in terms of Scala 3 compatibility. However, applying this to all minor lines of Scala 3 would be an enormous amount of work, way beyond our capabilities.
The lighter version of the aforementioned approach was to keep our current compatibility guarantees, but at the same time provide backports of important bug fixes to the previous minor versions. That would still be a massive workload for us, and it would have the additional drawback of discouraging people from updating to the new minor releases.
This was the solution that we invested the most into. It was discussed in detail in [the previous blog post]({{ site.baseurl }}/blog/2022/04/12/scala-3.1.2-released.html#configurable-scala-output-version). To summarize: in 3.1.2, we have shipped a new experimental compiler flag that allowed developers to make the compiler generate output that is usable by older versions of the compiler. While this would allow all projects to receive important fixes, it would probably result in a strange state where all the libraries are released with output version 3.1, similar to how nearly all java libraries are still released with output version 8, even though Java 8 is eight years old.
More importantly, during work on 3.2, we realized that maintaining this flag may be more challenging than anticipated. With time, it will be increasingly harder to be sure that our handling of it is correct. Thus we have decided to remove the possibility of configuring the output version altogether in Scala 3.2.
Finally, we have reached the agreement that the best course of action would be to split Scala development into two lines, called “Scala LTS” and “Scala Next”. Both would live as separate branches in the compiler repository. Both will have separate, but possibly synchronized releases. Porting changes from one to the other should be easy and common.
Scala Next will be the line that the development of the language is taking place on. It would receive frequent minor updates and all experimental language features will leave here.
Scala LTS on the other hand will be the stable long-term support line. One LTS will only receive patch updates which means that all releases of the same LTS line will be forward and backward compatible in terms of output. The only changes between releases will be bug fixes, non-language changes (doctool, semanticDB, reporting), and small quality-of-life enhancements (only if we are sure we are not breaking any compatibility guarantees. After more than two years we may decide to nominate one of the releases from the Next line as the new LTS. After that, we guarantee that the old LTS will be supported for at least one additional year. That means that the time of support for each LTS release is at least three years.
The new model doesn’t change our compatibility guarantees. We are keeping forward and backward compatibility within a single minor version and backward compatibility across different minor versions. That means that the code compiled with Scala LTS can be used as a dependency by all Scala Next versions newer than said LTS, but not the other way around.
Once the LTS & Next model is implemented, we suggest the following practices for different groups of users:
You should stick to the LTS version. It will give you the best tooling support and a stable compiler. If you are migrating a bigger project from Scala 2, you should migrate it straight to the newest LTS.
Unless your library is built around some new language feature that is available only on Next, you should stick to LTS to have the widest possible user base. If your library contains multiple modules, some of them may require Scala Next. It is perfectly fine as long as you keep in mind that modules compiled with Scala LTS cannot depend on them (the other way around is ok). Also, users using your library from Scala LTS won’t be able to access modules built with Next.
Remember that whenever you are bumping a minor version of the compiler (e.g., moving from the previous Scala LTS line to the next), you also need to bump the minor version of your library.
Feel free to use Scala Next. Then you will be able to use all libraries, both those for Scala LTS and Scala Next. You also will be able to test the newest and experimental features of the language.