Outline Typing

[retronym]

A new mode for PipelineMain that can shorten the critical path of a build and extract more parallelism from the post-typer compiler phases (at the cost of instantiating multiple instances of Global).

First, we compile all sources with -Youtline -Ystop-after:pickler. The first option disables typechecking of the RHS of definitions that have ascribed types.

The RHS is not skipped when it contains a super call that might need a super accessor. (hat tip to @smarter for finding this corner case in dotty). Are there more like it that we haven't identified?

The pickles are then exported and made available to downstream compilation, as is also the case with the strategy=Pipeline. The sources for the current projects are then compiled in parallel in chunks, also using the exported pickles on the classpath.

Optionally, we can share the name table between Global instances to reduce the memory and bookkeeping overhead of the traditional approach of having this inside the cake. For backwards binary compatibility reasons, this is implemented with an abundance of casts and other hacks.

Testing this out guardian-frontend. In the past I've experimented with the full multi-module build, but in this case I want to focus on the common project that all other depend on and that is the bottleneck for a full clean build.

Pipeline + Outline

$ mkdir -p /tmp/pickle-cache && ./build/pack/bin/scala -J-Dsun.reflect.inflationThreshold=0  -J-Xmx4G -Dscala.pipeline.cache.name.table=true -Dscalac.filebasedcache.defer.close.ms=30000 -Dscala.pipeline.picklecache=/tmp/pickle-cache -Dscala.pipeline.group.size=128  -Dscala.pipeline.strategy=outlinetypepipeline -Dscala.pipeline.strip.external.classpath=true -Dscala.pipeline.cache.macro.classloader=true -Dscala.pipeline.use.jar=true scala.tools.nsc.PipelineMainTest /Users/jz/code/guardian-frontend/common/target/compile.args
...
====== ITERATION 20=======
parallelism = 8, strategy = OutlineTypePipeline
Exported external classpath in 4 ms
Wrote project dependency graph to: /Users/jz/code/scala/projects.dot
/Users/jz/code/guardian-frontend/common:compile: scalac outline: start
/Users/jz/code/guardian-frontend/common:compile: scala outline: done 1378 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (1/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (2/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (4/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (3/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (4/4): done 1729 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (3/4): done 1898 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (1/4): done 3074 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (2/4): done 4698 ms
PROGRESS: 6 / 6
 Wall Clock: 6077 ms
Chrome trace written to /Users/jz/code/scala/build-OutlineTypePipeline-20.trace

Pipeline

$ ... -Dscala.pipeline.strategy=pipeline ...
====== ITERATION 20=======
parallelism = 8, strategy = Pipeline
Exported external classpath in 4 ms
Wrote project dependency graph to: /Users/jz/code/scala/projects.dot
/Users/jz/code/guardian-frontend/common:compile: scalac: start
/Users/jz/code/guardian-frontend/common:compile: scalac outline: done 4232 ms
/Users/jz/code/guardian-frontend/common:compile: scalac: exported pickles 52 ms
/Users/jz/code/guardian-frontend/common:compile: scalac: done 4523 ms
PROGRESS: 3 / 3
 Wall Clock: 8807 ms
Chrome trace written to /Users/jz/code/scala/build-Pipeline-20.trace

Traditional

$ ... -Dscala.pipeline.strategy=traditional ...
====== ITERATION 20=======
parallelism = 8, strategy = Traditional
Wrote project dependency graph to: /Users/jz/code/scala/projects.dot
/Users/jz/code/guardian-frontend/common:compile: scalac (1/1): start
/Users/jz/code/guardian-frontend/common:compile: scalac (1/1): done 9343 ms
PROGRESS: 3 / 3
Wall Clock: 9409 ms
Chrome trace written to /Users/jz/code/scala/build-Traditional-20.trace

Summary

• The main benefit of the new strategy here is reducing the "time-to-pickles" down to 1.4s (vs 9.3 for traditional vs 4.2 for non-outline pipeline)
• The wall clock time is also reduced somewhat (6.1s vs 8.8s), which may be interesting in itself when enough cores are available. There will be rapidly diminishing returns as the chunk size gets too small, however, due to the current approach of using a Global instance per-chunk.

Caveats

• Incompatible with the non-local optimizations in the backend as .class files for symbols backed by exported pickles aren't available. This shouldn't be a big problem in "dev mode".
• To get the most of out of the caching of macro classloaders, a hand crafted -Ymacro-classpath setting is needed with the macro implementation.

[retronym]

/cc @smarter @jvican

[smarter]

The RHS is not skipped when it contains a super call that might need a super accessor. (hat tip to @smarter for finding this corner case in dotty). Are there more like it that we haven't identified?

I think protected accessors are generate pre-pickler in Scala 2 (in Dotty they aren't, but I'm not 100% sure that handles all cases correctly), so that probably leads to the same issue we have with super accessors.

Regarding performance, in scala/scala3#4767 I observed that the lock in https://github.com/lampepfl/dotty/blob/29aeb02be0c08a20373027a0d80deaa87cb699ff/compiler/src/dotty/tools/dotc/classpath/ZipAndJarFileLookupFactory.scala#L181 (which corresponds to https://github.com/scala/scala/blob/bec2441a24cbf3159a1118049b857a5ae7c452eb/src/compiler/scala/tools/nsc/classpath/ZipAndJarFileLookupFactory.scala#L253 in scala/scala) seemed to be a bottleneck, but I haven't had time to experiment with alternative approaches like having an option to make the cache per compiler instance.

[retronym]

@smarter AFAICT protected accessors are only called from elsewhere in the same compilation unit which wouldn't be a problem. We should follow dotty's lead in Scala 2.14. and move generation post-pickler IMO.

I've never really looked at this area before. Some aspects look wrong.

We don't emit scala protected methods as ACC_PROTECTED, apparently because "protected cannot be used, since inner classes 'see' protected members,". But can't we just use protected accessors for these? (Often these aren't actually needed if the callsite and definition are in the same package)

So that leaves test/files/jvm/protectedacc.scala with examples of times we actually do emit a protected accessor. The only one generated in that test is for:

      class JavaInteraction extends java.io.CharArrayReader {
        <artifact> def protected$count(x$1: p.b.JavaInteraction): Int = x$1.count;
        <artifact> def protected$setcount(x$1: p.b.JavaInteraction, x$2: Int): Unit = x$1.count = x$2;
        <paramaccessor> private[this] val arr: Array[Char] = _;
        def <init>(arr: Array[Char]): p.b.JavaInteraction = {
          JavaInteraction.super.<init>(arr);
          ()
        };
        class Inner extends scala.AnyRef {
          def <init>(): JavaInteraction.this.Inner = {
            Inner.super.<init>();
            ()
          };
          def m: Unit = {
            scala.Console.println("count before: ".+(JavaInteraction.this.protected$count(JavaInteraction.this)));
            JavaInteraction.this.protected$setcount(JavaInteraction.this, JavaInteraction.this.protected$count(JavaInteraction.this).+(1));
            scala.Console.println("count after: ".+(JavaInteraction.this.protected$count(JavaInteraction.this)))
          }
        }
      };

It seems to me like protected$count should be emitted as (Scala) private, in order to give it an expanded name later on.

For bonus points we would emit it as a static method like Java's access$N methods.