Default to emitting .tasty files (not .hasTasty) #4467

nicolasstucki · 2018-05-05T12:40:09Z

TODO

Default to emitting .tasty files
Fix filesystem bug when reading tasty in jars (partally?)

allanrenucci · 2018-05-22T15:20:57Z

compiler/test/dotty/tools/vulpix/ParallelTesting.scala

@@ -404,8 +404,11 @@ trait ParallelTesting extends RunnerOrchestration { self =>
      tastyOutput.mkdir()
      val flags = flags0 and ("-d", tastyOutput.getAbsolutePath) and "-from-tasty"

-      def hasTastyFileToClassName(f: JFile): String =
-        targetDir.toPath.relativize(f.toPath).toString.dropRight(".hasTasty".length).replace('/', '.')
+      def hasTastyFileToClassName(f: JFile): String = {


name is now misleading

allanrenucci · 2018-05-22T15:22:27Z

compiler/test/dotty/tools/vulpix/ParallelTesting.scala

+      def hasTastyFileToClassName(f: JFile): String = {
+        val pathStr = targetDir.toPath.relativize(f.toPath).toString.replace('/', '.')
+        if (pathStr.endsWith(".hasTasty")) pathStr.dropRight(".hasTasty".length)
+        else pathStr.dropRight(".tasty".length)


.stripSuffix(".hasTasty").stripSuffix(".hasTasty")?

allanrenucci · 2018-05-24T09:00:36Z

compiler/src/dotty/tools/dotc/transform/PatternMatcher.scala

@@ -700,7 +700,7 @@ object PatternMatcher {
          }
          val newArgs =
            for {
-              (rhs, actual) <- seenVars.toList
+              (rhs, actual) <- seenVars.toList.sortBy(_._2.name.toString)


We have an ordering on Name (i.e. Names.NameOrdering), you shouldn't need to do toString

This is just to test if this is the source of the idempotency failure.

dottybot · 2018-06-29T12:57:05Z

performance test scheduled: 30 job(s) in queue, 1 running.

dottybot · 2018-06-29T16:33:37Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4467/ to see the changes.

Benchmarks is based on merging with master (bdf3ef4)

nicolasstucki · 2018-07-07T16:53:29Z

Rebased

nicolasstucki · 2018-07-16T09:08:01Z

@allanrenucci could you review the last commit

allanrenucci · 2018-07-16T11:49:03Z

compiler/src/dotty/tools/dotc/core/classfile/ClassfileParser.scala

+              val path = classfile.path.stripSuffix(".class") + ".tasty"
+              val stream = cl.getResourceAsStream(path)
+              if (stream != null) {
+                val tasty = new Array[Byte](stream.available())


From the Javadoc

Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not. It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.

allanrenucci · 2018-07-16T11:50:35Z

compiler/src/dotty/tools/dotc/core/classfile/ClassfileParser.scala

-              val jarFile = JarArchive.open(io.File(jar.jpath))
-              try readTastyForClass(jarFile.jpath.resolve(classfile.path))
-              finally jarFile.close()
+              val cl = new URLClassLoader(Array(jar.jpath.toUri.toURL))


Are you sure we won't have to deal with the same issues using a URLClassLoader?

I would say no because the classloader never instanciates an class with a reference to it. Hence the classloader should be GCed at some point. I will experiment a bit with the profiler to see if I can observe any leaks.

Not only leaks. Can the jar be read using an URLClassLoader while at the same time it is read using nio.file.FileSystems. What if someone close the FileSystems while you're reading the jar with URLClassLoader?

I tried it and it does not affect the class loader. I closed the file system in the middle of reading from the stream and some other places.

allanrenucci · 2018-07-17T09:16:21Z

compiler/src/dotty/tools/dotc/core/classfile/ClassfileParser.scala

+              if (stream != null) {
+                val tasty = Array.newBuilder[Byte]
+                tasty.sizeHint(stream.available())
+                while (stream.available() > 0) // TODO improve performance


I think the proper way to read from an InputStream, is to read() until -1 is returned

Solution based on http://www.gregbugaj.com/?p=283&cpage=1

nicolasstucki · 2018-07-19T07:48:29Z

Rebased

odersky · 2018-07-19T09:29:25Z

Ready to merge?

nicolasstucki · 2018-07-19T09:36:07Z

It is ready to merge. But after it is merged everyone needs to clean their out/ directory (to run sbt dotty-bootstrapped/...).

nicolasstucki · 2018-07-19T13:01:37Z

I ran the sbt-dotty/scripted tests locally and source-dependencies / continuations failed. Will test it on the CI

nicolasstucki · 2018-07-19T13:58:52Z

sbt-dotty/scripted passed on the CI http://dotty-ci.epfl.ch/lampepfl/dotty/6302

nicolasstucki added the stat:wip label May 5, 2018

nicolasstucki self-assigned this May 5, 2018

Blaisorblade changed the title ~~Always emit .tasty files (not .hasTasty)~~ Default to emitting .tasty files (not .hasTasty) May 5, 2018

nicolasstucki closed this May 7, 2018

nicolasstucki reopened this May 22, 2018

allanrenucci reviewed May 22, 2018

View reviewed changes

nicolasstucki force-pushed the always-emit-tasty branch 5 times, most recently from 94c8c88 to 5de4a9d Compare May 24, 2018 06:48

allanrenucci reviewed May 24, 2018

View reviewed changes

nicolasstucki force-pushed the always-emit-tasty branch 6 times, most recently from 8de6223 to ab6d5fa Compare May 24, 2018 15:34

nicolasstucki force-pushed the always-emit-tasty branch 3 times, most recently from 3a3cc05 to 930cff2 Compare June 18, 2018 16:44

nicolasstucki force-pushed the always-emit-tasty branch 5 times, most recently from f7fb52d to 308b1fb Compare June 21, 2018 09:58

nicolasstucki added stat:needs review and removed stat:wip labels Jun 21, 2018

nicolasstucki requested a review from allanrenucci June 21, 2018 11:21

nicolasstucki removed their assignment Jun 21, 2018

smarter mentioned this pull request Jul 6, 2018

[WIP] Parallelize the compiler via two-pass compilation #4767

Closed

nicolasstucki force-pushed the always-emit-tasty branch from c94b032 to c5d2c6b Compare July 7, 2018 16:53

allanrenucci reviewed Jul 16, 2018

View reviewed changes

nicolasstucki force-pushed the always-emit-tasty branch from dd7a295 to 4f05fbd Compare July 16, 2018 14:57

allanrenucci reviewed Jul 17, 2018

View reviewed changes

allanrenucci approved these changes Jul 18, 2018

View reviewed changes

nicolasstucki mentioned this pull request Jul 19, 2018

Check tasty file hash #4809

Merged

nicolasstucki added 7 commits July 19, 2018 09:48

Always emit .tasty files (not .hasTasty)

6adce71

Make tests support .hasTasty and .tasty files

374aa8b

Change -Yemit-tasty to -Yemit-tasty-in-class and invert semantics

a645566

Fix loading tasty file from jar in concurrent setting

b328403

Rename isHasTastyFile to isTastyFile

e939049

Use classloader instead of jar filesystem to load tasty

609d017

Improve reading from InputStream

de291c1

Solution based on http://www.gregbugaj.com/?p=283&cpage=1

nicolasstucki force-pushed the always-emit-tasty branch from 6641938 to de291c1 Compare July 19, 2018 07:48

nicolasstucki added stat:do not merge and removed stat:do not merge stat:needs rebase labels Jul 19, 2018

nicolasstucki force-pushed the always-emit-tasty branch from 578b580 to de291c1 Compare July 19, 2018 13:59

nicolasstucki merged commit fe8d050 into scala:master Jul 19, 2018

allanrenucci deleted the always-emit-tasty branch July 19, 2018 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default to emitting .tasty files (not .hasTasty) #4467

Default to emitting .tasty files (not .hasTasty) #4467

nicolasstucki commented May 5, 2018 •

edited

Loading

allanrenucci May 22, 2018

allanrenucci May 22, 2018

allanrenucci May 24, 2018

nicolasstucki May 24, 2018

dottybot commented Jun 29, 2018

dottybot commented Jun 29, 2018

nicolasstucki commented Jul 7, 2018

nicolasstucki commented Jul 16, 2018

allanrenucci Jul 16, 2018

allanrenucci Jul 16, 2018

nicolasstucki Jul 17, 2018

allanrenucci Jul 17, 2018

nicolasstucki Jul 18, 2018

allanrenucci Jul 17, 2018

nicolasstucki Jul 18, 2018

nicolasstucki commented Jul 19, 2018

odersky commented Jul 19, 2018

nicolasstucki commented Jul 19, 2018 •

edited

Loading

nicolasstucki commented Jul 19, 2018

nicolasstucki commented Jul 19, 2018

Default to emitting .tasty files (not .hasTasty) #4467

Default to emitting .tasty files (not .hasTasty) #4467

Conversation

nicolasstucki commented May 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dottybot commented Jun 29, 2018

dottybot commented Jun 29, 2018

nicolasstucki commented Jul 7, 2018

nicolasstucki commented Jul 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolasstucki commented Jul 19, 2018

odersky commented Jul 19, 2018

nicolasstucki commented Jul 19, 2018 • edited Loading

nicolasstucki commented Jul 19, 2018

nicolasstucki commented Jul 19, 2018

nicolasstucki commented May 5, 2018 •

edited

Loading

nicolasstucki commented Jul 19, 2018 •

edited

Loading