Scala sh rewrite #13081

BarkingBad · 2021-07-15T15:56:58Z

No description provided.

BarkingBad · 2021-07-16T11:00:40Z

Does anyone know why we couldn't create val of type Regex then reuse it and we had to inline the regex in the if condition? My last commit changes this and then the CI passes. Previously it was failing with for CompilationTests.tastyBootstrap:

[info] Test dotty.tools.dotc.CompilationTests.tastyBootstrap started
possible data race involving globally reachable variable root in class Pattern: java.util.regex.Pattern#Node
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable matchRoot in class Pattern: java.util.regex.Pattern#Node
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable buffer in class Pattern: Array[Int]
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable predicate in class Pattern: java.util.regex.Pattern#CharPredicate
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable namedGroups in class Pattern: java.util.Map[String, Integer]
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable groupNodes in class Pattern: Array[java.util.regex.Pattern#GroupHead]
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable topClosureNodes in class Pattern: java.util.List[java.util.regex.Pattern#Node]
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable localTCNCount in class Pattern: Int
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable hasGroupRef in class Pattern: Boolean
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable capturingGroupCount in class Pattern: Int
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
possible data race involving globally reachable variable localCount in class Pattern: Int
  use -Ylog:checkReentrant+ to find out more about why the variable is reachable.
Compilation failed for: 'dotty1 from tastyBootstrap/dotty1'                     
Error:  Test dotty.tools.dotc.CompilationTests.tastyBootstrap failed: java.lang.AssertionError: Expected no errors when compiling, failed for the following reason(s):
Error:  
Error:    - encountered 1 test failures(s), took 34.712 sec
Error:      at dotty.tools.vulpix.ParallelTesting$CompilationTest.checkCompile(ParallelTesting.scala:897)
Error:      at dotty.tools.dotc.CompilationTests.$anonfun$2(CompilationTests.scala:288)
Error:      at scala.collection.immutable.List.map(List.scala:250)
Error:      at dotty.tools.dotc.CompilationTests.tastyBootstrap(CompilationTests.scala:288)
Error:      at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:      at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
Error:      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:      at java.lang.reflect.Method.invoke(Method.java:567)
Error:      ...

Also, for coursier tests I need artifacts to be published. Is it OK @smarter or I should redesign it, for example move it to the separate CI action?

smarter · 2021-07-16T14:08:37Z

Does anyone know why we couldn't create val of type Regex

The analysis done by CheckReentrant is imprecise and incorrectly thinks that java regexps are not thread-safe, but it's actually fine to use them, so we just turn off the check using @sharable: https://github.com/lampepfl/dotty/blob/839ef3203e654a38b431558eea52821c974a1734/compiler/src/dotty/tools/repl/ParseResult.scala#L114 (though we could also special-case regexps in CheckReentrant)

Also, for coursier tests I need artifacts to be published. Is it OK @smarter or I should redesign it

What's important is that if I run test locally, all tests should succeed, so we can't have tests enabled by default that require some extra step.

compiler/src/dotty/tools/MainGenericRunner.scala

smarter · 2021-07-16T14:10:40Z

compiler/src/dotty/tools/MainGenericRunner.scala

+  def process(args: List[String], settings: Settings): Settings = args match
+    case Nil =>
+      settings
+    case "-repl" :: tail =>


In which situation is it necessary to explicitly pass -repl?

Actually, never. But dist/bin script provides such option, so we wanted to recreate this

If it's never used let's just remove it then.

In which situation is it necessary to explicitly pass -repl?

In dist/bin/scalac, it's needed, but doesn't seem to be needed in dist/bin/scala.

compiler/src/dotty/tools/MainGenericRunner.scala

smarter · 2021-07-16T14:24:46Z

The last commit committed .class and .tasty files that shouldn't be in git.

BarkingBad · 2021-07-16T14:55:37Z

The last commit committed .class and .tasty files that shouldn't be in git.

I did it on purpose for testing -run but I can recompile the .scala file indeed.

lrytz · 2021-07-19T11:50:36Z

This PR addresses coursier/coursier#2058, right?

BarkingBad · 2021-07-19T17:13:52Z

@lrytz I was not aware about this issue. However I am afraid it will take some time till we get it released to maven central, so the coursier descriptor can be created for scala3 as it will be release with 3.0.3

smarter · 2021-07-19T17:20:13Z

if this is deemed important enough, we could backport it to the 3.0.2 branch.

…on, apply requested changes

philwalk · 2021-07-20T00:37:18Z

@BarkingBad - I like the idea of migrating functionality from sh script to scala code, although I'm not entirely clear as to the scope of this PR. Will there be support for jline console functionality?

BarkingBad · 2021-07-20T09:09:12Z

What do you mean @philwalk? If you mean that coursier tests cannot initialize jline I think it's the internal problem from calling in the virtual terminal. Calling REPL works just fine for both shell script and when we eventually turn it into coursier executable

philwalk · 2021-07-20T15:29:16Z

What do you mean @philwalk?

I'm not referring specifically to the coursier use-case, but to whether and how a Windows cygwin or msys environment (for example) might be affected by this PR. Possibly I just need to understand some context.

When I launch a REPL session from a cygwin bash terminal, here's what I see with these changes:

# philwalk@d5 ~/workspace/dotty-sh-rewrite
$ bin/scala
Starting scala3 REPL...
←[?1h←=←[90m~←[0m
←[?2004h←[34mscala> ←[0m:q
:q
←[?1l←>←[?1000l←[?2004l

By contrast, here's what happens with the current scala release:

$ /opt/scala3/bin/scala
scala> :q

Also, there is some other lost functionality compared to the current release (e.g., SCALA_OPTS is no longer supported).

BTW, I developed a scala implementation of dist/bin/common that I can contribute, if such a thing makes sense here.

philwalk · 2021-07-21T22:56:29Z

A fix for the cygwin console problem is to copy the following line from dist/bin/scalac line 72 to dist/bin/scala just before calling java (just before the eval $JAVACMD, near line 32)

[ -z "${ConEmuPID-}" -o -n "${cygwin-}" ] && export MSYSTEM= PWD= # workaround for #12405

BarkingBad · 2021-07-22T13:16:47Z

Thank you for your response @philwalk. I will try to setup VM Windows to test it as well when I settle up with other issues

philwalk · 2021-07-22T19:01:54Z

@BarkingBad
We're actually closer than I thought. Here are some remaining gaps in the implementation, AFAICT:

The -Dscript.path=<script-path> property needs to be set in dist/bin/scala, and the script compiler/test-resources/scripting/scriptPathh.sc should have caused the build to fail. An updated version of BashScriptsTests has a new test to verify this property #13132.

The decision as to whether an argument is a <script-path> should be generous and accept an argument even if the specified file does not exist; otherwise, when an interactive user misspells a script path, all of the arguments intended for the script will be sent to the compiler, resulting in incomprehensible error messages, or worse.

The release version of dist/bin/scala discards scala_args when launching REPL mode, since they are intended as the args that go to scala when running in script mode. It would be nice if they could be handled properly in REPL mode as well, although that hasn't been implemented yet.

BarkingBad · 2021-07-27T11:58:15Z

Hi @philwalk

I implemented your cygwin fix.

Also referring to your comment #13081 (comment) the stacktrace is from the Run branch since I didn't implement @ arguments, therefore he tried to run the classpath as a compiled file. (As you can see it does not enter dotty.tools.scriptin.Main.main in the stacktrace)

The second comment #13081 (comment) about script.path is in my opinion not valid in terms of what we are trying to do, which is not to process args in bash script but in scala entry Main. I added setting up the value of script.path in the scripting.Main and also test it in CoursierTests

I rebased onto your branch with tests (since they are not merged yet) and ran these tests locally - all passed 😄

philwalk · 2021-07-27T14:14:06Z

@BarkingBad

I added setting up the value of script.path in the scripting.Main

Yes, that's better. Everthing looks good to me!

BarkingBad · 2021-07-28T08:43:52Z

Could you take a look @philwalk one more time and if everything is OK give approval? I need it to merge the changes

philwalk

@BarkingBad - everything looks good, I only have one suggestion:

in CoursierScalaTests.scala, the scriptTest only verifies that the property is defined, it could also verify that the value endsWith scriptPath.sc.

bishabosha · 2021-08-02T12:36:54Z

can we backport this?

BarkingBad · 2021-08-02T12:45:34Z

I think we could, it would be beneficial for all coursier users I guess

dwijnand · 2021-08-02T13:02:51Z

I'd say no: it's ~500 lines of fresh code, let's leave the backports for the critical bugfixes.

SethTisue · 2021-08-11T04:26:46Z

Hey, this is great! 👏

I was able to figure out how to use this from Coursier today, with a little extra effort; see coursier/coursier#2058 (comment) for details

MainGenericRunner should work almost

4b2ac53

BarkingBad force-pushed the scala-sh-rewrite branch 2 times, most recently from 46dbb4f to 86095f7 Compare July 15, 2021 16:32

Add coursier tests

2c46c3e

BarkingBad force-pushed the scala-sh-rewrite branch from 86095f7 to 2c46c3e Compare July 15, 2021 17:15

Fix tests

630579b

BarkingBad force-pushed the scala-sh-rewrite branch from 33991ab to 630579b Compare July 16, 2021 10:18

BarkingBad marked this pull request as ready for review July 16, 2021 11:00

BarkingBad requested review from smarter and philwalk July 16, 2021 11:01

smarter reviewed Jul 16, 2021

View reviewed changes

compiler/src/dotty/tools/MainGenericRunner.scala Outdated Show resolved Hide resolved

smarter reviewed Jul 16, 2021

View reviewed changes

compiler/src/dotty/tools/MainGenericRunner.scala Show resolved Hide resolved

Handle run setting in scala script

216b70b

BarkingBad force-pushed the scala-sh-rewrite branch 3 times, most recently from 1fae969 to 04af902 Compare July 19, 2021 19:00

Handle save scala option, move coursier tests to isolated configurati…

72c279f

…on, apply requested changes

BarkingBad force-pushed the scala-sh-rewrite branch from 04af902 to 72c279f Compare July 19, 2021 22:01

Fix running REPL from the shell script

0c340bc

dwijnand mentioned this pull request Jul 27, 2021

Scala runner doesn't do executable jars #12569

Closed

BarkingBad force-pushed the scala-sh-rewrite branch from e8fc083 to 7e43057 Compare July 27, 2021 11:40

BarkingBad force-pushed the scala-sh-rewrite branch 2 times, most recently from 2275440 to ca5b9a7 Compare July 27, 2021 12:40

Fix cygwin problems. Handle @ and -color settings.

12eeb9f

BarkingBad force-pushed the scala-sh-rewrite branch from ca5b9a7 to 12eeb9f Compare July 27, 2021 12:55

anatoliykmetyuk assigned BarkingBad Jul 27, 2021

Fix coursier test

f3a72f2

philwalk approved these changes Jul 28, 2021

View reviewed changes

Test if script path provided by env var is correct

314c3c9

BarkingBad enabled auto-merge August 2, 2021 09:47

BarkingBad merged commit d0c0eff into scala:master Aug 2, 2021

bishabosha added the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Aug 2, 2021

SethTisue mentioned this pull request Aug 11, 2021

cs does not install scala runner for scala 3 coursier/coursier#2058

Closed

This was referenced Aug 11, 2021

use coursier instead of sbt for Scala 3 scripts scala/community-build#1466

Merged

ScriptMain doesn't work with Scala 3 sbt/sbt#6274

Open

griggt mentioned this pull request Aug 20, 2021

scala 3 repl does not work with -cp argument, needs -repl as workaround #12973

Closed

BarkingBad mentioned this pull request Sep 3, 2021

Basic option processing is still broken for REPL #13443

Closed

Kordyjan removed the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Jul 26, 2022

Kordyjan added this to the 3.1.0 milestone Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala sh rewrite #13081

Scala sh rewrite #13081

BarkingBad commented Jul 15, 2021

BarkingBad commented Jul 16, 2021

smarter commented Jul 16, 2021

smarter Jul 16, 2021

BarkingBad Jul 16, 2021

smarter Jul 16, 2021

philwalk Jul 16, 2021

smarter commented Jul 16, 2021

BarkingBad commented Jul 16, 2021

lrytz commented Jul 19, 2021

BarkingBad commented Jul 19, 2021

smarter commented Jul 19, 2021

philwalk commented Jul 20, 2021 •

edited

Loading

BarkingBad commented Jul 20, 2021

philwalk commented Jul 20, 2021 •

edited

Loading

philwalk commented Jul 21, 2021

BarkingBad commented Jul 22, 2021

philwalk commented Jul 22, 2021 •

edited

Loading

BarkingBad commented Jul 27, 2021 •

edited

Loading

philwalk commented Jul 27, 2021 •

edited by bishabosha

Loading

BarkingBad commented Jul 28, 2021

philwalk left a comment

bishabosha commented Aug 2, 2021

BarkingBad commented Aug 2, 2021

dwijnand commented Aug 2, 2021

SethTisue commented Aug 11, 2021 •

edited

Loading

Scala sh rewrite #13081

Scala sh rewrite #13081

Conversation

BarkingBad commented Jul 15, 2021

BarkingBad commented Jul 16, 2021

smarter commented Jul 16, 2021

smarter Jul 16, 2021

Choose a reason for hiding this comment

BarkingBad Jul 16, 2021

Choose a reason for hiding this comment

smarter Jul 16, 2021

Choose a reason for hiding this comment

philwalk Jul 16, 2021

Choose a reason for hiding this comment

smarter commented Jul 16, 2021

BarkingBad commented Jul 16, 2021

lrytz commented Jul 19, 2021

BarkingBad commented Jul 19, 2021

smarter commented Jul 19, 2021

philwalk commented Jul 20, 2021 • edited Loading

BarkingBad commented Jul 20, 2021

philwalk commented Jul 20, 2021 • edited Loading

philwalk commented Jul 21, 2021

BarkingBad commented Jul 22, 2021

philwalk commented Jul 22, 2021 • edited Loading

BarkingBad commented Jul 27, 2021 • edited Loading

philwalk commented Jul 27, 2021 • edited by bishabosha Loading

BarkingBad commented Jul 28, 2021

philwalk left a comment

Choose a reason for hiding this comment

bishabosha commented Aug 2, 2021

BarkingBad commented Aug 2, 2021

dwijnand commented Aug 2, 2021

SethTisue commented Aug 11, 2021 • edited Loading

philwalk commented Jul 20, 2021 •

edited

Loading

philwalk commented Jul 20, 2021 •

edited

Loading

philwalk commented Jul 22, 2021 •

edited

Loading

BarkingBad commented Jul 27, 2021 •

edited

Loading

philwalk commented Jul 27, 2021 •

edited by bishabosha

Loading

SethTisue commented Aug 11, 2021 •

edited

Loading