-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
scalafixOnCompile setting key #140
Conversation
0e81719
to
eb0f7cc
Compare
// We can't just run scalafix from here, as it would introduce a cyclic dependency | ||
// since semantic rules require compilation products. Instead we run scalafix within | ||
// a custom state where compile has not been overriden. | ||
val extracted = Project.extract(currentState) | ||
val withOldCompile = Compat.append( | ||
extracted, | ||
Seq(compile.in(ref, config) := oldCompile), | ||
currentState | ||
) | ||
extracted.runInputTask( | ||
scalafix.in(ref, config), | ||
input = "", | ||
withOldCompile | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to work, but it is very hacky, as:
- the state "reload" is slow
- compilations for a given project/config and most annoyingly all the projects it depends on are not de-duped during the liftetime of a task execution
Still trying to come up with something better (forcing the fullClasspath
value lookup to transitively depend on a non-"custom" compile
).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2. compilations for a given project/config and most annoyingly all the projects it depends on are not de-duped during the liftetime of a task execution
Yes, because runInputTask
is running outside of the task graph, which is dangerous for race conditions. See these references for when sbt-buildinfo used to do the same thing, and the problems it caused: sbt/sbt-buildinfo@7f3a548#diff-0a369498a5a8db3ac8fa606b544c9810
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the pointers, that confirms my concerns! How did you get rid of it there? I am just wondering if it would be applicable here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the implementation so the values were defined within the Task (monad): sbt/sbt-buildinfo@v0.7.0...v0.9.0#diff-60231996cd2e40cc694705deb78dc6d6R27; notice the case BuildInfoKey.TaskValue
, which doesn't do the unsafe extracted.runTask
(the case BuildInfoKey.Task
is still there for backwards-compatibility and extreme cases.)
126e66c
to
b8e35c2
Compare
if (runScalafixAfterCompile) { | ||
// We can't just run scalafix from here, as it would introduce a cyclic dependency | ||
// since semantic rules require compilation products. Instead we run scalafix within | ||
// a custom state where compile has not been overriden. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where's the cycle? val oldCompile = compile.value
means "compile and store the result (CompileAnalysis
) as oldCompile
. So scalafix should find all the compilation products it needs in the right places in the target directories.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
on master: scalafix -> fullClasspath -> compile
introduced in this PR: compile -> scalafix
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, instead of redefining compile
, try something like:
scalafix := Def.taskDyn {
// ...
if (runScalafixAfterCompile)
scalafix.triggeredBy(compile.in(config))
else
scalafix
}.value
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried that earlier but as far as I remember, it did not give me what I wanted: a failure in scalafix
woudn't cascade to compile
, given the fire-and-forget semantics of triggeredBy
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, yes, that is to be expected.
So, thinking back, why does scalafix need fullClasspath but compile doesn't? It looks like (ignoring a bunch of noise and detail) compile is defined in terms of dependencyClasspath
and classDirectory
, and products
(a dependency of fullClasspath
) is defined as
def makeProducts: Initialize[Task[Seq[File]]] = Def.task {
compile.value
copyResources.value
classDirectory.value :: Nil
}
So I wonder if your redefinition of compile could run old compile
and copyResources
, and then run scalafix
, having redefined scalafix
as depending on dependencyClasspath
and classDirectory
, the last of which should have the classfiles from compile
and the resources from copyResources
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
having redefined scalafix as depending on dependencyClasspath and classDirectory, the last of which should have the classfiles from compile and the resources from copyResources.
By removing that dependency, the problem is that scalafix
could run against stale class/semanticDB files, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess there is something to do with scalafixRunExplicitly
, which can control in which direction the dependency is so it's never cyclic for a given task execution, I'll give it a try.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another way to see it is: given scalafix will always depend on compilation, then "runScalafixAfterCompile" roughly means "redefine compile
as scalafix
".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I came up with a reasonable implementation. It is a bit more tricky than I anticipated because the real scalafix
can run any rule given that it's a input key, while the scalafixOnCompile
can only trigger a no-arg scalafix
which uses .scalafix.conf
to find rules to run. That's why I changed a bit the behavior (see last paragraph of first comment).
Is that what you had in mind?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, looks good (only fast-scanned, this late at night 😄).
Nice to know it actually worked! (famous last words...)
Would it be possible to achieve similar functionality using scalafix.in(Compile) :=
scalafix.in(Compile).toTask(" ").triggeredBy(compile in Compile).value, |
@olafurpg see inline discussion: compile wouldn't fail on lint error. I naively expected this behavior for visibility, but maybe it's better that way? |
My apologies! I missed that conversation, makes sense! |
b8e35c2
to
be85a8b
Compare
"To run on test sources use test:scalafix or scalafixAll." | ||
"For example: scalafix RemoveUnused. " + | ||
"To run on test sources use test:scalafix or scalafixAll. " + | ||
"When invoked directly, prior compilation will be triggered for semantic rules." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not very proud of the "When invoked directly" wording but I am trying to get the attention of power users that would call/use scalafix input keys from another context (like another input meta input key that would aggregate scalafmt & scalafix for example). In that case, scalafixRunExplicitly
will be false, so Scalafix might run on stale semanticdb/class files. I haven't found a way to work around that, but I think it's such a corner case and the current implementation is quite readable that it's not worth trying to address it.
Users can control whether scalafix (running the default rules declared in .scalafix.conf) should be run as part of `compile` at the project & configuration level. The value of scalafixOnCompile is ignored when invoking directly scalafix or scalafixAll, as the CLI arguments (if there are any) take precedence over the default rules. That means for example that `scalafix --check` is safe to run even though scalafixOnCompile := true, as there will not be any rewrite triggered by the implicit compilation.
be85a8b
to
7562e24
Compare
// passively consume compilation output without triggering compile as it can result in a cyclic dependency | ||
val classpath = | ||
dependencyClasspath.in(config).value.map(_.data.toPath) :+ | ||
classDirectory.in(config).value.toPath |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have left out copyResources
since I don't see why Scalafix would need it but it might be a wrong assumption.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just tested this locally and it seems to work great! Although I'm personally not a fan of "format/fix on compile", it's pretty clear that a lot of people will love this addition.
I am concerned about redefining compile
even when fix on compile is disabled. I wonder how this interacts with other plugins that also override compile
, for example sbt-scalafmt 🤔 Can users decide the ordering in which formatting/fixing runs?
To minimize the risk of causing unintended issues, I wonder if we should extract this setting in a separate plugin that users need to explicitly enable? I took a stab at implementing this in the diff here https://github.com/scalacenter/sbt-scalafix/compare/master...olafurpg:scalafixOnCompile?expand=1
I'm not sure which approach is better, just thinking out loud. I am totally fine with the approach in this PR with scalafixOnCompile
. You have more experience using this feature so I leave the decision to you
I can't think of any downside redefining As for what happens when I'd tend to stick to the current implementation for simplicity unless we have factual elements that demonstrate its shortcomings. @dwijnand, since you are here, your thoughts on this are more than welcome! |
I think Sébastien said it best (somewhere): if a plugin changes the default build settings/tasks, then it should be opt-in enabled, not auto-trigger from being on the classpath. But it's not a practice that is even documented or even clearly, expressly understood between major sbt plugin authors and sbt maintainers, so it's clearly not strongly observed by plugins in the ecosystem. So, given that, I don't have any immediate feedback to give (reason why I just +1'd). But, eventually, it might become better to make the "on compile" part opt-in. I don't even know what the behaviour would be if you did |
Thanks a lot for the context @dwijnand ! Since we are pre 1.0 (which maybe wasn't the case for @olafurpg should I merge this as-is and you open a PR with your extra commits? Or you just push stuff here (I guess collaborators can do that) ? |
Pinging @sjrd to weigh in on the discussion starting at #140 (review) Do you think that it's acceptable/idiomatic to redefine the |
@bjaglin It's a good point that sbt-scalafmt formats before compile so there shouldn't be an integration issue. However, I suppose users would ideally prefer Scalafmt to format after Scalafix when you enable scalafix+scalafmt on compile. Pinging in @poslegm who has worked on the sbt-scalafmt plugin, I wonder if it's worth considering changing the sbt-scalafmt plugin to make it possible to format after Scalafix completes? |
@olafurpg: @dwijnand explained my position quite well with regards to modifying existing tasks and settings by default. In general, if you're modifying a task or setting that is not defined by your plugin, there's one big no no: do not make your plugin auto-triggered. Because of how plugins get automatically loaded simply because of being on the classpath (including through transitive dependencies), having auto-triggered plugins that modify existing stuff can cause surprising behaviors, sometimes difficult or impossible to disable. That said, the way that Another big danger is that any So overall, from a technical standpoint of overriding Now, you didn't ask my opinion, but I can't comment on a thread like this without giving it anyway: modifying files in a task (either input files or files produced by earlier tasks) is a very bad thing, and you should never do that, ever. The parallelism of sbt means that things can execute concurrently on already created files, causing very bad issues. If you want to modify things, use a command instead (and no, you can't execute a command from a task, not in a sound way anyway). OK, I said it once, I will now hold my peace about this. |
Good point, which I guess goes beyond the scope of this PR as Since the |
nevermind, that wouldn't prevent tasks depending on |
I guess we could still use |
I think it's OK to make changes in sbt-scalafmt because these are projects from the same ecosystem. But I can't figure out yet exactly how to track the scalafix finish from another plugin. |
Thank you @sjrd for pitching in! That was helpful. With all things considered, I think it's OK to merge this PR as is. I'm not convinced it's worth complicating the installation setup with a separate I agree with you that modifying files during compile is a bad thing. I have tried to prevent people from running formatting/fixing on compile since 2016 with little luck so far 😅 As for modifying files from any task I agree in principle but I haven't yet figured out how to implement it in practice so far 🤷 |
@poslegm It's a good question. I feel like it might be possible to achieve this by defining a callback hook in sbt-scalafix that sbt-scalafmt can register. // sbt-scalafix
val scalafixOnCompileHook = taskKey[Seq[File] => Unit]("Callback hook after Scalafix has completed fixing a file")
scalafix := {
val filesToFix = // .. command like usual
scalafixOnComipileHook.?.value.foreach(hook => hook(filesToFix))
}
// sbt-scalafmt
TaskKey[Seq[File] => Unit]("scalafixOnCompileHook") := { files => formatFiles(files) } You can achieve something similar by introducing a direct dependency between the plugins but I don't think that's necessary. Either way, feel free to ignore these thoughts. This is not blocking this PR, I'm just thinking out loud. |
👍 let's follow-up on implementation details in scalacenter/scalafix#1206 |
I'm somewhat wary of compile := Def.taskDyn {
val oldCompile =
compile.value // evaluated first, before the potential scalafix evaluation
val runScalafixAfterCompile =
scalafixOnCompile.value && !scalafixRunExplicitly.value
if (runScalafixAfterCompile)
scalafix
.toTask("")
.map(_ => oldCompile)
else Def.task(oldCompile)
}.value, When Play needed post processing, Josh and I split manipulateBytecode := compileIncremental.value, but you could probably do: Compile / manipulateBytecode := {
val orig = (Compile / manipulateBytecode).value // respect other plugins
println("post processing")
orig
} and here's a dynamic version of that.. lazy val condition = settingKey[Boolean]("")
condition := true
Compile / manipulateBytecode := (Def.taskDyn {
val orig = (Compile / manipulateBytecode).value
if (condition.value) Def.task {
println("post processing")
orig
}
else Def.task { orig }
}).value |
Thanks for chiming-in @eed3si9n!
Could you explain why it's better to redefine (with a dynamic wrapper task ) a subtask of compile rathe than It seems to work just fine, but I didn't manage to write a proper motivation in the commit message, thus my question. |
@bjaglin I don't think there will be noticeable performance impact. One potential benefit of hooking into |
3702195
to
ad08428
Compare
In the case of Scalafix, I think it would be desirable to persist incremental compile analysis files even in case Scalafix fails. As a user, I would be surprised if a Scalafix linting error would cause sbt to re-compile the Scala sources. |
ad08428
to
7562e24
Compare
Random thought: have you considered using a |
We have: it wouldn't fail verbosely in case of a lint error. |
I think I will go ahead with this as-is. Outstanding comments
I'll follow-up with a PR with a tentative fix for that based on #140 (comment).
Given the change that this brings, I feel like it's still better to override
Will be handled separately, see scalacenter/scalafix#1206. |
I agree @bjaglin. LGTM 🚀 |
Users can control whether Scalafix should be run (with the rules defined in
.scalafix.conf
) as part ofcompile,
at the project & configuration level.The value of
scalafixOnCompile
is ignored when invoking directlyscalafix
orscalafixAll
, as the CLI arguments (if there are any) take precedence over the default rules. That means for example thatscalafix --check
is safe to run even thoughscalafixOnCompile := true
, as there will not be any rewrite triggered by the implicit compilation.Enabling
scalafixOnCompile
wil automatically turn on caching, which can be disabled by an explicitscalafixCaching := false
.