Better support for running scripts.

[bacchanalia]

cabal run can run scripts, but the utility of this is limited because under the current design it recompiles on every invocation.

I propose adding the following:

• fake packages for scripts to be written under ~/.local/share (or os equivalent)
  • specifically ~/.local/share/cabal/<abs path of script without extension>/ to avoid collisions
• the option --tmp-script-artifacts (current behaviour. keep as default?)
• the option --persist-script-artifacts (new behaviour)
• the option --script-artifact-dir=DIR (place fake package in user supplied directory)
• script support for cabal build and cabal repl
• updates to the relevant docs

I have done a quick test with hard coded paths, and it works as expected without further changes. The only changes to run necessary are to add the flags and pass the correct directory based on them. I haven't scoped out the work needed to add support to build and repl, but I assume it would be very similar to how it works now in run.

[jneira]

Related:

• cabal triggers a full build each time a script is executed #6354
• Meta-issue: Remove need for fake-package #6977
• cabal v2-repl scriptfile.hs #6149
• Avoid linking of custom Setup scripts #7393

EDIT: Added #7393 as suggested by @Mikolaj

[Mikolaj]

also #7393

[jneira]

the option --tmp-script-artifacts (current behaviour. keep as default?)
the option --persist-script-artifacts (new behaviour)
the option --script-artifact-dir=DIR (place fake package in user supplied directory)

bikeshedding:

• no option set or -no-persist-fake-package > current behavior
• persist-fake-package > save to ./local/share
• persist-fake-package = dir > save to dir

Trying to generalize the behaviour for fake packages and minimize the number of flags, independently of where are being used.
Not sure if the generalization makes sense, in that case replace persist-fake-package by persist-script-build.
Otoh the proper solution should be remove the need of fake-package at all in the entire codebase (see #6977), which could open the door to other improvements

[bacchanalia]

When else are fake packages used other than to build scripts?

edit: This is discussed in issue #6977. I shouldn't comment before coffee.

[bacchanalia]

Cache folders on different OS's (please correct me if I'm wrong):

• on Linux save to ~/.cache (not ~/.local/share)
• on Windows save to %LOCALAPPDATA%
• on OSX save to ~/Library/Caches

It probably makes sense to add a utility function (in Distribution.Simple.Utils?) to get the cache folder.
Rather than putting the fake-package directory structures directly in <cache>/cabal, it might be a better idea to put them in <cache>/cabal/fake-packages in case other uses for <cache>/cabal come up.

[bacchanalia]

@Mikolaj I'm not sure I see the relevance of #7393?

@fendor What's the status of the work you were doing on #6149?

@jneira re #6977: I'm not sure caching makes sense for other uses of fake package. At the very least you'd need a completely separate scheme for where to place/how to identify cached instances.

[Mikolaj]

@bacchanalia: re #7393, you complain "it recompiles on every invocation", and ghc --run would be able to execute a script without producing any compilation artefacts (and without linking) at all. Therefore, it could potentially be used as, in addition to or instead of --tmp-script-artifact. If we assume scripts are never computationally intensive, perhaps it could be used instead of cabal run always? Am I missing something?

[bacchanalia]

@Mikolaj running a script with cabal run allows you do have a cabal block

{- cabal:
build-depends: ...
-}

while ghc --run presumably just uses the global environment.

If we assume scripts are never computationally intensive

I don't think this is a good assumption. Scripts and packages have different ergonomics, and when you might prefer one over the doesn't neatly fall on that boundary, for me, anyway.

Another consideration is start-up time. On my machine running hello.hs with runghc takes 25ms, while cached cabal run only takes 15ms (and hopefully that could go lower, but that's out of scope) which is important in some applications.

[Mikolaj]

Fine. And what about ghc --run (or --eval, but I don't remember the details) underlying --tmp-script-artifact? Is there a value in the temporary artifacts being written somewhere at all?

[fendor]

What's the status of the work you were doing on #6149?

Stalled. I don't know how much everyone else knows about this feature, so I'll repeat a bit how this currently works: The script section (which is quite literally an executable section from a .cabal file [btw, you can break it if you put main-is: SomeName.hs into the script section]) is parsed and based on it, a .cabal file is generated and written to a project file in $TMP. The source is copied to that directory as well, iirc, and then cabal build && cabal exec is basically executed on that project in $TMP.

Doing the same for v2-repl File.hs doesn't cut it, since updates to our original script file won't be noticed, we lose the current working directory, etc... I played around with workarounds, e.g. after invoking cabal v2-repl in that directory, cd to the previous working directory, and listing the full path to the script instead of copying the source script... I can't remember whether there was a blocking issue with cding and listing the full path to the source file, but it was so weird that I wondered whether I would make everything worse by applying duct-tape to this design and stopped with my experiments.

[bacchanalia]

@Mikolaj That might be preferable in the --tmp-script-artifact case, but there might be some value in having consistency between the cases both in terms of implementation and use? Also a user might want to inspect the output. I don't really feel strongly about it either way.

[Mikolaj]

Sure, it's your call as the implementer and, personally, I'd not expend any extra effort (unless minimal) until users express interest.

[bacchanalia]

@fendor In the thread for #6149 you mention that stack handles :r correctly. Did you investigate what they do?

[fendor]

Did you investigate what they do?

No

[jneira]

Well, i think the goal is clear: avoid script recompilation in each run (as reported in #6354).
But i would try to not make the cli for caching the script compilation depends on the existence of a fake package, as it will be removed sooner or later.

Imho cabal runscript.hs should be cached by default. Like cabal build or cabal install cache everything by default without any special flag. The cache should be invalidated automatically in front of source changes, maybe adding cabal clean runscript.hs to consistently wipe out a corrupted cache.

That behaviour could use underneath the cache of fake-package build products for now. If fake-package is removed it could cache only the executable itself and nothing more. But no cli will be harmed in the process :-)

[jneira]

cabal repl script.hs and cabal build script.hs is another different beast, it will suppose change those commands and dont know how cabal build could be done without writing to disk the build products as the intent afaiu is only build the script without executing it but reuse all the work done. So they will need the fake-package or something similar in a known location.

i think if we would have the script cached cabal build would have less sense? we could use cabal run script.hs --only-dependencies

But i think cabal repl script.hs could be done on the fly without writing anything to disk without fake-package.

So i would separate these things:

1. cache script using fake packages as described above
2. change cabal repl to add support for scripts
3. consider add support in cabal build. I think there are some value in being consistent between commands and make them work with the same targets.

And i would mark the first one as pr welcomed right now if everyone agree

[fendor]

But i think cabal repl script.hs could be done on the fly without writing anything to disk without fake-package.

Currently we have to go over lib:Cabal (as we need to build stuff and that only happens in lib:Cabal), we can't pass an in-memory PackageDescription to lib:Cabal, so to have external dependencies, we currently have to serialise a PackageDescription to disk and point lib:Cabal to it.
However, I agree, we should do this on-the-fly without writing anything to disk. This requires us (probably) to remove fake-package logic and provide a proper implementation in exe:cabal (violating a bit the separation between lib:Cabal and exe:cabal, but who likes separation of concern anyway? ;D).

[bacchanalia]

i think if we would have the script cached cabal build would have less sense?

The point of cabal build would precisely be to cache the script so you can choose to pay the compilation time a different time from when you run it.

[bacchanalia]

@fendor I was just looking at this code. exe:cabal calls itself, which then uses lib:Cabal as a library. I don't really understand why that's necessary and the the original invocation of exe:cabal can't just use the library directly.

[jneira]

The point of cabal build would precisely be to cache the script so you can choose to pay the compilation time a different time from when you run it.

yeah thanks for confirming my guess:

dont know how cabal build could be done without writing to disk the build products as the intent afaiu is only build the script without executing it but reuse all the work done.

---

But i think cabal repl script.hs could be done on the fly without writing anything to disk without fake-package.

... but it would not be cached, fire up a repl is faster than compiling and we avoid the linker step. For small/medium scripts it would not be noticeable but maybe it will do for big ones.

Another approach could be reconsider the remove of fake-package of course

---

Nevertheless i think we could focus in the first step i mentioned above (cache script using fake packages as described above) in cabal run and start to work on this, thoughts about that @bacchanalia?

if the work in cabal run can be reused later in build and repl, we could extract the relevant code at that moment

[bacchanalia]

@jneira I agree with you about the way to proceed

• cabal run is straight forward to make changes to with the existing fake package machinery
• cabal build should work the same way and be able to make use of that work
• cabal clean should probably also get work, if we have build artifacts around we'll want to be able to get rid of them

cabal repl is the outlier in the group in that it's harder to make work with a fake package for the reasons described in #6149
What I'd be tempted to do is bring in fsnotify and use it to keep the fake package in sync with the script. Alternately wait on it until after the work on #6977

[jneira]

What I'd be tempted to do is bring in fsnotify and use it to keep the fake package in sync with the script

hmm i think there is an actual machinery in cabal for that, i would try to reuse it if possible

https://github.com/haskell/cabal/blob/master/cabal-install/src/Distribution/Client/FileMonitor.hs

[bacchanalia]

After discussing with @jneira, the new proposal is change with default behavior without adding flags. There are several reasons for this:

• The implementation of the new flags does not seem to be straightforwards
• The behavior is more consistent with the way the rest of cabal works
• cabal already has too many flags
• My main motivation for the flags was a desire to be conservative and keep the old behavior around, but I'm not sure it's something people actually want.

[bacchanalia]

Does any have thoughts on <cachedir>/cabal/script-builds vs <cabal_dir>/script-cache?

[Mikolaj]

Does XDG specification give any hints? We are moving to XDG slowly (and backward-compatibly).

Edit: see #7386 and #680, but there are also tickets about freeze files caches and perhaps there are any other caches already?

[bacchanalia]

That's an argument in favor of <cachedir>/cabal/script-builds. On linux <cachedir> would be ${XDG_CACHE_HOME:-~/.cache}

edit: or maybe not? The argument for <cabal_dir> is that it can be set with an environment variable, but also presumably once the XDG proposals go though <cabal_dir> would be in <cachedir> anyway.

[bacchanalia]

I had an idea for cabal clean which is that if cabal clean is run without args outside of a project context it should go through script-builds find every one that does not have an associated script on the filesystem, and delete them.

[fgaz]

I think it'd be best to avoid mixing xdg and non-xdg. I think xdg should be adopted as a whole (#680)

What about using the store as cache by just installing the fake package as a remote package? The only problem I see is that it could quickly increase the size of the store since it's immutable

I had an idea for cabal clean which is that if cabal clean is run without args outside of a project context it should go through script-builds find every one that does not have an associated script on the filesystem, and delete them.

or it could be a part of #3333