Ship remove Stdlibs in 1.9+

[LilithHafner]

The following script works on Julia 1.8.4

using DelimitedFiles
temp = tempname()
open(temp, "w") do io
    println(io, "1,2,3")
end
println(sum(readdlm(temp, ',', Int)))

Ideally it should work on all future Julia versions. In interactive use, once DelimitedFiles.jl is no longer a stdlib, the user will be prompted to install the package at the REPL. In package use, it should already be in the environment. As a scripts without a Project.toml, IIUC, the plan is to make this fail with ERROR: ArgumentError: Package DelimitedFiles not found in current path..

An alternate proposal from @IanButterworth on slack is

for scripts, if the package name cannot be resolved, and its in a special list known to Julia of names that used to be stdlibs, then auto-install it and carry on

Much like the interactive REPL prompt, but install & warn rather than prompt.

EDIT to clarify: This would be backed by downloading and loading DelimitedFiles with Pkg, not by packaging DelimitedFiles with Julia.

[brenhinkeller]

A second for Ian's proposal!

[KristofferC]

ERROR: ArgumentError: Package DelimitedFiles not found in current path..

A more accurate error message for this case, saying that the stdlib has been moved and to do Pkg.add("DelimitedFiles") is enough imho.

[Seelengrab]

Should the error also mention to consider using a project specific environment? I imagine the people hit by this could benefit from that the most.

[KristofferC]

I think that is a bit excessive. You can also hit a similar error even when using project envs when using a Manifest from 1.8 on 1.9:

julia> VERSION
v"1.9.0-beta2.8"

shell> cat Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.8.4"
manifest_format = "2.0"
project_hash = "739f2f3af706c750957bd6ec39e0874ba8eb265d"

[[deps.DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

julia> using DelimitedFiles
ERROR: ArgumentError: Package DelimitedFiles [8bb1440f-4735-579b-a4ab-409b98df4dab] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.
 ...

[LilithHafner]

A motivating use case for this is when I give someone a Julia script wrapped in a bash script that they can double-click to run (e.g. ManualImageCoding.jl via QuickDraw). It doesn't matter how explanatory the error message is because they don't speak Julia. It either works or it doesn't. Combining this with a Julia version shared across multiple users could lead to a situation where either user A's clickable application breaks inexplicably (to them) or other uses are pinned to a lower Julia version.

[Seelengrab]

If you're controlling the installation & execution of the script either way, I don't see an advantage over specifying --project in the shell script - something similar would be needed for general purpose standalone packages either way, since they presumably need to have a minimal launch_me.jl for starting in the first place.

For example, instead of activating the environment of the package, activating an environment for this installation seems more appropriate, to allow versioning of the package (irrespective of standalone packages not being affected by this, since they must already have DelimitedFiles in their dependencies).

[LilithHafner]

I don't see an advantage over specifying --project in the shell script

It is easier for non-technical users to handle a single file than a folder containing multiple files whose relationship must remain consistent.

they presumably need to have a minimal launch_me.jl for starting in the first place.

The shell script I distributed was a single file that is a cross platform relocatable executable (provided Julia is installed on that platform). I did not have any launch_me.jl nor did I have any user facing project or version management. The contents of the file were akin to:

julia -e 'import Pkg; Pkg.activate("ManualImageCoding", shared=true); Pkg.add(url="https://github.com/LilithHafner/ManualImageCoding.jl"); using ManualImageCoding; main()'

In my particular use case, my functionality was nontrivial so I put it all in a package. My dependance on DelimitedFiles is via a package, so my users would be fine either way. But if I had a simpler objective (e.g. horizontally concatenating delimited files) that might not have been the case.

I think that removing stdlibs without this feature is technically breaking. While IMO it would be an acceptable minor change, it's better to have this feature and avoid technical breakage altogether.

[brenhinkeller]

I think that removing stdlibs without this feature is technically breaking.

Yeah, it's absolutely breaking, and I think it's astounding that there is so much apparent opposition to a simple fix (just automatically install the ex-stdlib on using) that would make it entirely non-breaking

[giordano]

Yeah, it's absolutely breaking

The code still works as is. What changes is that you need to re-resolve the environment, which is something aside, and this is something you generally need to do anyway when changing versions of Julia.

[brenhinkeller]

No, it does not:

$ cat foo.jl
using DelimitedFiles
temp = tempname()
open(temp, "w") do io
    println(io, "1,2,3")
end
println(sum(readdlm(temp, ',', Int)))

$ julia-18 foo.jl
6
$ julia-19 foo.jl
ERROR: LoadError: ArgumentError: Package DelimitedFiles not found in current path.
- Run `import Pkg; Pkg.add("DelimitedFiles")` to install the DelimitedFiles package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1485 [inlined]
 [2] macro expansion
   @ ./lock.jl:267 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1466
in expression starting at /Users/cbkeller/foo.jl:1

[brenhinkeller]

Any way you try to spin it, it is a breaking change to the behavior of the Julia executable

And, incidentally, it's one that just impacted my entire class, thanks to the breakage it caused in the julia-vscode addon (c.f. https://julialang.slack.com/archives/C6LV2R5JB/p1672886310671379)

[LilithHafner]

If we decide 👍, then this needs action before 1.9. If we decide 👎 then we should document that this sort of breakage is not covered by SymVer. Either way it seems worthy of triaging.

[giordano]

Semver is about API. The API of DelimitedFiles hasn't changed, pretending that's not true is disingenuous.

[fredrikekre]

This "breakage" is akin to the "breakage" you see every minor release since there is a new default package environment. The issue here is pretty narrow, and only applies if you use vanilla Julia without any other packages, right? That seems pretty uncommon.

[brenhinkeller]

If you want to finagle a definition of "breaking" that diverges this strongly from "things that used to work don't work any more", I don't think I'm the one who is going to come across as ""disengenouous"" to a common audience.

[giordano]

Your definition of "breaking" would not allow changing the default random number generator stream. Or changing the algorithm to compute floating point operations. But semver is about API, and API isn't about exact numerical values. Like it isn't about keeping the exact same package environment, which as a matter of fact has changed in every single minor version of Julia, and also some patch numbers.

[elextr]

Perhaps the compatibility limits this FAQ mentions needs to be more prominent in the documentation (perhaps here.

[IanButterworth]

I'm surprised by the divide here. I think it's fair to say that generally the minor release approach is to try to avoid being breaking if possible.

Clearly if someone uses vanilla julia for scripting something that uses DelimitedFiles without an environment, that currently works, but would break in 1.9.

(Or, say another stdlib that might be removed in the future, because DelimitedFiles is a bit of a trial as a minor one to try first)

What's the issue with the suggestion to do a special autoinstall with a warning?

One alternative I guess would be to deprecate DelimitedFiles, so keep it in 1.9 with a deprecation warning, then hard remove it in 1.10 with no autoinstall. That seems more standard?

[Seelengrab]

The divide is purely along the lines of what people consider "part of julia version X.Y.Z". Some take the view that the stdlib is part of julia itself. Others take the view that the stdlib are packages like any other, except that they are shipped with each release, ready to use without having to download them seperately. It's extremely unfortunate that using those packages without explicitly adding them to an environment worked, and that this gave the impression of julia itself being the same as the stdlib packages.

The reason I am against a permanent, silent autoinstall is that it only furthers the sentiment that version management is not important or only for "technical" users. This is an issue of reproducibility, plain and simple, and expecting things to stay reproducible across even minor versions, without managing versions in the slightest, to me at least is a fools errand.

One alternative I guess would be to deprecate DelimitedFiles, so keep it in 1.9 with a deprecation warning, then hard remove it in 1.10 with no autoinstall. That seems more standard?

That sounds like a good approach, allowing users to manage their expectations. As long as we're clear about the upgrade path, e.g. via something along the lines of

The <insert stdlib name> standard library is deprecated as a standard library and will be removed in a future julia version. It will still be installable via ] add <insert stdlib name>, but will no longer be shipped with a default installation of julia.

To keep your script working in future julia versions, please make sure to ] add DelimitedFiles to the project environment of the script. If you are a user and see this message, please notify the author of the script you're using of this message.

, we should be good in terms of communicating the intent. This is not necessarily something a third party, non-technical user (e.g. the users @LilithHafner for QuickDraw mentioned above) are expected to fix, since they are not the creators of the code they're using.

[IanButterworth]

Sounds good, one comment

It's extremely unfortunate that using those packages without explicitly adding them to an environment worked, and that this gave the impression of julia itself being the same as the stdlib packages.

But at least that wouldn't have worked for Pkg, the environment management stuff would've needed to go into Base or
Pkg would likely have always needed to be special. There are so many bare import Pkg; Pkg... in automated jobs out there

[brenhinkeller]

The reason I am against a permanent, silent autoinstall is that it only furthers the sentiment that version management is not important or only for "technical" users.

@Seelengrab I don't think purposely breaking things to teach people a lesson about the importance of version management is a good idea.

It's extremely unfortunate that using those packages without explicitly adding them to an environment worked, and that this gave the impression of julia itself being the same as the stdlib packages.

The fact that stdlibs worked without adding them to an environment was very much intentional. I don't know if you remember, but prior to 0.7 pretty much everything that is now a stdlib used to just be part of the language (no using required!). Stdlibs were a compromise!

Now, maybe you don't like that compromise and wish they had just been excised entirely, but they were not.

[KristofferC]

Stdlibs were a compromise!

Stdlibs were introduced so that issues about them could be moved off the 1.0 milestone precisely because they could be moved out from Julia in the future. That's the whole reason you have to explicitly add stdlibs to your Project file so that moving them out would not break packages.

[KristofferC]

What's the issue with the suggestion to do a special autoinstall with a warning?

Because for a new user (where DelimitedFiles have always been a normal package) it becomes very weird in that some packages auto install from scripts but some don't. The harm of having these type of inconsistencies is to me much greater than an explicit error message telling someone in an error message to write add DelimitedFiles once and then done.

As been said, you already get exactly this error message if you run a script on a new Julia version with any other package other than the stdlibs.

[jakobnissen]

I don't see how to proposed inconsistency of some packages automatically installing is worse than the current inconsistency of some packages seemingly spontaneously no longer working.

Imagine the same new user being sent a script from a co-worker with using DelimitedFiles. Surely, it's also going to be confusing that the script doesn't work for the new user, despite it working for the collaborator?

More broadly speaking, to me it sure looks like a breaking change. Code, which did not rely on internals, or undocumented behaviour, or behaviour explicitly mentioned to be unstable across releases, no longer works. DelimitedFiles is documented in the official documention as a stdlib with no warning that it is experimental and might be removed.

To me, it looks clear-cut. It follows the definition of breaking changes stated in the Julia FAQ (as well as the practical meaning in that a completely innocuous script suddenly stop working). Under what definition of "breaking changes" is this not breaking?

From this thread, it sounds like one argument is that it's fair game to introduce breaking changes to the environment in minor versions (e.g. changing the resolver), and since removing DelimitedFiles is a matter of the environment changing, it's OK. I think this misses the crucial detail that the change is exactly that something which was previously insensitive to the environment now becomes part of the environment and hence subject to change. In that sense it's no different from moving something from Base to a stdlib - only the environment needs to change for the code to work, but that's not relevant. What's relevant is that something that used to work even without resolving the environment no longer does.

[adienes]

Just a reminder that semver is meant to solve a human coordination problem, not a legal definition problem. I expect everybody on this thread might agree that letter of the law could be interpreted as this technically not "breaking," but spirit of the law says it's bad when a short script stops working.

fwiw, I think a 1.9 deprecation and 1.10 excision makes by far the most sense to me. Especially if it would be possible to install the packages 'on the spot' in 1.9 so that the warning goes away (and breakage is preemptively fixed for 1.10)

[KristofferC]

I don't see how to proposed inconsistency of some packages automatically installing is worse than the current inconsistency of some packages seemingly spontaneously no longer working.

That is not an inconsistency in 1.9. And the DelimitedFiles package still works perfectly fine. You just have to explicitly install it.

More broadly speaking, to me it sure looks like a breaking change.

No running code has to be changed so this is an installation process question.

fwiw, I think a 1.9 deprecation and 1.10 excision makes by far the most sense to me.

Moving DelimitedFiles back into 1.9 is very unlikely to happen so I don't think it is worth spending too much time on that angle.

[KristofferC]

I must say, the type of argumentation made in this thread is frankly quite disappointing. Realistically, everyone agrees that having to copy-paste a command from the resulting error message (- Run `import Pkg; Pkg.add("DelimitedFiles")`....) when doing using DelimitedFiles is not going to cause any significant real issues in practice. People have to do that all the time when it comes to virtually all packages being used in e.g. the default environment when upgrading Julia. This is a common error message and it is straight forward to resolve the problem.

So the argumentation made in this thread is not pragmatic, but rather ideological. And that is just a very non-productive way of thinking. Julia and the Julia ecosystem are simply better in all ways with DelimitedFiles being a free package:

• Julia's download size is smaller
• Julia's default startup time is better
• Users of DelimitedFiles can get improvements that are not tied to Julia's release schedule (like Improve inference for DLMStore constructor JuliaData/DelimitedFiles.jl#24 which everyone on 1.9 now gets to enjoy immediately).
• Development of DelimitedFiles is easier.
• DelimitedFiles can be made potentially faster by depending on packages like Parsers.jl.

And the kicker is that much of the design of stdlibs was exactly made so that they could be moved out in the future. Here is a 4-year issue about it JuliaLang/Pkg.jl#411. Like, why do you think you have to add stdlibs to your package as dependencies? Why are stdlibs structured like normal packages with a project file? It is precisely so that they can be moved out without it breaking code (and just require an extra installation step).

[IanButterworth]

What if the autoinstall came with a deprecation warning, that it will not be auto installed in 1.10.

That process/system could be used for future excised stdlibs too.

A one minor version deprecation helper that's noisy and clearly marked as special behavior.

I'm saying this all in the spirit of trying to avoid breakage if we can.

[brenhinkeller]

@KristofferC you seem to be confusing the proposal on the table here with an objection to having ever moved stdlibs out of the system image. The proposal on the table here has absolutely no impact on Julia's download size or startup time.

The desire to avoid having scripts that work in 1.8 break in 1.9 is also about the farthest thing from "ideological" I can imagine, FWIW.

[Seelengrab]

The argument presented here would be applicable for ANY stdlib that would be excised in the future though, so the discussion really isn't exclusively about DelimitedFiles. I think most people would rather not rehash the same discussion about stdlib X in a version transition from 1.y to 1.(y+1) again.

[mikmoore]

A more accurate error message for this case, saying that the stdlib has been moved and to do Pkg.add("DelimitedFiles") is enough imho.

I wouldn't bother to explain that the package has been moved. I think that's noise. An experienced user will shrug and know exactly how to fix it (even if they're surprised by the error) while a "pure consumer" won't care. The suggestion to install it (and copy-paste code) is sufficient and generalized to non-stdlib packages as well.

The error message is great for real packages (especially with the interactive prompt)

julia> using Tullio
 │ Package Tullio not found, but a package named Tullio is available from a registry.
 │ Install package?
 │   (@v1.8) pkg> add Tullio
 └ (y/n/o) [y]: n
ERROR: ArgumentError: Package Tullio not found in current path.
- Run `import Pkg; Pkg.add("Tullio")` to install the Tullio package.

but is misleading for packages that aren't available from a known registry

julia> using NotARealPackage
ERROR: ArgumentError: Package NotARealPackage not found in current path.
- Run `import Pkg; Pkg.add("NotARealPackage")` to install the NotARealPackage package.

I would suggest that we give a different error message when the package is not known. On the other hand, they'll get to the error if they try the suggestion and maybe deferring this this means we don't have to go check the package server right now (which may have some merit).

It may also be worth considering that if we're running from the command line (ie isinteractive()==false) that the command be given not as import Pkg; Pkg.add(...) but as something copy-pastable to a command line (a la julia -e "import Pkg; Pkg.add(...)") to be more helpful to script callers who may be unfamiliar with Julia.

[StefanKarpinski]

My thought here is this:

1. Ship Julia with a new read-only depot at a path like builtin = "/path/to/julia/builtin"
2. Add builtin at the end of the default DEPOT_PATH
3. Have $builtin/environments/builtin/{Project,Manifest}.toml pre-populated with the former stdlibs with [compat] entries capping the former builtins at non-breaking versions.
4. Add @builtin at the end of the default LOAD_PATH.

So altogether you'd have something like this:

DEPOT_PATH = [
    "~/.julia",
    "/usr/local/share/julia",
    "/usr/share/julia",
    "/usr/share/julia/builtin",
]
LOAD_PATH = ["@", "@v#.#", "@stdlib", "@builtin"]

And /usr/share/julia/builtin/environments/builtin/{Project,Manifest}.toml exist with the appropriate contents. That way DelimitedFiles remains loadable at a compatible version from Main but if someone adds DelimitedFiles to a project, then they can get a version that isn't capped at ^1.

[oscardssmith]

Triage's comments are:

1. This was a breaking change.
2. We need to fix this for 1.9
3. @StefanKarpinski's proposal here seems like the way to go (and the autodownload suggest isn't the right way forwardt due to downsides of network connections and reproducibility concerns if code is reliant on servers existing in perpetuity).

[StefanKarpinski]

Regarding how @KristofferC's list of benefits:

• Julia's download size is smaller: no, but the binary will be smaller
• Julia's default startup time is better: yes
• Users of DelimitedFiles can get improvements that are not tied to Julia's release schedule: yes, but if they're incompatible, then they have to go into a 2.x release of DelimitedFiles
• Development of DelimitedFiles is easier: yes—make releases whenever, can make breaking changes (although we will continue to ship a 1.x compatible version in builtins)
• DelimitedFiles can be made potentially faster by depending on packages like Parsers.jl: yes, this is fine, although we'll have to also make some version of Parers builtin then.

[KristofferC]

My thought here is this:...

It seems unlikely to me that backporting such a big change is a good idea this late in the release process.

[KristofferC]

oscardssmith changed the title "Automatically install excised stdlibs in scripts" to "Ship remove Stdlibs in 1.9"

@oscardssmith could you elaborate on what the title of the issue means now? I don't understand it.

[KristofferC]

@StefanKarpinski's suggestion seems overly complex to me. Considering the following branch: master...kc/upgradable_stdlibs.

What it does is that it puts back DelimitedFiles as an stdlib (but crucially does not load it into the sysimage). It also updates Pkg to not consider DelimitedFiles to be an stdlib (https://github.com/JuliaLang/Pkg.jl/compare/master…kc/external_stdlib).

This gives the following behavior:

❯ JULIA_DEPOT_PATH=/tmp/depot ./julia -q
(v1.10) pkg> st
Status `/private/tmp/depot/environments/v1.10/Project.toml` (empty project)

julia> using DelimitedFiles
[ Info: Precompiling DelimitedFiles [8bb1440f-4735-579b-a4ab-409b98df4dab]

julia> pathof(DelimitedFiles) # shipped
"/Users/kristoffercarlsson/julia/usr/share/julia/stdlib/v1.10/DelimitedFiles/src/DelimitedFiles.jl"

(v1.10) pkg> add DelimitedFiles
    Updating registry at `/tmp/depot/registries/General.toml`
   Resolving package versions...
   Installed DelimitedFiles ─ v1.9.1
    Updating `/private/tmp/depot/environments/v1.10/Project.toml`
  [8bb1440f] + DelimitedFiles v1.9.1
    Updating `/private/tmp/depot/environments/v1.10/Manifest.toml`
  [8bb1440f] + DelimitedFiles v1.9.1
  [a63ad114] + Mmap
Precompiling environment...
  ✓ DelimitedFiles
  1 dependency successfully precompiled in 2 seconds
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version

julia> exit()

❯ JULIA_DEPOT_PATH=/tmp/depot ./julia -q
julia> using DelimitedFiles

julia> pathof(DelimitedFiles) # versioned
"/tmp/depot/packages/DelimitedFiles/aGcsu/src/DelimitedFiles.jl"

So you can load DelimitedFiles by default, and you can add it with the package manager and then it is going to use the versioned one. Basically, as soon as DelimitedFiles end up in a project/manifest it will be versioned. It is only if loading it from an environment stack that does not have DelimitedFiles in it that the shipped one is used.

At every Julia release the shipped version presumably would update to the newest one upstream and when a breaking version is released of e.g. DelimitedFiles I guess we will ship the old DelimitedFiles forever or something. And it also limits upgradable stdlibs to not use any external packages until they have a breaking version (or at least not without freezing the version that will be shipped by Julia). Not really great.

The point of @StefanKarpinski's proposal seems to be the ability to add [compat] to the @builtin project, but Pkg does not use any compat in any other environment than the active one so this seems pointless.

[DilumAluthge]

Considering the following branch: master...kc/upgradable_stdlibs.

For those (like me) that were looking for the corresponding Pkg branch, it’s available here: Pkg#master…kc/external_stdlib

[StefanKarpinski]

Well, that's certainly a small enough patch. I'm fine with that for the time being. I do think we need the concept of bundled packages eventually (and ideally all stdlibs and jlls eventually just become this), but we can definitely do this for now.

You're absolutely right that there's no point to the [compat] entry in a builtin project file from the end user's perspective. I was thinking of it from our perspective where the way we would update it is by activating the bundled environment at build time and using Pkg to do an up operation with compat keeping it from updating to something incompatible. (I'm thinking "bundled" may be a better name since "builtin" sounds more tightly coupled than "stdlib" even.)

That's why I want to go in this direction: right now stdlibs and jlls are these special snowflakes that we need to go through special incantations to update and manage. It's waay more annoying than installing and updating normal packages and jlls. How many files do you need to edit by hand to update a stdlib? It's so annoying that we have a GitHub action for it! Updating a JLL is at least as annoying and we don't have an automated way to do it. Why not use the very same excellent tooling we use for normal packages and jlls to manage the packages and jlls that we ship with Julia? If @stdlib was just a named environment that we ship with Julia, then we could! @staticfloat and I have discussed this over time and have been gradually taking steps to get there, and I was seeing this as a natural step in that direction.

That being said, there's no need to rush it. Maybe what I was calling @builtin should just be @stdilb. In the meantime, we can do it this way and figure out the rest in a future release.

[KristofferC]

A consequence of the decision here is that we will always have to bundle SparseArrays which means Julia will always be GPL by default and we cannot ship e.g. MKL with Julia for potentially significant performance improvements for a lot of people. Hope it was worth it.

[StefanKarpinski]

I'm pretty sure that's not true. The FSF is very clear that mere aggregation does not constitute creation of a derived work: https://www.gnu.org/licenses/gpl-faq.en.html#MereAggregation. At this point Julia doesn't require or link by default against any GPL software. Nothing GPL is loaded unless you explicitly go out of your way to load it. Since Julia is literally capable of loading any dynamic library at runtime, the fact that you could load a library does not mean that it's part of Julia—you can't argue that Julia must be bound by the licensee of every library in existence just because it can load them. The only thing special about SuiteSparse is that it happens to be shipped side by side in the same tarball—you could delete it and everything would work fine unless you tried to load the SuiteSparse package. The SuiteSparse .so isn't linked into libjulia or anything either. Quoting the FSF FAQ, in the case of mere aggregation of two programs, "if one of the programs is covered by the GPL, it has no effect on the other program." So I think at this point we can already consider base Julia to be GPL-free.