Parallel precompile now seems to fail precompiling the package itself #53859

KristofferC · 2024-03-26T15:24:08Z

(Foo) pkg> precompile
   Installed Example ─ v0.5.3
Precompiling all packages...
  1 dependency successfully precompiled in 1 seconds

(Foo) pkg> st
Status `~/julia/Foo/Project.toml`
  [7876af07] Example v0.5.3

julia> using Foo # should have precompiled `Foo` already
[ Info: Precompiling Foo [16a8b8d3-1370-5883-bb6f-f9266484db3f]

The text was updated successfully, but these errors were encountered:

KristofferC · 2024-03-27T15:19:12Z

(Might actually not be an issue on 1.11)

krynju · 2024-03-29T09:33:16Z

I hit a similiar case with one package on both 1.11 and nightly (28.03.24)
I would also get precompile invalidations and runtime precompile on packages depending on this package during build

➜  Abc git:(main) ✗ julia +1.11
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-alpha2 (2024-03-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> activate .
  Activating project at `~/P/packages/Abc`

julia> using Abc
Precompiling Abc...
  1 dependency successfully precompiled in 5 seconds. 128 already precompiled.
[ Info: Precompiling Abc [6198f965-9602-40a9-911d-ddd886f47855] (cache misses: nonresolveable depot (2))

julia> using Abc

(Abc) pkg> instantiate
Precompiling project...
  ✓ Abc
  1 dependency successfully precompiled in 5 seconds. 129 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version

(Abc) pkg> instantiate
Precompiling project...
  ✓ Abc
  1 dependency successfully precompiled in 4 seconds. 129 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version



➜  Abc git:(main) ✗ julia +nightly
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.12.0-DEV.265 (2024-03-28)
 _/ |\__'_|_|_|\__'_|  |  Commit b18d2cc7046 (0 days old master)
|__/                   |

(@v1.12) pkg> activate .
  Activating project at `~/P/packages/Abc`

julia> using Abc
Precompiling Abc...
  1 dependency successfully precompiled in 5 seconds. 128 already precompiled.
[ Info: Precompiling Abc [6198f965-9602-40a9-911d-ddd886f47855] (cache misses: nonresolveable depot (2))

(Abc) pkg> instantiate
Precompiling all packages...
  ✓ Abc
  1 dependency successfully precompiled in 5 seconds. 128 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version

fatteneder · 2024-03-29T14:36:57Z

Do you have a minimal reproducer?
Or is the source code of the offending pkgs openly available?

krynju · 2024-03-29T14:57:52Z

It's proprietary code, but I managed to get an MWE

I think this issue is different from what OP reported, but maybe it's related? Looks more similiar to #53831
I couldn't reproduce the issue from OP on other packages

module Abc

using AWS: @service

@service ECR

end

AWS.jl as the only dependecy

Found three packages like this and this was the common thing in them

fatteneder · 2024-03-29T15:47:29Z

In #52346 we already addressed an issue related to the AWS pkg. @IanButterworth

Will need to have a closer look again.

fatteneder · 2024-03-30T23:40:44Z

@KristofferC What's the setup to reproduce the Example.jl issue?
I can't reproduce on 313f933 or #53905.

IanButterworth · 2024-03-31T05:29:17Z

The active project needs to be a package. That should be all that's required I believe.

fatteneder · 2024-04-01T11:51:03Z

I cloned Example.jl and tried with

activating its package environment -- no recompilation
generating a separate project and adding Example.jl as a dependency -- no recompilation

In any case, if the original issue was related to relocation, then I think it should be fixed after #53905.

…files (#53905) Fixes #53859 (comment), which was actually fixed before in #52346, but #52750 undid that fix. This PR does not directly address #53859, because I can not reproduce it atm. --- The `@depot` resolution logic for `include()` files is adjusted as follows: 1. (new behavior) If the cache is not relocatable because of an absolute path, we ignore that path for the depot search. Recompilation will be triggered by `stale_cachefile()` if that absolute path does not exist. Previously this caused any `@depot` tags to be not resolved and so trigger recompilation. 2. (new behavior) If we can't find a depot for a relocatable path, we still replace it with the depot we found from other files. Recompilation will be triggered by `stale_cachefile()` because the resolved path does not exist. 3. (this behavior is kept) We require that relocatable paths all resolve to the same depot. 4. (new behavior) We no longer use the first matching depot for replacement, but instead we explicitly check that all resolve to the same depot. This has two reasons: - We want to scan all source files anyways in order to provide logs for 1. and 2. above, so the check is free. - It is possible that a depot might be missing source files. Assume that we have two depots on `DEPOT_PATH`, `depot_complete` and `depot_incomplete`. If `DEPOT_PATH=["depot_complete","depot_incomplete"]` then no recompilation shall happen, because `depot_complete` will be picked. If `DEPOT_PATH=["depot_incomplete","depot_complete"]` we trigger recompilation and hopefully a meaningful error about missing files is thrown. If we were to just select the first depot we find, then whether recompilation happens would depend on whether the first relocatable file resolves to `depot_complete` or `depot_incomplete`.

KristofferC · 2024-04-02T11:03:59Z

I had forgotten the uuid entry in my Project.toml for Foo.

…files (#53905) Fixes #53859 (comment), which was actually fixed before in #52346, but #52750 undid that fix. This PR does not directly address #53859, because I can not reproduce it atm. --- The `@depot` resolution logic for `include()` files is adjusted as follows: 1. (new behavior) If the cache is not relocatable because of an absolute path, we ignore that path for the depot search. Recompilation will be triggered by `stale_cachefile()` if that absolute path does not exist. Previously this caused any `@depot` tags to be not resolved and so trigger recompilation. 2. (new behavior) If we can't find a depot for a relocatable path, we still replace it with the depot we found from other files. Recompilation will be triggered by `stale_cachefile()` because the resolved path does not exist. 3. (this behavior is kept) We require that relocatable paths all resolve to the same depot. 4. (new behavior) We no longer use the first matching depot for replacement, but instead we explicitly check that all resolve to the same depot. This has two reasons: - We want to scan all source files anyways in order to provide logs for 1. and 2. above, so the check is free. - It is possible that a depot might be missing source files. Assume that we have two depots on `DEPOT_PATH`, `depot_complete` and `depot_incomplete`. If `DEPOT_PATH=["depot_complete","depot_incomplete"]` then no recompilation shall happen, because `depot_complete` will be picked. If `DEPOT_PATH=["depot_incomplete","depot_complete"]` we trigger recompilation and hopefully a meaningful error about missing files is thrown. If we were to just select the first depot we find, then whether recompilation happens would depend on whether the first relocatable file resolves to `depot_complete` or `depot_incomplete`. (cherry picked from commit d8d3842)

KristofferC added regression Regression in behavior compared to a previous version compiler:precompilation Precompilation of modules labels Mar 26, 2024

KristofferC added this to the 1.12 milestone Mar 26, 2024

KristofferC self-assigned this Mar 26, 2024

KristofferC modified the milestones: 1.12, 1.11 Mar 26, 2024

This was referenced Mar 30, 2024

Relocatability issues with services files JuliaCloud/AWS.jl#579

Open

Avoid repeated precompilation when loading from non-relocatable cachefiles #53905

Merged

fatteneder closed this as completed in #53905 Apr 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel precompile now seems to fail precompiling the package itself #53859

Parallel precompile now seems to fail precompiling the package itself #53859

KristofferC commented Mar 26, 2024 •

edited

Loading

KristofferC commented Mar 27, 2024

krynju commented Mar 29, 2024

fatteneder commented Mar 29, 2024

krynju commented Mar 29, 2024 •

edited

Loading

fatteneder commented Mar 29, 2024 •

edited

Loading

fatteneder commented Mar 30, 2024

IanButterworth commented Mar 31, 2024

fatteneder commented Apr 1, 2024

KristofferC commented Apr 2, 2024

Parallel precompile now seems to fail precompiling the package itself #53859

Parallel precompile now seems to fail precompiling the package itself #53859

Comments

KristofferC commented Mar 26, 2024 • edited Loading

KristofferC commented Mar 27, 2024

krynju commented Mar 29, 2024

fatteneder commented Mar 29, 2024

krynju commented Mar 29, 2024 • edited Loading

fatteneder commented Mar 29, 2024 • edited Loading

fatteneder commented Mar 30, 2024

IanButterworth commented Mar 31, 2024

fatteneder commented Apr 1, 2024

KristofferC commented Apr 2, 2024

KristofferC commented Mar 26, 2024 •

edited

Loading

krynju commented Mar 29, 2024 •

edited

Loading

fatteneder commented Mar 29, 2024 •

edited

Loading