cache compile dep tracking #18150

vtjnash · 2016-08-19T22:10:20Z

This implements the dependency tracking for compilecache that I was envisioning from the beginning. The idea here is to not only fix #18069 (the first commit), but also to make sure the situation seen on JuliaBox can't happen. Instead, it will prefer using a stale cache over replacing it with one that can't be loaded into the current session. In a different (new) session, it will rebuild the whole cache as normal.

For example, after this PR, the following works, whereas in the existing design, loading FixedPointNumbers would require first calling reload(:Compat):

julia> using Compat
shell> touch ~/.julia/v0.6/Compat/src/Compat.jl
julia> using FixedPointNumbers # needs Compat

tkelman · 2016-08-19T22:19:17Z

base/loading.jl

+        end
+        provides_concrete && return false # this is the file we want
+
+        # now check if this file is fresh relative to it's source files


tkelman · 2016-08-19T22:47:55Z

Please elaborate in some comments what is meant by "concrete dependency." That term is used many times here but not explained at all.

ViralBShah · 2016-08-22T03:13:42Z

@tkelman We will need a new release of 0.4 based on this - perhaps this and any other things can form the basis of a final 0.4 release.

For now, we have a temporary workaround on JuliaBox.

ViralBShah · 2016-08-22T03:14:26Z

Cc @nkottary @tanmaykm

tkelman · 2016-08-22T20:08:37Z

base/loading.jl

 end

+# these return either the array of modules loaded from the path / content given
+# an Exception that describes why it couldn't be loaded


or an Exception ?

ViralBShah · 2016-08-23T03:06:08Z

Merge?

tkelman · 2016-08-23T17:51:21Z

doc/manual/modules.rst

+For file dependencies, a change is determined by examining whether the modification time
+of each file loaded by `include` or added explicity by ``include_dependency`` is unchanged,
+or equal to the modification time truncated to the nearest second
+(to accomodate systems that can't copy mtime with sub-second accuracy).


accommodate has 2 m's, and should spell out mtime otherwise it's jargon

tkelman · 2016-08-24T02:03:14Z

doc/manual/modules.rst

+or explicit dependencies declared by ``include_dependency(path)`` in the module file(s).
+
+For file dependencies, a change is determined by examining whether the modification time (mtime)
+of each file loaded by `include` or added explicity by ``include_dependency`` is unchanged,


rst, needs double backticks

fix #18069

this gives the caller more control over when an how errors get printed

…pile caches this ensures that we only recompile a dependency if either: it can be loaded into the current session, or the user explicitly requests it (via reload) thereby reducing the occurrences and improving the accuracy of the "uuid didn't match" error message :)

ViralBShah · 2016-08-29T10:46:43Z

@tkelman Is this PR included in 0.5rc3? If not, are you thinking to put out 0.5rc4 with this included - or is this slated for 0.5.1?

fix #18069 (cherry picked from commit 94ce57d) ref #18150

this gives the caller more control over when an how errors get printed (cherry picked from commit 5abccb6) ref #18150

…pile caches this ensures that we only recompile a dependency if either: it can be loaded into the current session, or the user explicitly requests it (via reload) thereby reducing the occurrences and improving the accuracy of the "uuid didn't match" error message :) (cherry picked from commit 0df6e61) ref #18150

fix #18115 (cherry picked from commit 5276a4d) ref #18150

tkelman · 2016-08-29T10:53:47Z

#18276

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150 (cherry picked from commit cfceca4) ref #18279

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150 (cherry picked from commit 10f3c35)

fix #18115 (cherry picked from PR #18150 commit 5276a4d)

fix #18069 (cherry picked from PR #18150 commit 94ce57d)

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after JuliaLang#18150

also add plenty of helpful debugging information enabled by the environment variable: JULIA_DEBUG_LOADING this bug would cause compilation to become confused and likely tardy at recompiling files when it should that would up with longer dependency chains where one of the files had been updated this is a fix for #18150

also add plenty of helpful debugging information enabled by the environment variable: JULIA_DEBUG_LOADING this bug would cause compilation to become confused and likely tardy at recompiling files when it should that would up with longer dependency chains where one of the files had been updated this is a fix for #18150 (cherry picked from commit 718bd3e)

vtjnash added backport pending 0.4 needs tests Unit tests are required for this change needs docs Documentation for this change is required labels Aug 19, 2016

tkelman reviewed Aug 19, 2016
View reviewed changes

vtjnash force-pushed the jn/cache-compile-dep-tracking branch from b83891f to d2d520e Compare August 22, 2016 19:28

tkelman reviewed Aug 22, 2016
View reviewed changes

vtjnash removed needs docs Documentation for this change is required needs tests Unit tests are required for this change labels Aug 22, 2016

vtjnash changed the title ~~wip: cache compile dep tracking~~ cache compile dep tracking Aug 22, 2016

ViralBShah added this to the 0.5.x milestone Aug 23, 2016

vtjnash force-pushed the jn/cache-compile-dep-tracking branch from d2d520e to ba9a921 Compare August 23, 2016 17:41

tkelman reviewed Aug 23, 2016
View reviewed changes

vtjnash force-pushed the jn/cache-compile-dep-tracking branch from ba9a921 to b104b5f Compare August 23, 2016 19:38

tkelman reviewed Aug 24, 2016
View reviewed changes

vtjnash added 3 commits August 24, 2016 11:54

fix sort / fallback order for incremental precompile

94ce57d

fix #18069

change incremental serialize to use deferred errors

5abccb6

this gives the caller more control over when an how errors get printed

vtjnash deleted the jn/cache-compile-dep-tracking branch August 25, 2016 01:49

This was referenced Aug 25, 2016

Warnings related to package precompilation #12252

Closed

error loading just-precompiled module if an imported module was loaded before precompiling #12508

Closed

stevengj mentioned this pull request Aug 25, 2016

Add warning to Pkg.update() if updated modules were already imported #18239

Closed

tkelman pushed a commit that referenced this pull request Aug 29, 2016

fix sort / fallback order for incremental precompile

4568dd1

fix #18069 (cherry picked from commit 94ce57d) ref #18150

tkelman pushed a commit that referenced this pull request Aug 29, 2016

change incremental serialize to use deferred errors

a51c614

this gives the caller more control over when an how errors get printed (cherry picked from commit 5abccb6) ref #18150

tkelman pushed a commit that referenced this pull request Aug 29, 2016

document __init__ behavior during precompile

746cc0c

fix #18115 (cherry picked from commit 5276a4d) ref #18150

tkelman added a commit that referenced this pull request Aug 29, 2016

Rename the FooBar type in test/core.jl

cfceca4

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150

tkelman mentioned this pull request Aug 29, 2016

Rename the FooBar type in test/core.jl #18279

Merged

tkelman added a commit that referenced this pull request Aug 29, 2016

Rename the FooBar type in test/core.jl

31887cb

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150 (cherry picked from commit cfceca4) ref #18279

stevengj pushed a commit that referenced this pull request Aug 29, 2016

Rename the FooBar type in test/core.jl (#18279)

10f3c35

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150

tkelman added a commit that referenced this pull request Aug 29, 2016

Rename the FooBar type in test/core.jl (#18279)

f34a3a8

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after #18150 (cherry picked from commit 10f3c35)

vtjnash added a commit that referenced this pull request Aug 30, 2016

document __init__ behavior during precompile

355445a

fix #18115 (cherry picked from PR #18150 commit 5276a4d)

vtjnash added a commit that referenced this pull request Aug 30, 2016

fix sort / fallback order for incremental precompile

917acce

fix #18069 (cherry picked from PR #18150 commit 94ce57d)

mfasi pushed a commit to mfasi/julia that referenced this pull request Sep 5, 2016

Rename the FooBar type in test/core.jl (JuliaLang#18279)

c45b35e

so `JULIA_CPU_CORES=1 ./julia test/runtests.jl core compile` passes after JuliaLang#18150

vtjnash mentioned this pull request Sep 5, 2016

fix an inverted return conditional from stale_cachefile #18366

Merged

tkelman removed backport pending 0.5 labels Sep 8, 2016

GregPlowman mentioned this pull request Jan 10, 2017

Error loading module on remote workers #19960

Closed

stevengj mentioned this pull request Jun 29, 2017

Error on Julia v 0.6 cstjean/ScikitLearn.jl#30

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cache compile dep tracking #18150

cache compile dep tracking #18150

vtjnash commented Aug 19, 2016

tkelman Aug 19, 2016

tkelman commented Aug 19, 2016

ViralBShah commented Aug 22, 2016

ViralBShah commented Aug 22, 2016

tkelman Aug 22, 2016

ViralBShah commented Aug 23, 2016

tkelman Aug 23, 2016

tkelman Aug 24, 2016

ViralBShah commented Aug 29, 2016

tkelman commented Aug 29, 2016

cache compile dep tracking #18150

cache compile dep tracking #18150

Conversation

vtjnash commented Aug 19, 2016

tkelman Aug 19, 2016

Choose a reason for hiding this comment

tkelman commented Aug 19, 2016

ViralBShah commented Aug 22, 2016

ViralBShah commented Aug 22, 2016

tkelman Aug 22, 2016

Choose a reason for hiding this comment

ViralBShah commented Aug 23, 2016

tkelman Aug 23, 2016

Choose a reason for hiding this comment

tkelman Aug 24, 2016

Choose a reason for hiding this comment

ViralBShah commented Aug 29, 2016

tkelman commented Aug 29, 2016