improve precompiles #20793

JeffBezanson · 2017-02-24T22:14:25Z

Be able to print any type in parse-able syntax. This lets us precompile many more things, such as closures, more operators, types containing Symbol("text/plain"), etc.
Put the code for printing things to precompile in a better place. We want to print the observed concrete types that led to compilation, not something else (previously precompile.jl contained some abstract types, which the precompile function actually ignores to avoid poisoning the cache).
The precompile statements needed to be updated for 0.6 anyway.

The system image is a bit bigger after this, but the repl is really snappy.

vtjnash

what's the performance of the usual using DataFrames tests (from an empty lib code cache) after this change?

vtjnash · 2017-02-24T22:21:41Z

src/rtutils.c

-            n += jl_printf(out, ".");
+            if (!hidden) {
+                n += jl_printf(out, ".");
+                if (globfunc && !jl_id_start_char(u8_nextchar(sn,&i))) {


missing spaces after a handful of the ,'s in here

vtjnash · 2017-02-24T22:23:06Z

src/rtutils.c

@@ -526,11 +528,44 @@ static size_t jl_static_show_x_(JL_STREAM *out, jl_value_t *v, jl_datatype_t *vt
    }
    else if (vt == jl_datatype_type) {
        jl_datatype_t *dv = (jl_datatype_t*)v;
-        if (dv->name->module != jl_core_module) {
+        jl_sym_t *globname = dv->name->mt != NULL ? dv->name->mt->name : NULL;
+        int globfunc = globname && jl_get_global(dv->name->module, globname) &&


jl_get_global can mutate m & print deprecation warnings

jl_get_global can also allocate?

vtjnash · 2017-02-24T22:26:13Z

src/rtutils.c

+        else if (globfunc) {
+            n += jl_printf(out, "typeof(");
+        }
+        if (dv->name->module != jl_core_module || !jl_module_exports_p(jl_core_module, sym)) {


jl_module_exports_p can allocate and can't be used here (or should be fixed not to allocate?)

Yes I can fix it not to allocate.

JeffBezanson · 2017-02-24T22:44:50Z

I enabled TRACE_COMPILE and used the repl with stderr redirected to a file. However it looks like this will not work, since various closures seem to end up with different compiler-generated names on different platforms. I don't know exactly why, but with any platform-conditional code it's bound to happen.

KristofferC · 2017-02-24T23:08:35Z

FWIW, the REPL feels very snappy indeed!

JeffBezanson · 2017-02-24T23:53:09Z

Comments addressed.

I manually removed the closures. Hopefully there will still be a bit of improvement.

ararslan · 2017-02-27T20:04:15Z

Bump. @JeffBezanson, we you able to determine what the performance is of using DataFrames as @vtjnash asked in his review?

vtjnash · 2017-02-27T20:12:08Z

src/rtutils.c

-        n += jl_printf(out, ":%s", jl_symbol_name((jl_sym_t*)v));
+        char *sn = jl_symbol_name((jl_sym_t*)v);
+        // TODO check for valid identifier
+        int quoted = strchr(sn,'/') && strcmp(sn,"/") && strcmp(sn,"//") && strcmp(sn,"//=");


missing spaces after the ,'s here too

Ok... this isn't in the formatting guidelines though.

JeffBezanson · 2017-02-27T21:40:48Z

I tried but I haven't been able to get DataFrames to load on master yet.

ararslan · 2017-02-27T21:53:04Z

I haven't been able to get DataFrames to load on master yet.

Ah, yeah... Try checking out DataArrays to master then checking out DataFrames to anj/06.

JeffBezanson · 2017-03-06T18:54:53Z

Ok, with this change precompiling DataFrames goes from 44 seconds to 50 seconds. Does that mean too many precompiles or not enough?

vtjnash · 2017-03-06T20:12:54Z

DataFrames goes from 44 seconds

Is this a recent regression? The only acceptable answer here is < 20s.

JeffBezanson · 2017-03-17T18:33:04Z

Adding to milestone since updating precompiles should go with the release process, and also to remind us to look at the DataFrames compile time regression.

ararslan · 2017-03-17T18:42:48Z

As a quick aside regarding DataFrames, it's now installable directly from Pkg.add on 0.6 without having to check out specific branches, as a working release has been tagged.

JeffBezanson · 2017-03-23T18:29:16Z

DataFrames compile time regression persists.

JeffBezanson · 2017-03-23T18:56:13Z

Looking at ENABLE_TIMINGS, most of the increases vs. 0.5 seem to be in inference.

- improve static_show to readably print a much larger set of types - add TRACE_COMPILE option to print `precompile` calls

JeffBezanson · 2017-03-24T15:45:37Z

Ok, I updated this to just improve the mechanism.

Package load times can be improved a bit by adding precompile statements, and there might be a couple percent slowdown from #20343, but we still have a large performance gap vs. 0.5.

vtjnash · 2017-03-28T10:49:17Z

OK, thanks. I reviewed the biggest contributors to inference time increase and think I can start working on a patch. That doesn't need to hold this up. It would be good to make sure the new precompile file includes whatever entries that ./julia -e 'using Glob' also lists. I think that should at least prevent further regression in the timing for typical packages.

vtjnash · 2017-03-28T12:04:43Z

CI failures are known issues

JeffBezanson mentioned this pull request Feb 24, 2017

Can the REPL be made more efficient? #15101

Closed

vtjnash requested changes Feb 24, 2017

View reviewed changes

kshyatt added the compiler:precompilation Precompilation of modules label Feb 24, 2017

JeffBezanson force-pushed the jb/precompilemore branch from 292f474 to 56f59d7 Compare February 24, 2017 23:50

JeffBezanson force-pushed the jb/precompilemore branch from 56f59d7 to 0179f27 Compare February 25, 2017 18:23

vtjnash reviewed Feb 27, 2017

View reviewed changes

JeffBezanson added this to the 0.6.0 milestone Mar 17, 2017

improve precompile mechanism a bit

f794796

- improve static_show to readably print a much larger set of types - add TRACE_COMPILE option to print `precompile` calls

JeffBezanson force-pushed the jb/precompilemore branch from 0179f27 to f794796 Compare March 24, 2017 05:07

JeffBezanson mentioned this pull request Mar 27, 2017

compile time of using ApproxFun is almost 10x slower #21173

Closed

vtjnash approved these changes Mar 28, 2017

View reviewed changes

JeffBezanson merged commit 0810648 into master Mar 28, 2017

JeffBezanson mentioned this pull request Mar 28, 2017

update precompile.jl #21196

Closed

tkelman deleted the jb/precompilemore branch March 28, 2017 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve precompiles #20793

improve precompiles #20793

JeffBezanson commented Feb 24, 2017

vtjnash left a comment

vtjnash Feb 24, 2017

vtjnash Feb 24, 2017

yuyichao Feb 24, 2017

vtjnash Feb 24, 2017

JeffBezanson Feb 24, 2017

JeffBezanson commented Feb 24, 2017

KristofferC commented Feb 24, 2017

JeffBezanson commented Feb 24, 2017

ararslan commented Feb 27, 2017

vtjnash Feb 27, 2017

JeffBezanson Feb 27, 2017

JeffBezanson commented Feb 27, 2017

ararslan commented Feb 27, 2017

JeffBezanson commented Mar 6, 2017

vtjnash commented Mar 6, 2017

JeffBezanson commented Mar 17, 2017

ararslan commented Mar 17, 2017

JeffBezanson commented Mar 23, 2017

JeffBezanson commented Mar 23, 2017

JeffBezanson commented Mar 24, 2017

vtjnash commented Mar 28, 2017

vtjnash commented Mar 28, 2017

improve precompiles #20793

improve precompiles #20793

Conversation

JeffBezanson commented Feb 24, 2017

vtjnash left a comment

Choose a reason for hiding this comment

vtjnash Feb 24, 2017

Choose a reason for hiding this comment

vtjnash Feb 24, 2017

Choose a reason for hiding this comment

yuyichao Feb 24, 2017

Choose a reason for hiding this comment

vtjnash Feb 24, 2017

Choose a reason for hiding this comment

JeffBezanson Feb 24, 2017

Choose a reason for hiding this comment

JeffBezanson commented Feb 24, 2017

KristofferC commented Feb 24, 2017

JeffBezanson commented Feb 24, 2017

ararslan commented Feb 27, 2017

vtjnash Feb 27, 2017

Choose a reason for hiding this comment

JeffBezanson Feb 27, 2017

Choose a reason for hiding this comment

JeffBezanson commented Feb 27, 2017

ararslan commented Feb 27, 2017

JeffBezanson commented Mar 6, 2017

vtjnash commented Mar 6, 2017

JeffBezanson commented Mar 17, 2017

ararslan commented Mar 17, 2017

JeffBezanson commented Mar 23, 2017

JeffBezanson commented Mar 23, 2017

JeffBezanson commented Mar 24, 2017

vtjnash commented Mar 28, 2017

vtjnash commented Mar 28, 2017