Enable precompilation

[nalimilan]

DataFrames.precompile() precompiles all methods that are used when running tests.
This is useful to run benchmarks like H2Oai where the time of the first run should include the time needed to specialize on a particular operation on a particular dataset, but not the time needed for the more general compilation of common parts.
It could also be useful for users who would like to generate a precompiled image.

To avoid making the package slower to precompile or load, the precompilation code is only included when actually used.

Some timings on a clean Julia 1.5 session:

# Without precompilation
julia> df = DataFrame(x=[1, 1, 2, 2], y=rand(4));

julia> @time combine(groupby(df, :x), :y => sum);
  7.392566 seconds (16.59 M allocations: 850.477 MiB, 8.03% gc time)

# With precompilation
julia> @time DataFrames.precompile()
277.652629 seconds (494.20 M allocations: 24.648 GiB, 3.38% gc time)

julia> df = DataFrame(x=[1, 1, 2, 2], y=rand(4));

julia> @time combine(groupby(df, :x), :y => sum);
  0.527538 seconds (329.43 k allocations: 17.070 MiB)

@jangorecki Does it sound OK to you to spend 277 seconds on precompilation before running the H2Oai benchmarks? We could precompile only the methods which are relevant for you if you prefer.

See #1502.

[jangorecki]

Looks good like this. Please do not restrict to functions only from db-benchmark. Code should be as general as possible. Specializing it for particular cases would be little bit like cheating (fyi @bkamins). As long as there is no "startup" task in db-benchmark you will be good, and such task won't happen anytime soon :)
Eventually there is a plan for a "full workflow" task, that includes reading, processing and writing, but IMO we can precompile only when it is worth. As long as this is an option, and not installing different package it is fine.

[bkamins]

@nalimilan - please let me know if we want to add it to 0.22 release goal. Thank you!

[nalimilan]

I've updated the PR to include two levels of precompilation:

• a first level which takes 37s and is applied automatically when installing the package
• a second level which takes a very long time and is only applied when calling DataFrames.precompiled (just like the original state of the PR)

Current master:

julia> @time using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
  7.056511 seconds (3.10 M allocations: 180.010 MiB, 1.52% gc time)

julia> df = DataFrame(x=[1, 1, 2, 2], y=rand(4));

julia> @time combine(groupby(df, :x), :y => sum);
  5.426770 seconds (14.92 M allocations: 760.778 MiB, 7.97% gc time)

# New clean session
julia> @time using DataFrames
  1.370335 seconds (1.80 M allocations: 114.414 MiB, 0.91% gc time)

This PR:

julia> @time using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
 37.902554 seconds (3.56 M allocations: 218.251 MiB, 0.47% gc time)

julia> df = DataFrame(x=[1, 1, 2, 2], y=rand(4));

julia> @time combine(groupby(df, :x), :y => sum);
  2.215117 seconds (2.82 M allocations: 143.991 MiB, 1.27% gc time)

# New clean session
julia> @time using DataFrames
  2.200718 seconds (2.26 M allocations: 152.705 MiB, 1.47% gc time)

julia> DataFrames.precompile(true)

julia> df = DataFrame(x=[1, 1, 2, 2], y=rand(4));

julia> @time combine(groupby(df, :x), :y => sum);
  0.591102 seconds (482.27 k allocations: 24.417 MiB)

[bkamins]

I will test it before merging but it looks excellent.

[bkamins]

The downside is that we should update these codes each time we make a release - is this correct?

[bkamins]

You need to update the all=true version because of the removal Symbol support in CategoricalValue (the case we have just fixed on master), as now:

julia> @time DataFrames.precompile(true)
ERROR: TypeError: in CategoricalValue, in T, expected T<:Union{AbstractChar, AbstractString, Number}, got Type{Symbol}

[bkamins]

Results of my benchmarks (and in general I think it is really nice; the only strange thing is that DataFrame constructor takes longer to execute, but it is fast so it is not a problem - it might be just noise):

This PR:

julia> @time using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
 25.607132 seconds (2.87 M allocations: 181.689 MiB, 0.50% gc time)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.039932 seconds (91.70 k allocations: 5.010 MiB)
4×2 DataFrame
 Row │ x      y        
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.470576
   2 │     1  0.370083
   3 │     2  0.560629
   4 │     2  0.794492

julia> @time combine(groupby(df, :x), :y => sum)
  1.450754 seconds (2.48 M allocations: 126.870 MiB, 2.82% gc time)
2×2 DataFrame
 Row │ x      y_sum    
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.840659
   2 │     2  1.35512

fresh session:

julia> @time using DataFrames
  1.435035 seconds (1.66 M allocations: 120.235 MiB, 6.26% gc time)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.038710 seconds (91.70 k allocations: 5.010 MiB)
4×2 DataFrame
 Row │ x      y        
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.397124
   2 │     1  0.423037
   3 │     2  0.591869
   4 │     2  0.362487

julia> @time combine(groupby(df, :x), :y => sum)
  1.414719 seconds (2.48 M allocations: 126.876 MiB, 1.54% gc time)
2×2 DataFrame
 Row │ x      y_sum    
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.820161
   2 │     2  0.954356

master

julia> @time using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
  4.429047 seconds (2.41 M allocations: 142.747 MiB, 2.76% gc time)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.011212 seconds (3.61 k allocations: 261.883 KiB)
4×2 DataFrame
 Row │ x      y         
     │ Int64  Float64   
─────┼──────────────────
   1 │     1  0.992411
   2 │     1  0.0578703
   3 │     2  0.745452
   4 │     2  0.201506

julia> @time combine(groupby(df, :x), :y => sum)
  3.445880 seconds (14.16 M allocations: 722.170 MiB, 7.72% gc time)
2×2 DataFrame
 Row │ x      y_sum    
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  1.05028
   2 │     2  0.946958

fresh session:

julia> @time using DataFrames
  0.684325 seconds (1.19 M allocations: 81.294 MiB)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.011208 seconds (3.61 k allocations: 261.883 KiB)
4×2 DataFrame
 Row │ x      y         
     │ Int64  Float64   
─────┼──────────────────
   1 │     1  0.335812
   2 │     1  0.339437
   3 │     2  0.0230391
   4 │     2  0.156288

julia> @time combine(groupby(df, :x), :y => sum)
  3.572502 seconds (14.16 M allocations: 722.197 MiB, 10.75% gc time)
2×2 DataFrame
 Row │ x      y_sum    
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.675249
   2 │     2  0.179327

[nalimilan]

The downside is that we should update these codes each time we make a release - is this correct?

Well we don't have to, as precompile will fail silently if the method doesn't exist. But it would be better to update it after making large changes, otherwise precompilation will have less effect.

You need to update the all=true version because of the removal Symbol support in CategoricalValue (the case we have just fixed on master), as now:

Ah right, I had run this with the old version waiting for the test fix. I've pushed a commit to fix it.

I'm not sure why the constructor is slightly slower with precompilation. Maybe the method table is larger, or the GC has more work to do. I guess it's still worth it since taking 0.02s longer isn't really noticeable for users.

[bkamins]

I guess it's still worth it since taking 0.02s longer isn't really noticeable for users.

agreed

[bkamins]

Another test with precompile(true). As you can see it has an impact on time of constructor execution

First run

julia> @time using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
 24.461958 seconds (2.86 M allocations: 180.183 MiB, 0.46% gc time)

julia> @time DataFrames.precompile(true)
184.200906 seconds (429.83 M allocations: 21.758 GiB, 4.35% gc time)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.011488 seconds (180 allocations: 32.188 KiB)
4×2 DataFrame
 Row │ x      y        
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.11737
   2 │     1  0.340595
   3 │     2  0.304888
   4 │     2  0.210472

julia> @time combine(groupby(df, :x), :y => sum)
  0.433025 seconds (482.27 k allocations: 24.415 MiB)
2×2 DataFrame
 Row │ x      y_sum    
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.457965
   2 │     2  0.51536

Second run

julia> @time using DataFrames
  1.390826 seconds (1.65 M allocations: 118.753 MiB, 6.34% gc time)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.127018 seconds (91.68 k allocations: 5.008 MiB, 57.73% gc time)
4×2 DataFrame
 Row │ x      y         
     │ Int64  Float64   
─────┼──────────────────
   1 │     1  0.0532204
   2 │     1  0.35766
   3 │     2  0.936712
   4 │     2  0.842535

julia> @time combine(groupby(df, :x), :y => sum)
  1.387860 seconds (2.48 M allocations: 126.842 MiB, 1.79% gc time)
2×2 DataFrame
 Row │ x      y_sum   
     │ Int64  Float64 
─────┼────────────────
   1 │     1  0.41088
   2 │     2  1.77925

Third run

julia> @time using DataFrames
  1.393187 seconds (1.65 M allocations: 118.755 MiB, 6.31% gc time)

julia> @time df = DataFrame(x=[1, 1, 2, 2], y=rand(4))
  0.126041 seconds (91.68 k allocations: 5.008 MiB, 58.16% gc time)
4×2 DataFrame
 Row │ x      y        
     │ Int64  Float64  
─────┼─────────────────
   1 │     1  0.875997
   2 │     1  0.648122
   3 │     2  0.961639
   4 │     2  0.386705

julia> @time combine(groupby(df, :x), :y => sum)
  1.450556 seconds (2.48 M allocations: 126.842 MiB, 1.96% gc time)
2×2 DataFrame
 Row │ x      y_sum   
     │ Int64  Float64 
─────┼────────────────
   1 │     1  1.52412
   2 │     2  1.34834

[bkamins]

The benefit of removing CategoricalArrays.jl precompilation is ~2 sec faster first precompile and 0.1 sec faster startup

[bkamins]

I am done adding the suggestions. GitHub messes up the display, but I hope it will be clear for you what I suggest to remove.

[nalimilan]

I've just realized that this should probably be tested, to avoid making coverage too low, and because it could break (e.g. when changing types). It takes about 100s in a session after running tests. I guess that's OK?

[bkamins]

I guess that's OK?

It is OK. We have ~15 minutes testing time anyway. It takes 180 seconds on my machine to test it.

Please add this test to a separate file (so that it can be easily disabled when developing).

Thank you!

[ronisbr]

I really want to add precompilation in PrettyTables.jl, but I might need a help, since I have never done something like this for a package (except for small tests).

[nalimilan]

It is OK. We have ~15 minutes testing time anyway. It takes 180 seconds on my machine to test it.

Please add this test to a separate file (so that it can be easily disabled when developing).

@bkamins OK done.

I really want to add precompilation in PrettyTables.jl, but I might need a help, since I have never done something like this for a package (except for small tests).

@ronisbr I'm not too familiar either, I just took inspiration from FixedPointNumbers and SnoopCompile docs. You can also use the same sequence of calls that I put in a comment at the top of src/other/precompile.jl in this PR.

FWIW, here's the precompile calls that SnoopCompile generated for PrettyTables when running the DataFrames tests (with a time threshold of 0.01 or 0.02, I don't remember exactly). You'll probably get similar ones with PrettyTables tests, but if not it could make sense to include them anyway to make DataFrames faster.

    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Bool})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Char})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Float64})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Int32})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Int64})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Irrational{:π}})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Markdown.MD})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),String})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),Symbol})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),UndefInitializer})
    Base.precompile(Tuple{Core.kwftype(typeof(PrettyTables._parse_cell_text)),NamedTuple{(:autowrap, :cell_data_type, :cell_first_line_only, :column_width, :compact_printing, :has_color, :linebreaks, :renderer),Tuple{Bool,DataType,Bool,Int64,Bool,Bool,Bool,Val{:print}}},typeof(PrettyTables._parse_cell_text),UnitRange{Int64}})
    Base.precompile(Tuple{typeof(PrettyTables._fill_row_name_column!),Array{String,2},Array{Int64,2},Array{Array{String,1},2},Array{Array{Int64,1},2},Array{Int64,1},Array{String,1},Array{Int64,1},Array{Int64,1},Int64,Bool,Val{:print},String})
    Base.precompile(Tuple{typeof(PrettyTables._print_table_data),IOContext{Base.GenericIOBuffer{Array{UInt8,1}}},PrettyTables.Display,PrettyTables.ColumnTable,Array{Array{String,1},2},Array{Array{Int64,1},2},Array{Int64,1},Array{Int64,1},Int64,Array{Int64,1},Array{Int64,1},Array{Union{NTuple{4,Int64}, Symbol},1},Array{Union{Int64, Symbol},1},Array{Symbol,1},NTuple{4,Char},Tuple{},Symbol,Int64,Tuple{PrettyTables.Highlighter},Bool,Bool,Bool,PrettyTables.TextFormat,Crayons.Crayon,Crayons.Crayon,Crayons.Crayon})
    Base.precompile(Tuple{typeof(PrettyTables._print_table_header!),IOContext{Base.GenericIOBuffer{Array{UInt8,1}}},PrettyTables.Display,SubArray{String,1,Array{String,1},Tuple{StepRange{Int64,Int64}},true},Array{String,2},Array{Int64,2},Array{Int64,1},Array{Int64,1},Int64,Int64,Array{Int64,1},Array{Int64,1},Array{Symbol,1},Array{Symbol,1},Tuple{},Bool,Bool,PrettyTables.TextFormat,Crayons.Crayon,Array{Crayons.Crayon,1},Array{Crayons.Crayon,1},Crayons.Crayon,Crayons.Crayon})
    Base.precompile(Tuple{typeof(copy),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(PrettyTables.compact_type_str),Tuple{Array{Any,1}}}})

[ronisbr]

@nalimilan normally precompile statements are divided by Julia version and architecture in the packages I looked in the past. Why this is not required here?

[bkamins]

Why this is not required here?

I would say it would be desirable, but it is simply hard do to. We removed 32-bit specific precompilation statements and things that fail on Julia 1.0 manually.

Indeed some things might fail on e.g. Julia 1.3 which we do not check, but if we get such a report I will quickly release patch.

Let us hope that for 1.0 release we will have a more robust workflow for this.

[nalimilan]

I'm not super familiar with precompilation, but if you look at the FixedPointNumbers example that's not the case. AFAICT the only issue with changing Julia versions is that the precompile calls may use unexported Base functions (my last commit), but in theory this could be fixed by dropping them automatically (or checking that they exist).

BTW, I've just noticed that Plots.jl uses CompileBot, which appears to automate what I did here. I guess we could use that in the future (not necessarily via a GitHub Action as running locally works too).

[ronisbr]

@bkamins and @nalimilan one question. I added precompilation locally here to PrettyTables.jl and I could reduce the time to render the first table from 1.8s to 1.0s. However, the loading time (using), increase from 0.16 to 0.38. Is it ok?

[bkamins]

I guess it is OK as it reduces total time to first plot 😄. The question is by how much using DataFrames time is increased.

[ronisbr]

I think it is probably linear, like, it will increase 0,22s here.

[bkamins]

Not necessarily. From my experience there can be some differences as packages might have the same dependencies for instance.

[ronisbr]

Ok! Let me try.

[ronisbr]

Without precompilation of PrettyTables:

1. Load time: 1.21s
2. Time to first table: 3.02s

With precompilation of PrettyTables:

1. Load time: 1.44s
2. Time to first table: 2.65s

IMHO, unless we can improve the precompilation, I do not see a real gain here.

[bkamins]

yeah - it is roughly the same

[nalimilan]

Yeah it doesn't seem great. What did you include in the precompilation phase?

[ronisbr]

Yeah it doesn't seem great. What did you include in the precompilation phase?

I only used the default options of CompileBot. Maybe should I use SnoopCompile directly?

[nalimilan]

I've never used CompileBot. Did it generate many precompile statements? You could also copy the ones I posted above, and which should particularly help DataFrames.

[ronisbr]

I've never used CompileBot. Did it generate many precompile statements? You could also copy the ones I posted above, and which should particularly help DataFrames.

Yes, there are a lot of statements! All those lines you pasted were also generated by CompileBot.

[nalimilan]

OK. So I guess there's nothing to do at this point then. (Precompilation only stores the lowered Julia code currently, so LLVM still has to compile it for the specific CPU, and depending on cases that may be the larger share of the work.)