Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the time-to-first-request by adding pregenerated precompile statements #805

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

DilumAluthge
Copy link
Member

@DilumAluthge DilumAluthge commented Mar 10, 2022

Summary

This pull request makes two changes:

  1. Reduce the time-to-first-request by adding a pregenerated list of precompile statements.
  2. Add a GitHub Actions workflow for automatically regenerating list of precompile statements.

Description

This pull request reduces the time to first HTTP.get("https://example.com/") as shown in the following table. All times are in seconds.

Branch Sample Size Median Δ vs master Mean ± Standard Deviation Δ vs master
master 20 5.252848 5.318844 ± 0.241039
dpa/precompile 20 3.723808 -1.529040 3.947450 ± 0.104822 -1.371394

All measurements were taken using Julia version 1.9.0-DEV.165:

$ julia --project -e 'import InteractiveUtils; InteractiveUtils.versioninfo()'
Julia Version 1.9.0-DEV.165
Commit 8076517c97 (2022-03-09 19:46 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin20.6.0)
  CPU: 4 × Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, haswell)
  Threads: 1 on 4 virtual cores

Note: this pull request does not make any web requests when the user does Pkg.precompile() or import HTTP.

List of time measurements

Time measurements for the master branch (click to expand)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.332501 seconds (7.34 M allocations: 373.428 MiB, 3.00% gc time, 98.47% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.323024 seconds (7.34 M allocations: 373.428 MiB, 3.16% gc time, 98.49% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.236754 seconds (7.34 M allocations: 373.428 MiB, 3.13% gc time, 98.42% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.235521 seconds (7.34 M allocations: 373.428 MiB, 3.15% gc time, 98.30% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.219502 seconds (7.34 M allocations: 373.428 MiB, 3.14% gc time, 98.42% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.235383 seconds (7.34 M allocations: 373.428 MiB, 3.15% gc time, 98.35% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  6.321999 seconds (7.34 M allocations: 373.428 MiB, 2.64% gc time, 98.65% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.250412 seconds (7.34 M allocations: 373.428 MiB, 3.05% gc time, 98.41% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.323672 seconds (7.34 M allocations: 373.428 MiB, 3.16% gc time, 98.44% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.255284 seconds (7.34 M allocations: 373.428 MiB, 3.09% gc time, 98.32% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.339982 seconds (7.34 M allocations: 373.428 MiB, 3.25% gc time, 98.50% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.318435 seconds (7.34 M allocations: 373.428 MiB, 3.17% gc time, 98.35% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.211436 seconds (7.34 M allocations: 373.428 MiB, 3.14% gc time, 98.43% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.285089 seconds (7.34 M allocations: 373.428 MiB, 2.98% gc time, 98.49% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.203090 seconds (7.34 M allocations: 373.428 MiB, 3.12% gc time, 98.39% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.342181 seconds (7.34 M allocations: 373.428 MiB, 3.08% gc time, 98.47% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.191504 seconds (7.34 M allocations: 373.428 MiB, 3.37% gc time, 98.46% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.272862 seconds (7.34 M allocations: 373.428 MiB, 3.03% gc time, 98.45% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.238501 seconds (7.34 M allocations: 373.428 MiB, 3.32% gc time, 98.49% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.239750 seconds (7.34 M allocations: 373.428 MiB, 3.05% gc time, 98.46% compilation time)
Time measurements for the dpa/precompile branch (click to expand)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  4.626500 seconds (76.96 k allocations: 4.030 MiB, 97.84% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  4.705703 seconds (76.96 k allocations: 4.030 MiB, 97.70% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  4.739228 seconds (76.96 k allocations: 4.030 MiB, 97.95% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.817014 seconds (76.96 k allocations: 4.030 MiB, 97.74% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.720888 seconds (76.96 k allocations: 4.030 MiB, 97.91% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.679500 seconds (76.96 k allocations: 4.030 MiB, 97.84% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.669254 seconds (76.96 k allocations: 4.030 MiB, 97.68% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.733760 seconds (76.96 k allocations: 4.030 MiB, 97.75% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  5.142341 seconds (76.96 k allocations: 4.030 MiB, 97.91% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.845968 seconds (76.96 k allocations: 4.030 MiB, 97.80% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.713366 seconds (76.96 k allocations: 4.031 MiB, 97.76% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.703498 seconds (76.96 k allocations: 4.030 MiB, 97.80% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.734557 seconds (76.96 k allocations: 4.030 MiB, 97.74% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.746249 seconds (76.96 k allocations: 4.030 MiB, 97.76% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.782660 seconds (76.96 k allocations: 4.031 MiB, 97.74% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.741732 seconds (76.96 k allocations: 4.030 MiB, 97.64% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.693289 seconds (76.96 k allocations: 4.030 MiB, 97.79% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.713094 seconds (76.96 k allocations: 4.030 MiB, 97.84% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.726727 seconds (76.96 k allocations: 4.030 MiB, 97.77% compilation time)
$ julia --project -e 'import HTTP; @time(HTTP.get("https://example.com"));'
  3.713672 seconds (76.96 k allocations: 4.030 MiB, 97.88% compilation time)

How to regenerate the list of precompile statements

To regenerate the list of precompile statements, there are two options:

  1. If you are a committer on this repository, you can go to the Actions tab of this repository, select the Precompile workflow, and manually trigger a new workflow run. This will automatically open a pull request to update the precompile statements.
  2. You can clone the repository locally and run the following command: julia --project contrib/precompile_generate.jl

@codecov-commenter
Copy link

codecov-commenter commented Mar 10, 2022

Codecov Report

Attention: Patch coverage is 0% with 61 lines in your changes missing coverage. Please review.

Project coverage is 74.79%. Comparing base (96a0b34) to head (73b1344).

Current head 73b1344 differs from pull request most recent head a0df8f8

Please upload reports for the commit a0df8f8 to get more accurate results.

Files Patch % Lines
src/precompile.jl 0.00% 61 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (96a0b34) and HEAD (73b1344). Click for more details.

HEAD has 15 uploads more than BASE | Flag | BASE (96a0b34) | HEAD (73b1344) | |------|------|------| ||3|18|
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #805      +/-   ##
==========================================
- Coverage   83.06%   74.79%   -8.28%     
==========================================
  Files          32       40       +8     
  Lines        3089     2567     -522     
==========================================
- Hits         2566     1920     -646     
- Misses        523      647     +124     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

end
""")
println(io, """
@static if Base.VERSION < v"1.9-"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thanks for the PR @DilumAluthge ! I am surprised that this is only enabled for Julia 1.9 and up. We found that adding precompile calls also gave improvements in Julia 1.8.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DilumAluthge why this is only enabled for 1.9 and later?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I was under the impression that without JuliaLang/julia#43990, we can't cache CodeInstances from other packages, and thus the benefit of this would be far less noticeable.

@fonsp
Copy link
Member

fonsp commented Mar 25, 2022

Awesome! I would also like to add precompile statements for HTTP's server abilities, but this PR gives me all the tools to do that in a future PR 😊

@rikhuijzer
Copy link
Contributor

How about writing to a release branch based on the version mentioned in Project.toml? Then, only the release branch contains all the precompile statements and that branch is the one where people download the package from. This avoids having to look at the generated output in the diff all the time. Especially for larger packages, the generated output will contain quite a lot of lines.

@fredrikekre
Copy link
Member

I thought this type of explicit precompile statements wasn't recommended anymore? Experience with Plots.jl also suggest it is a pretty fragile method.

@rikhuijzer
Copy link
Contributor

I thought this type of explicit precompile statements wasn't recommended anymore?

Depends on the use-case. If precompile can infer all types on the execution path, then it will work great. If it can't, it will waste time on trying to guess types, give up and then the types have to be inferred again during running time.

precompile(Tuple{typeof(Base.bytesavailable), Sockets.TCPSocket})
precompile(Tuple{typeof(Base.eof), Sockets.TCPSocket})
precompile(Tuple{typeof(Base.alloc_buf_hook), Sockets.TCPSocket, UInt64})
precompile(Tuple{Base.var"#readcb_specialized#671", Sockets.TCPSocket, Int64, UInt64})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
precompile(Tuple{Base.var"#readcb_specialized#671", Sockets.TCPSocket, Int64, UInt64})

This line causes

(HTTP) pkg> precompile
Precompiling project...
  ✗ HTTP
  0 dependencies successfully precompiled in 4 seconds. 4 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

Failed to precompile HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] to /home/rik/.julia/compiled/v1.9/HTTP/jl_pwtQbZ.
ERROR: LoadError: UndefVarError: #readcb_specialized#671 not defined
Stacktrace:
 [1] _precompile()
   @ HTTP ~/git/HTTPDilum/src/precompile.jl:46
 [2] top-level scope
   @ ~/git/HTTPDilum/src/HTTP.jl:627
 [3] include
   @ ./Base.jl:427 [inlined]
 [4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base ./loading.jl:1423
 [5] top-level scope
   @ stdin:1
in expression starting at /home/rik/git/HTTPDilum/src/HTTP.jl:1
in expression starting at stdin:1

on Julia 1.9.0-DEV.428.

@rikhuijzer
Copy link
Contributor

@DilumAluthge It's best to do compilation time benchmarks with @eval: @time @eval HTTP.get("https://example.com");. From the docstring of @time:

In some cases the system will look inside the @time expression and compile some of the called code before execution of the top-level expression begins. When that happens, some compilation time will not be counted. To include this time you can run @time @eval ....

In this case, I do think that your benchmark is valid though. I could reproduce similar numbers on Julia 1.9.0-DEV.428:

master

$ julia --project -e 'import HTTP; @time @eval HTTP.get("https://example.com");'
  4.518353 seconds (5.63 M allocations: 368.014 MiB, 2.61% gc time, 91.25% compilation time)

This PR

$ julia --project -e 'import HTTP; @time @eval HTTP.get("https://example.com");'
  3.231371 seconds (147.83 k allocations: 8.562 MiB, 86.99% compilation time)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants