-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #18736 from ninjin/nin/pairconv
RFC: Pair to Pair conversions
- Loading branch information
Showing
2 changed files
with
8 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Executing the daily benchmark build, I will reply here when finished:
@nanosoldier
runbenchmarks(ALL, isdaily = true)
3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels
3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since #18304 (which significantly improved performance in many
nullable
tests, see #18304 (comment)) severalnullable
benchmarks have appeared quite noisy. Do those benchmarks need some sort of re-calibration? Or is some of the recent variation in those benchmarks significant? Best!3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only way to know for sure is to see if you can reproduce regressions locally. The Nullable benchmarks have definitely been really noisy recently, and some other benchmarks seen to have gotten weirdly noisier as well. Some of my recalibrations should land on Nanosoldier next week, which should help; according to my test runs, more thorough GC sweeps cuts down on a lot of problems, but doesn't fix everything.
On the subject of why the Nullable benchmarks might be noisy: In my experience, any benchmark which performs a significant number of heap allocations is prone to variance when run in the context of our server. I don't really understand why (AFAICT, it's not because of GC, which would be the obvious candidate). The variance is unfortunately tough to reproduce; I can run the same benchmark script that Nanosoldier runs, in the same environment, on the same machine, etc. and never see a timing difference of more than 1% between identical Julia builds. However, if I trigger the process via the server's job submission system instead of running it manually, all of sudden 20% changes in performance start popping up...
3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps the job scheduler assigns its processes lower priority than those run from a shell?
3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In both cases, I used
cset
to forcibly bound resources to the benchmarking process, and shield those resources from the scheduler. This document contains all the tricks that I've tried in order to mitigate noise, if you're interested.3398c7e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That writeup is fantastic --- thanks for sharing!