-
-
Notifications
You must be signed in to change notification settings - Fork 726
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GH2099: Cache compiled script on disk #2650
Conversation
Related to #2584 @tstewart65 do you possible have some numbers on the performance improvements? Would be nice to see how much time we save with just caching the compilation. |
Results from a very simple cake script that exists of one task and all it does it print out "Hello". Note: Test are an average of 6 runs with tools and nugget packages already installed. Next simple test that shows even more saving and uses a version of Cake.Recipe. The test consists of 3 simple tasks but two of the tasks uses the RequireTool call tools.cake. I made a change to the RequiresTool call to use a unique name for each script instead of a GUID so this script could be cached also. Real world example is a project that I am currently working on. Below is the Verbose output from the two different basic tests no cache and cached.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR. Had a quick look and left some thoughts.
18f1cbd
to
04d2e50
Compare
04d2e50
to
b4d5851
Compare
I updated the code to the changes made in #3381 and rebased on latest |
42648c0
to
27e9079
Compare
c6d4613
to
94e263f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
I created cake-build/resources#104 to keep track of the need to update the |
55896df
to
d4f657f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to see all changes made in Cake.Core
moved out to Cake
. Not a hill I'm prepared to die on, but I'd argue it makes more sense there as script cache, and compilation is something that happens in Cake
, and not in Cake.Core
.
@bjorkstromm are you proposing we move the Configuration parts from |
No, I'm only proposing the additions in this PR (one extension method, and two constants) would be better suited living in Cake than Cake.Core. |
Yip makes sense to me. Agree 👍 |
312609a
to
23fb41c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
* Enable * Config: Settings.EnableScriptCache=true * Arg: --settings_enablescriptcache=true * Env: CAKE_SETTINGS_ENABLESCRIPTCACHE=true * Cache Location * Config: Paths.Cache = /path/to/cache * Arg: --paths_cache=/path/to/cache * Env: CAKE_PATHS_CACHE=/path/to/cache * fixes cake-build#2099 Co-authored-by: devlead@users.noreply.github.com
23fb41c
to
f4c9ac7
Compare
@tstewart65 your changes have been merged, thanks for your contribution 👍 |
First of all, amazing work and I am really happy to see this being built into Cake. We use the same scripts for many of our repositories, but would like to cache this in a generic way on our build agents. For example, we have 150 repositories that would be using this feature and it would be hard to explicitly set a different path for each of them in cake.config. I checked the source, but could not find an easy answer to whether this will cache in the repository itself or in the temp path. If it's in the temp path, how will it differentiate between different working directories, etc? Basically what we are looking for is how we can cache this:
If this all works as I hope, this will shave up 4 minutes of each build since we have 6 stages and each of them need to (re)compile the scripts right now which takes up about 45 seconds (large scripts). |
@GeertvanHorrik by default it will end up in a cache folder within the default tools folder. But cache location fully configurable
|
Thanks for the fast reply. We even templated our cake.config. Would it be possible to use variables or how does it determine the final subfolder inside the cache. For example (and I am happy to contribute), something like this:
%temp%\cake-scripts-cache[some-unique-hash] I see 2 things that would be needed here:
|
So currently the implementation is cake/src/Cake/Infrastructure/Scripting/RoslynScriptSession.cs Lines 100 to 104 in 6e9b6a8
So configured cache path + filename without path + dll / hash Default configuration handling is here
|
Thanks. I think that I need 1 more sublevel for the cache (otherwise it will keep being invalidated), but maybe I should try the feature first and only if it turns out to be an issue, force the cache directory per repository. Looking forward to try this one out, I assume it will be part of Cake 2.2? Any ETA or pre-release available? |
Pre-releases are always available via our public Azure Artifacts feed 2.2 is preliminarily planned to be released this or next week. As you're using [Paths]
Cache=%TEMP%/tools |
I would need 1 more level in caching, e.g.: [Paths] I am probably overthinking and should just take the cake.config out of the repository template. |
@GeertvanHorrik That actually already works, I'll pr an integration test to demonstrate and fix a typo in this one. |
That is perfect, then I can't wait to try the new version and can keep cake.config in my repository synchronizer template. |
This pull request is what started RFC-02. At the time I was not able to submit a pull request for the changes.
Differences from RFC-02
I have been using several versions of Cake with this applied and have had no issues.
@mholo65 posted this in the RFC Comments
"Caching compilation is not alone enough. We must also cache registered tools resolved via #tool directive. Addins (and their dependencies) must also be loaded into app domain / load context in same order as the first time when resolved via #addin directive."
For the way this is implemented I feel just caching the compilation is enough. I need all of the lines of the script in order to do the hash for determining if anything has changed. The tools and addins are still being found like usual. The main changes were made in RoslynScriptSession and in there it just determines if it needs to compile the script or run the cached DLL.