Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge two experimental_local_disk_cache options to one #4874

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions site/docs/remote-caching.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ make builds significantly faster.
* [Delete content from the remote cache](#delete-content-from-the-remote-cache)
* [Known Issues](#known-issues)
* [External Links](#external-links)
* [Build cache](#build-cache)
* [Bazel remote execution (in development)](#bazel-remote-execution-in-development)

## Remote caching overview
Expand Down Expand Up @@ -306,6 +307,31 @@ You may want to delete content from the cache to:
* Create a clean cache after a cache was poisoned
* Reduce the amount of storage used by deleting old outputs

## Build cache

Bazel can use a directory on the filesystem as a remote cache. This is
useful for sharing build artifacts when switching branches and/or working
on multiple workspaces of the same project i.e. multiple checkouts. Bazel
does not garbage collect the directory and thus one might want to set up a
cron job or similar to periodically clean up the directory. The build cache
can be enabled with the following flags.

```
build --build_cache=/path/to/build/cache
```

Additionally, one can pass a user specific path to the --build_cache flag
via the ~ character. Bazel will replace this with the current user's home
directory. This comes in handy when one wants to enable the build cache for
all developers of a project via the project's checked in `.bazelrc` file.

To have cache hits across different workspaces, additional option should be
used:
Copy link
Contributor Author

@davido davido Mar 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the listing of the env. variables. The use case I'm after is: the same user (no multiple uses involved) has different clones of the same project, say:

  • /home/davido/projects/gerrit
  • /home/davido/projects/gerrit2

After building in gerrit directory, the cache is populated. When I build in gerrit2 directory, without --experimental_strict_action_env I experienced cache misses. I havn't re-checked whether this is still the case, but @ulfjack recommended to use this option in this comment: #3042 (comment) and pointed to the discussion in context of this issue: #2574.

I should have probably said: "To have cache hits across different clones of the same project..."?


```
build --experimental_strict_action_env
```

## Known issues

**Input file modification during a Build**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ public Iterable<? extends ActionContext> getActionContexts() {
String buildRequestId = env.getBuildRequestId().toString();
String commandId = env.getCommandId().toString();

if (remoteOptions.experimentalRemoteSpawnCache || remoteOptions.experimentalLocalDiskCache) {
if (remoteOptions.experimentalRemoteSpawnCache || remoteOptions.buildCache != null) {
RemoteSpawnCache spawnCache =
new RemoteSpawnCache(
env.getExecRoot(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -190,27 +190,18 @@ public final class RemoteOptions extends OptionsBase {
)
public boolean experimentalRemoteSpawnCache;

// TODO(davido): Find a better place for this and the next option.
// TODO(davido): Find a better place for this option.
@Option(
name = "experimental_local_disk_cache",
defaultValue = "false",
category = "remote",
documentationCategory = OptionDocumentationCategory.UNCATEGORIZED,
effectTags = {OptionEffectTag.UNKNOWN},
help = "Whether to use the experimental local disk cache."
)
public boolean experimentalLocalDiskCache;

@Option(
name = "experimental_local_disk_cache_path",
name = "build_cache",
defaultValue = "null",
category = "remote",
documentationCategory = OptionDocumentationCategory.UNCATEGORIZED,
effectTags = {OptionEffectTag.UNKNOWN},
converter = OptionsUtils.PathFragmentConverter.class,
help = "A file path to a local disk cache."
help = "A path to a directory where Bazel can read and write actions and action outputs. "
+ "If the directory does not exist, it will be created."
)
public PathFragment experimentalLocalDiskCachePath;
public PathFragment buildCache;

@Option(
name = "experimental_guard_against_concurrent_changes",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,10 @@ public static SimpleBlobStore createRest(RemoteOptions options, Credentials cred
}
}

public static SimpleBlobStore createLocalDisk(RemoteOptions options, Path workingDirectory)
public static SimpleBlobStore createBuildCache(RemoteOptions options, Path workingDirectory)
throws IOException {
return new OnDiskBlobStore(
workingDirectory.getRelative(checkNotNull(options.experimentalLocalDiskCachePath)));
workingDirectory.getRelative(checkNotNull(options.buildCache)));
}

public static SimpleBlobStore create(
Expand All @@ -58,20 +58,20 @@ public static SimpleBlobStore create(
if (isRestUrlOptions(options)) {
return createRest(options, creds);
}
if (workingDirectory != null && isLocalDiskCache(options)) {
return createLocalDisk(options, workingDirectory);
if (workingDirectory != null && isBuildCache(options)) {
return createBuildCache(options, workingDirectory);
}
throw new IllegalArgumentException(
"Unrecognized concurrent map RemoteOptions: must specify "
+ "either Rest URL, or local cache options.");
}

public static boolean isRemoteCacheOptions(RemoteOptions options) {
return isRestUrlOptions(options) || isLocalDiskCache(options);
return isRestUrlOptions(options) || isBuildCache(options);
}

public static boolean isLocalDiskCache(RemoteOptions options) {
return options.experimentalLocalDiskCache;
public static boolean isBuildCache(RemoteOptions options) {
return options.buildCache != null;
}

private static boolean isRestUrlOptions(RemoteOptions options) {
Expand Down
4 changes: 2 additions & 2 deletions src/test/shell/bazel/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -318,9 +318,9 @@ sh_test(
)

sh_test(
name = "local_action_cache_test",
name = "build_cache_test",
size = "small",
srcs = ["local_action_cache_test.sh"],
srcs = ["build_cache_test.sh"],
data = [":test-deps"],
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ function test_local_action_cache() {
local execution_file="${TEST_TMPDIR}/run.log"
local input_file="foo.in"
local output_file="bazel-genfiles/foo.txt"
local flags="--experimental_local_disk_cache_path=$cache --experimental_local_disk_cache"
local flags="--build_cache=$cache"

rm -rf $cache
mkdir $cache
Expand Down