Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): use rust utilities for caching operations #17638

Merged
merged 6 commits into from
Jun 27, 2023

Conversation

Cammisuli
Copy link
Member

Current Behavior

Nx cache uses fast-glob, fs, and execSync to manage cache. (Finding what's needed to cache, copy and remove)

Expected Behavior

Nx cache uses rust utilities to manage these operations

Related Issue(s)

Fixes #

@Cammisuli Cammisuli requested review from a team, FrozenPandaz and vsavkin as code owners June 16, 2023 15:27
@vercel
Copy link

vercel bot commented Jun 16, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
nx-dev ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 27, 2023 2:00pm

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added walkdir and alphabetized the list

@nx-cloud
Copy link

nx-cloud bot commented Jun 16, 2023

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 6e5787e. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this branch


✅ Successfully ran 1 target

Sent with 💌 from NxCloud.

@@ -83,7 +71,7 @@ export class Cache {
const tdCommit = join(this.cachePath, `${task.hash}.commit`);

// might be left overs from partially-completed cache invocations
await remove(tdCommit);
await this.remove(tdCommit);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this yield performance benefits? 👀

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesnt use the node remove anymore, and the rust one appears to be faster.

packages/nx/src/native/walker.rs Outdated Show resolved Hide resolved
pub fn expand_outputs(directory: String, entries: Vec<String>) -> anyhow::Result<Vec<String>> {
let directory: PathBuf = directory.into();

let (existing_paths, not_found): (Vec<_>, Vec<_>) = entries.into_iter().partition(|entry| {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename not_found to globs?

res();
} catch (e) {
rej(e);
}
});
}

private async remove(path: string): Promise<void> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

followup: Can we make all these methods sync?

crossbeam-channel = '0.5'

fs_extra = "1.3.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is new too right?

Comment on lines +5 to +7
pub fn remove(src: String) -> anyhow::Result<()> {
fs_extra::remove_items(&[src]).map_err(anyhow::Error::from)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason why we wouldn't expose the ability to remove multiple. In theory, it makes no difference but it's a little cleaner I guess.

Is there a way to get the js signature to be ...srcs: string[]?

At the very least, we can make

Cache.remove(...srcs: string[]) {
  require('../native').remove(srcs);
}
Suggested change
pub fn remove(src: String) -> anyhow::Result<()> {
fs_extra::remove_items(&[src]).map_err(anyhow::Error::from)
}
pub fn remove(srcs: Vec<String>) -> anyhow::Result<()> {
fs_extra::remove_items(&[src]).map_err(anyhow::Error::from)
}

crossbeam-channel = '0.5'

fs_extra = "1.3.0"
futures = "0.3.28"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this used?

@FrozenPandaz FrozenPandaz merged commit d327210 into nrwl:master Jun 27, 2023
@github-actions
Copy link

github-actions bot commented Jul 3, 2023

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants