Skip to content
This repository has been archived by the owner on Jul 20, 2023. It is now read-only.

feat: adds cleaner utility for orphaned resources #34

Merged
merged 10 commits into from
Jan 7, 2021
Merged

Conversation

telpirion
Copy link
Contributor

This PR adds the clean.js script to clean up old, stale, or orphaned resources greater than 2 days old. It deletes the following types of resources:

  • datasets
  • models
  • training pipelines
  • endpoints
  • batch predictions

Fixes #26

@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jan 6, 2021
@codecov
Copy link

codecov bot commented Jan 6, 2021

Codecov Report

Merging #34 (77787f8) into master (bf0b768) will decrease coverage by 0.00%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
- Coverage   98.82%   98.81%   -0.01%     
==========================================
  Files          15       15              
  Lines       20493    20493              
  Branches      575      575              
==========================================
- Hits        20253    20251       -2     
- Misses        232      234       +2     
  Partials        8        8              
Impacted Files Coverage Δ
src/v1beta1/endpoint_service_client.ts 98.97% <0.00%> (-0.09%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bf0b768...77787f8. Read the comment docs.

@telpirion telpirion marked this pull request as ready for review January 6, 2021 18:15
@telpirion telpirion requested review from a team as code owners January 6, 2021 18:15
Copy link
Contributor

@bcoe bcoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple nits, thank you for doing this work 👏

samples/test/clean.js Show resolved Hide resolved
datasetDeletionOperations.push(deletionOp);
}

if (datasetDeletionOperations.length > MAXIMUM_NUMBER_OF_DELETIONS) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason you're bumping into an LRO issue, I expect, is that Promise.all will start all the work in parallel, leading to resource contention.

Instead you could just do:

for (const dataset of datasets) {
  await datasetDeletionOperations(dataset)
}

And I think wouldn't need to cap the amount of cleanup you do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add garbage collector mechanism for unneeded resources
2 participants