Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement transient file tracking and cleanup #26

Closed
4 tasks done
raulk opened this issue Jul 5, 2021 · 1 comment · Fixed by #49 or #52
Closed
4 tasks done

implement transient file tracking and cleanup #26

raulk opened this issue Jul 5, 2021 · 1 comment · Fixed by #49 or #52
Assignees
Milestone

Comments

@raulk
Copy link
Member

raulk commented Jul 5, 2021

We use a mount.Upgrader to mirror remote shards into local files. Those local files are currently (a) regular temporary files, and they are (b) not managed. We need to do the following in the upgrader:

  • pass in the transient root directory to upgraders, so that transients are created inside that directory (referred to as "scratch space")
  • replace the usage of temporary files with regular files under that directory.
  • add a method DeleteTransient() error in the upgrader that deletes the transient file under the lock.

In the DAG store management layer, we need to do the following:

  • expose a method GC() error that deletes the transient on all shards whose refcount is 0 (unused, i.e. not serving).
@raulk raulk added this to the v0.1 milestone Jul 5, 2021
@raulk raulk changed the title transient file tracking and cleanup implement transient file tracking and cleanup Jul 5, 2021
@raulk raulk closed this as completed in #49 Jul 10, 2021
@raulk
Copy link
Member Author

raulk commented Jul 10, 2021

The DAGStore#GC() error method was not implemented in #49. Reopening.

@raulk raulk reopened this Jul 10, 2021
@raulk raulk closed this as completed Jul 11, 2021
@raulk raulk self-assigned this Jul 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment