Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

initial implementation of 3to4 migration #39

Merged
merged 14 commits into from
Jul 1, 2016
Merged

initial implementation of 3to4 migration #39

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
0fa3ae4
initial implementation of 3to4 migration
whyrusleeping Jun 25, 2016
b5a024a
fix tests (no more 0.4.0-dev)
whyrusleeping Jun 26, 2016
7a4bd1f
finish initial implementation of 3 to 4 migration
whyrusleeping Jun 28, 2016
e25affb
add in deps
whyrusleeping Jun 28, 2016
2136df7
fixes to make 0.4.3 install and migration work properly
whyrusleeping Jun 29, 2016
4a4f038
add new test for 3-to-4
whyrusleeping Jun 29, 2016
f15036d
make tests for 3-to-4 not use docker (and work)
whyrusleeping Jun 29, 2016
72492a6
add in pollEndpoint dep
whyrusleeping Jun 29, 2016
7b724a7
fix pollEndpoint builds
whyrusleeping Jun 29, 2016
8885034
explicitly add buggy path clean refs
whyrusleeping Jun 29, 2016
5d60b7c
cleanup
whyrusleeping Jun 29, 2016
ba93f55
optimize to rename instead of rewrite
whyrusleeping Jul 1, 2016
da31d7c
handle more heavily mangled keys
whyrusleeping Jul 1, 2016
5f00c7f
clean up empty flatfs directories after migration
whyrusleeping Jul 1, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions ipfs-3-to-4/migration/migration.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,10 @@ func (m Migration) Revert(opts migrate.Options) error {
return err
}

if err := cleanEmptyDirs(filepath.Join(opts.Path, "blocks")); err != nil {
return err
}

log.Log("reverting stored public key records")
if err := rewriteKeys(newds, oldds, "pk", oldKeyFunc("/pk/"), validateNewKey, transferPubKey); err != nil {
return err
Expand Down Expand Up @@ -387,6 +391,38 @@ func transferBlocks(flatfsdir string) error {

fmt.Println()

err := cleanEmptyDirs(flatfsdir)
if err != nil {
fmt.Println(err)
}

return nil
}

func cleanEmptyDirs(dir string) error {
children, err := ioutil.ReadDir(dir)
if err != nil {
return err
}

for _, c := range children {
if !c.IsDir() {
continue
}

cdir := filepath.Join(dir, c.Name())
blocks, err := ioutil.ReadDir(cdir)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: File.Readdirnames would be a better choice. ioutil.ReadDir will perform a stat on each directory entry and then sort them by name. Not sure if it really matters so fell free to ignore.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are only few files in most dirs at a time, but using that instead wouldn't hurt in case of big repos.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevina I actually dont think it calls stat individually on each one. take a look at the go source in os/dir_unix.go

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@whyrusleeping the file you pointed me to only contains an implementation for readdirnames, readdir is implemeted in os/file_unix.go and it actually calls Readdirnames first and than calls lstat for each entry. I don't see how it could be implemented any other way because of how the related system calls on unix work.

if err != nil {
return err
}

if len(blocks) == 0 {
err := os.Remove(cdir)
if err != nil {
return err
}
}
}
return nil
}

Expand Down