Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix FileSize to return correct disk usage recursively #5384

Merged
merged 3 commits into from
Jan 19, 2021

Conversation

nonsense
Copy link
Member

@nonsense nonsense commented Jan 19, 2021

At the moment FileSize is not recursive and doesn't calculate correctly the disk usage of various paths.

This PR is fixing it, by recursively walking the path and adding together all file sizes.


Ultimately this is fixing a bug in the stores package, where we iterate over all storage reservations and then check how much disk space is used per sector:

	for id, ft := range p.reservations {
		for _, fileType := range storiface.PathTypes {
			if fileType&ft == 0 {
				continue
			}

			sp := p.sectorPath(id, fileType)
			used, err := ls.DiskUsage(sp)

sp in this case has the form of ./cache/s-t0127896-36 which is actually a directory with many .dat files, and not a single file.


TODO:

  • test with lotus-miner

@nonsense
Copy link
Member Author

nonsense commented Jan 19, 2021

I think this should be correct now. On my miner all the Reserved space is now gone as expected:

before

9c92d406-1e0b-4edd-93ba-1f413ca5431d:
        [####################################**************] 1.789 TiB/1.789 TiB 100%
        Unsealed: 2; Sealed: 1; Caches: 1; Reserved: 483.2 GiB
        Weight: 10; Use: Seal
        Local: /root/storage_sealing
        URL: http://127.0.0.1:2345/remote

after

9c92d406-1e0b-4edd-93ba-1f413ca5431d:
        [##########################################        ] 1.51 TiB/1.789 TiB 84%
        Unsealed: 2; Sealed: 2; Caches: 2; Reserved: 0 B
        Weight: 10; Use: Seal
        Local: /root/storage_sealing
        URL: http://127.0.0.1:2345/remote

(both before and after keep 2 sectors in the sealing directory)

return err
}
if !info.IsDir() {
stat := info.Sys().(*syscall.Stat_t)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this is a _unix.go file, I think we must always have a value in Sys(), so I don't think we need to check for nil pointer here.

@nonsense nonsense marked this pull request as ready for review January 19, 2021 17:20
@magik6k
Copy link
Contributor

magik6k commented Jan 19, 2021

Closes #3124

@magik6k magik6k linked an issue Jan 19, 2021 that may be closed by this pull request
@magik6k magik6k merged commit 48c6c55 into master Jan 19, 2021
@magik6k magik6k deleted the fix-storage-size-estimation branch January 19, 2021 18:22
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(LGTM)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reserving too much space
2 participants