[pkg] panic if no space is left #3636

huof6829 · 2022-09-13T10:08:59Z

Description

Type of change

Please delete options that are not relevant.

[] Bug fix (non-breaking change which fixes an issue)
[] New feature (non-breaking change which adds functionality)
[] Code refactor or improvement
[] Breaking change (fix or feature that would cause a new or changed behavior of existing functionality)
[] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[] make test
[] fullsync
[] Other test (please specify)

Test Configuration:

Firmware version:
Hardware:
Toolchain:
SDK:

Checklist:

[] My code follows the style guidelines of this project
[] I have performed a self-review of my code
[] I have commented my code, particularly in hard-to-understand areas
[] I have made corresponding changes to the documentation
[] My changes generate no new warnings
[] I have added tests that prove my fix is effective or that my feature works
[] New and existing unit tests pass locally with my changes
[] Any dependent changes have been merged and published in downstream modules

codecov · 2022-09-13T10:18:40Z

Codecov Report

Merging #3636 (b6bbdd2) into master (e9732a1) will decrease coverage by 0.28%.
The diff coverage is 80.72%.

@@            Coverage Diff             @@
##           master    #3636      +/-   ##
==========================================
- Coverage   74.95%   74.67%   -0.29%     
==========================================
  Files         269      267       -2     
  Lines       23819    23784      -35     
==========================================
- Hits        17854    17761      -93     
- Misses       5039     5090      +51     
- Partials      926      933       +7

Impacted Files	Coverage Δ
db/config.go	`100.00% <ø> (ø)`
db/trie/mptrie/node.go	`100.00% <ø> (ø)`
ioctl/newcmd/action/action.go	`79.62% <ø> (+0.09%)`	⬆️
ioctl/newcmd/node/nodedelegate.go	`70.19% <0.00%> (-0.95%)`	⬇️
ioctl/newcmd/node/nodeprobationlist.go	`90.90% <0.00%> (-4.33%)`	⬇️
server/itx/server.go	`60.00% <25.00%> (-2.21%)`	⬇️
action/protocol/execution/evm/evm.go	`44.57% <26.66%> (-0.86%)`	⬇️
db/trie/mptrie/hashnode.go	`36.36% <41.17%> (-54.55%)`	⬇️
db/trie/mptrie/merklepatriciatrie.go	`78.73% <46.80%> (-12.70%)`	⬇️
db/db_bolt.go	`72.13% <50.00%> (-0.39%)`	⬇️
... and 23 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

huangzhiran · 2022-09-13T12:29:21Z

pkg/fsnotify/fsnotify.go

+
+// checkDiskSpace returns the disk usage info
+func checkDiskSpace(ctx context.Context) {
+	usage, err := disk.Usage("/")


if os disk partition like this

/ /dev/sda /run /dev/sdb

and iotex-core run within /run, the check may not work

"/" checks all disks

@huangzhiran +1

/ /dev/sda 500G /run /dev/sdb 100MB

if run on /run, it will failed

huangzhiran · 2022-09-13T12:38:18Z

pkg/fsnotify/fsnotify.go

+				continue
+			}
+			// check disk space per write operation
+			if ev.Has(fsnotify.Write) {


fsnotify.Create needed too

It's still left enough space to run program even if Avail is 0 after run df -h .

huangzhiran · 2022-09-13T12:41:26Z

pkg/fsnotify/fsnotify.go

+			}
+			// check disk space per write operation
+			if ev.Has(fsnotify.Write) {
+				checkDiskSpace(ctx)


all filesystem write ops trig a check, may inefficient

millken · 2022-09-14T06:57:03Z

I think we don't need to monitor disk, just need to PANIC when write DB error is disk full

huof6829 · 2022-09-14T12:47:47Z

add log rotate and delete expired logs can be better.

Liuhaai · 2022-09-14T16:30:21Z

pkg/fsnotify/fsnotify.go

+		select {
+		case <-ctx.Done():
+			return
+		case ev, ok := <-watcher.Events:


when does an event come

dustinxie · 2022-09-19T02:09:06Z

pkg/watch/watch.go

+	if err := task.Start(ctx); err != nil {
+		log.L().Panic("Failed to start watch disk space", zap.Error(err))
+	}
+	return func() {


why return a func() here?

dustinxie · 2022-09-19T02:12:14Z

server/main.go

@@ -103,6 +104,10 @@ func main() {
 		glog.Fatalln("Cannot config global logger, use default one: ", zap.Error(err))
 	}

+	// check device
+	stopWatch := watch.Start(ctx, 5*time.Minute)
+	defer stopWatch()


just call watch.Start() in server.Start(), and call watch.Stop() in server.Stop()?

Just check like heartbeat. server contains the major components like api blockchain etc...

Liuhaai · 2022-09-27T19:48:03Z

pkg/watch/watch.go

+// permitted by law, all liability for your use of the code is disclaimed. This source code is governed by Apache
+// License 2.0 that can be found in the LICENSE file.
+
+package watch


package disk

Liuhaai · 2022-09-27T19:49:43Z

pkg/watch/watch.go

+)
+
+// Start creates a timer task to check device per watchInternal
+func Start(ctx context.Context, watchInternal time.Duration) func() {


func NewMonitor(time.Duration) *monitor func (*monitor) Start() func (*monitor) Stop()

millken · 2022-10-11T08:35:39Z

pkg/disk/monitor.go

+		return
+	}
+	// panic if left less than 2%
+	if usage.UsedPercent > 98.0 || usage.InodesUsedPercent > 98.0 {


if disk is 20TB, when there is %2=200GB of free disk, can't it be used?

leave enough disk space is required when disk run out soon.

millken · 2022-10-11T08:40:42Z

pkg/fsnotify/fsnotify.go

+
+// checkDiskSpace returns the disk usage info
+func checkDiskSpace(ctx context.Context) {
+	usage, err := disk.Usage("/")


@huangzhiran +1

/ /dev/sda 500G /run /dev/sdb 100MB

if run on /run, it will failed

CoderZhi · 2022-10-11T18:06:45Z

@huof6829 @millken has the problem been resolved?

huof6829 · 2022-10-12T02:46:51Z

fixed. I have no problem.

sonarcloud · 2022-10-12T09:52:11Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

add fsnotify

6dc29de

huof6829 requested a review from a team as a code owner September 13, 2022 10:09

huangzhiran reviewed Sep 13, 2022

View reviewed changes

Liuhaai reviewed Sep 14, 2022

View reviewed changes

pkg/fsnotify/fsnotify.go Outdated

select {

case <-ctx.Done():

return

case ev, ok := <-watcher.Events:

Copy link

Member

Liuhaai Sep 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when does an event come

huof6829 added 4 commits September 16, 2022 11:31

Merge branch 'master' into panic_no_space

0bbce28

modify as 5-minutes timer loop

d0d57a4

add comment

e4688a8

add comment

7c3bf2d

dustinxie reviewed Sep 19, 2022

View reviewed changes

huof6829 added 4 commits September 21, 2022 12:48

move to maintainment of StartServer

dc643c7

Merge branch 'master' into panic_no_space

2d5eac7

Merge branch 'master' into panic_no_space

0fd1091

add BenchmarkCheckSpace

9f3b564

Liuhaai reviewed Sep 27, 2022

View reviewed changes

huof6829 added 3 commits September 28, 2022 20:12

new struct

2d38686

Merge branch 'master' into panic_no_space

259f084

modify comments

7919ab0

Liuhaai approved these changes Sep 28, 2022

View reviewed changes

Merge branch 'master' into panic_no_space

41f1a09

millken reviewed Oct 11, 2022

View reviewed changes

huof6829 added 2 commits October 12, 2022 17:49

check db partion

b3d8b12

Merge branch 'master' into panic_no_space

b6bbdd2

huangzhiran approved these changes Oct 12, 2022

View reviewed changes

huof6829 merged commit 62e8eb7 into iotexproject:master Oct 13, 2022

huof6829 deleted the panic_no_space branch October 13, 2022 02:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pkg] panic if no space is left #3636

[pkg] panic if no space is left #3636

huof6829 commented Sep 13, 2022

codecov bot commented Sep 13, 2022 •

edited

Loading

huangzhiran Sep 13, 2022

huof6829 Sep 14, 2022

millken Oct 11, 2022

huangzhiran Sep 13, 2022

huof6829 Sep 14, 2022

huangzhiran Sep 13, 2022

millken commented Sep 14, 2022

huof6829 commented Sep 14, 2022

Liuhaai Sep 14, 2022

dustinxie Sep 19, 2022

dustinxie Sep 19, 2022

huof6829 Sep 20, 2022

Liuhaai Sep 27, 2022 •

edited

Loading

Liuhaai Sep 27, 2022

huof6829 Sep 28, 2022

millken Oct 11, 2022

huof6829 Oct 11, 2022

millken Oct 11, 2022

CoderZhi commented Oct 11, 2022

huof6829 commented Oct 12, 2022

sonarcloud bot commented Oct 12, 2022

[pkg] panic if no space is left #3636

[pkg] panic if no space is left #3636

Conversation

huof6829 commented Sep 13, 2022

Description

Type of change

How Has This Been Tested?

Checklist:

codecov bot commented Sep 13, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

millken commented Sep 14, 2022

huof6829 commented Sep 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Liuhaai Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CoderZhi commented Oct 11, 2022

huof6829 commented Oct 12, 2022

sonarcloud bot commented Oct 12, 2022

codecov bot commented Sep 13, 2022 •

edited

Loading

Liuhaai Sep 27, 2022 •

edited

Loading