fix multi cache dir #2639

zwwhdls · 2023-02-23T10:11:50Z

Ⅰ. Describe what this PR does

fix multi cache dir in juicefs runtime

Ⅱ. Does this pull request fix one issue?

fixes #2641

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Signed-off-by: zwwhdls <zww@hdls.me>

zwwhdls · 2023-02-23T10:12:42Z

/test fluid-e2e

cheyang · 2023-02-23T12:32:52Z

pkg/ddc/juicefs/transform_fuse.go

+		// community doc: https://juicefs.com/docs/community/command_reference/#juicefs-mount
+		// enterprise doc: https://juicefs.com/docs/cloud/commands_reference#mount
+		// /mnt/disk1/bigboot or /mnt/disk1/bigboot:/mnt/disk2/bigboot
+		storagePath = tiredStoreLevel.Path


Please fix unit test. thanks.

Signed-off-by: zwwhdls <zww@hdls.me>

zwwhdls · 2023-02-24T02:11:10Z

/retest

codecov · 2023-02-24T02:28:39Z

Codecov Report

Merging #2639 (7f45972) into master (ee79af5) will increase coverage by 0.05%.
The diff coverage is 88.70%.

❗ Current head 7f45972 differs from pull request most recent head 428d28e. Consider uploading reports for the commit 428d28e to get more accurate results

@@            Coverage Diff             @@
##           master    #2639      +/-   ##
==========================================
+ Coverage   67.58%   67.63%   +0.05%     
==========================================
  Files         369      369              
  Lines       21540    21579      +39     
==========================================
+ Hits        14558    14596      +38     
  Misses       5274     5274              
- Partials     1708     1709       +1

Impacted Files	Coverage Δ
pkg/ddc/eac/shutdown.go	`35.95% <0.00%> (ø)`
pkg/ddc/juicefs/operations/base.go	`80.35% <75.00%> (-0.74%)`	⬇️
pkg/ddc/eac/data_load.go	`78.84% <100.00%> (ø)`
pkg/ddc/eac/node.go	`57.14% <100.00%> (ø)`
pkg/ddc/eac/utils.go	`75.93% <100.00%> (+1.13%)`	⬆️
pkg/ddc/eac/worker.go	`67.81% <100.00%> (ø)`
pkg/ddc/juicefs/shutdown.go	`61.11% <100.00%> (+4.96%)`	⬆️
pkg/ddc/juicefs/transform_fuse.go	`78.45% <100.00%> (ø)`
pkg/ddc/juicefs/transform_volume.go	`94.11% <100.00%> (-0.06%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

zwwhdls · 2023-02-24T02:29:25Z

/test fluid-e2e

TrafalgarZZZ · 2023-02-24T06:32:01Z

/hold
hold because multi-dir path won't be cleaned up when shutting down juicefs engine

Signed-off-by: zwwhdls <zww@hdls.me>

zwwhdls · 2023-02-24T11:33:00Z

/test fluid-e2e

Signed-off-by: zwwhdls <zww@hdls.me>

fluid-e2e-bot · 2023-02-25T11:10:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheyang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cheyang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cheyang · 2023-02-25T12:00:32Z

pkg/ddc/juicefs/operations/base.go

+		return
+	}
+	return
+}


To prevent data loss from incorrect directory parameters, I suggest using regular expressions to validate before using "rm -rf" command. Is it possible for this case?

TrafalgarZZZ · 2023-02-27T02:07:03Z

/test fluid-e2e

TrafalgarZZZ · 2023-02-27T03:37:16Z

/retest

TrafalgarZZZ · 2023-02-27T04:00:12Z

/retest

TrafalgarZZZ · 2023-02-27T04:08:48Z

/retest

Signed-off-by: zwwhdls <zww@hdls.me>

zwwhdls · 2023-02-27T05:23:02Z

/test fluid-e2e

cheyang · 2023-02-27T05:59:57Z

pkg/ddc/juicefs/operations/base.go

+		// cache dir check
+		match := ValidCacheDir(dir)
+		if !match {
+			err = fmt.Errorf("dir %s is not valid", dir)


How about using the following info？

err = fmt.Errorf("invalid cache directory %s, skip cleaning up", dir)

@zwwhdls The code now returns an error but I think we should ignore the error if it's not a valid cache dir.

Signed-off-by: zwwhdls <zww@hdls.me>

TrafalgarZZZ · 2023-02-27T06:47:57Z

/unhold
/lgtm

* fix multi cache dir Signed-off-by: zwwhdls <zww@hdls.me> * fix unit test --------- Signed-off-by: zwwhdls <zww@hdls.me>

* fix multi cache dir Signed-off-by: zwwhdls <zww@hdls.me> * fix unit test --------- Signed-off-by: zwwhdls <zww@hdls.me> Signed-off-by: cheyang <cheyang@163.com>

* [juicefs] fix worker cache when set option (#2563) * fix worker cache when set option Signed-off-by: zwwhdls <zww@hdls.me> * update changelog in chart Signed-off-by: zwwhdls <zww@hdls.me> * fix unittest Signed-off-by: zwwhdls <zww@hdls.me> --------- Signed-off-by: zwwhdls <zww@hdls.me> Signed-off-by: cheyang <cheyang@163.com> * fix multi cache dir (#2639) * fix multi cache dir Signed-off-by: zwwhdls <zww@hdls.me> * fix unit test --------- Signed-off-by: zwwhdls <zww@hdls.me> Signed-off-by: cheyang <cheyang@163.com> * [Enhancement]CSI plugin checks mount point liveness before binding mount points (#2703) * Clean up broken mount point when NodeStageVolume Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Check mount point aliveness when NodePublishVolume Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Clean up broken mount point when NodeStageVolume Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Fix cleaning logic Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> --------- Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> Signed-off-by: cheyang <cheyang@163.com> * Prettify error messages for exec.Commands in Fluid (#2718) * Prettify error log message when calling NodePublishVolume Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Prettify error logs when calling helm-related funcs Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Use instead `errors.As` Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Use instead `errors.As` Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> * Set higher log level for helm exec Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> --------- Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> Signed-off-by: cheyang <cheyang@163.com> * update mount to check mountinfo, To #48327952 Signed-off-by: cheyang <cheyang@163.com> * update mount to check mountinfo, To #48327952 Signed-off-by: cheyang <cheyang@163.com> * Build docker images for v0.8.5, To #48327952 Signed-off-by: cheyang <cheyang@163.com> --------- Signed-off-by: zwwhdls <zww@hdls.me> Signed-off-by: cheyang <cheyang@163.com> Signed-off-by: dongyun.xzh <dongyun.xzh@alibaba-inc.com> Co-authored-by: Weiwei <zww@hdls.me> Co-authored-by: TzZtzt <trafalgarz@outlook.com>

fix multi cache dir

0b3044f

Signed-off-by: zwwhdls <zww@hdls.me>

cheyang requested a review from TrafalgarZZZ February 23, 2023 10:29

cheyang reviewed Feb 23, 2023

View reviewed changes

fix unit test

0371305

Signed-off-by: zwwhdls <zww@hdls.me>

cheyang approved these changes Feb 24, 2023

View reviewed changes

fluid-e2e-bot bot added the approved label Feb 24, 2023

fluid-e2e-bot bot added the do-not-merge/hold label Feb 24, 2023

cheyang self-requested a review February 24, 2023 10:01

fix cache clear when shutdown

b541f15

Signed-off-by: zwwhdls <zww@hdls.me>

zwwhdls added 2 commits February 24, 2023 19:35

fix static check

a71b50d

Signed-off-by: zwwhdls <zww@hdls.me>

fix go link

bff002c

Signed-off-by: zwwhdls <zww@hdls.me>

cheyang approved these changes Feb 25, 2023

View reviewed changes

cheyang reviewed Feb 25, 2023

View reviewed changes

cheyang self-requested a review February 25, 2023 12:00

check cachedir valid before remove

f1b5557

Signed-off-by: zwwhdls <zww@hdls.me>

cheyang reviewed Feb 27, 2023

View reviewed changes

zwwhdls added 2 commits February 27, 2023 14:07

update error msg

892dc0d

Signed-off-by: zwwhdls <zww@hdls.me>

fix valid cache dir

428d28e

Signed-off-by: zwwhdls <zww@hdls.me>

fluid-e2e-bot bot removed the do-not-merge/hold label Feb 27, 2023

fluid-e2e-bot bot assigned TrafalgarZZZ Feb 27, 2023

fluid-e2e-bot bot added the lgtm label Feb 27, 2023

fluid-e2e-bot bot merged commit 9834eca into fluid-cloudnative:master Feb 27, 2023

cheyang pushed a commit to cheyang/fluid that referenced this pull request Mar 23, 2023

fix multi cache dir (fluid-cloudnative#2639)

7fb9f64

* fix multi cache dir Signed-off-by: zwwhdls <zww@hdls.me> * fix unit test --------- Signed-off-by: zwwhdls <zww@hdls.me>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix multi cache dir #2639

fix multi cache dir #2639

zwwhdls commented Feb 23, 2023 •

edited by cheyang

Loading

zwwhdls commented Feb 23, 2023

cheyang Feb 23, 2023

zwwhdls Feb 24, 2023

zwwhdls commented Feb 24, 2023

codecov bot commented Feb 24, 2023 •

edited

Loading

zwwhdls commented Feb 24, 2023

TrafalgarZZZ commented Feb 24, 2023

zwwhdls commented Feb 24, 2023

fluid-e2e-bot bot commented Feb 25, 2023

cheyang Feb 25, 2023

TrafalgarZZZ commented Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

zwwhdls commented Feb 27, 2023

cheyang Feb 27, 2023

zwwhdls Feb 27, 2023

TrafalgarZZZ Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

fix multi cache dir #2639

fix multi cache dir #2639

Conversation

zwwhdls commented Feb 23, 2023 • edited by cheyang Loading

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

zwwhdls commented Feb 23, 2023

cheyang Feb 23, 2023

Choose a reason for hiding this comment

zwwhdls Feb 24, 2023

Choose a reason for hiding this comment

zwwhdls commented Feb 24, 2023

codecov bot commented Feb 24, 2023 • edited Loading

Codecov Report

zwwhdls commented Feb 24, 2023

TrafalgarZZZ commented Feb 24, 2023

zwwhdls commented Feb 24, 2023

fluid-e2e-bot bot commented Feb 25, 2023

cheyang Feb 25, 2023

Choose a reason for hiding this comment

TrafalgarZZZ commented Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

TrafalgarZZZ commented Feb 27, 2023

zwwhdls commented Feb 27, 2023

cheyang Feb 27, 2023

Choose a reason for hiding this comment

zwwhdls Feb 27, 2023

Choose a reason for hiding this comment

TrafalgarZZZ Feb 27, 2023

Choose a reason for hiding this comment

TrafalgarZZZ commented Feb 27, 2023

zwwhdls commented Feb 23, 2023 •

edited by cheyang

Loading

codecov bot commented Feb 24, 2023 •

edited

Loading