move dataflow affinity logic to fluidapp controller #4138

xliuqq · 2024-05-28T06:06:05Z

Ⅰ. Describe what this PR does

Separate dataflow affinity logic into two parts:

fluidapp controller injects the affinity to job' s label;
dataset(data operation) controller use job's label to fill the field NodeAffinity in OperationStatus

The advantages are:

Migrating Pod related functions to the fluid app controller can achieve the ability to downgrade functions, The control of featureGate has also been migrated to the fluid app controller component.
It also reduces the RBAC permission risk of the dataset controller.

Ⅱ. Does this pull request fix one issue?

fixes #XXXX

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

manual testing

Ⅴ. Special notes for reviews

serverless app pod should add fluid.io/managed-by:fluid label to ensure the completion of serverless tasks.
DataBackup uses pod not job, this will be fixed in later pr.

charts/fluid-dataloader/alluxio/templates/job.yaml

TrafalgarZZZ

Sorry for reviewing the code late. I may have a question for the design: what will happen when dataflow controller and dataflow-affinity controller concurrently operates on the given resources? Will there be some concurrency issues?

TrafalgarZZZ · 2024-06-12T12:48:41Z

pkg/controllers/v1alpha1/fluidapp/fluidapp_controller.go

-		Scheme: scheme,
-		SelectorsByObject: cache.SelectorsByObject{
-			&corev1.Pod{}: {Label: labels.SelectorFromSet(labels.Set{
-				common.InjectSidecarDone: common.True,


common.InjectSidecarDone: common.True seems missing in the func NewCache() in current code? Is that by design?

common.InjectSidecarDone is only for fuse app pod, not used for data operation pod. Thecommon.InjectSidecarDone is moved to func ShouldInQueue(pod *corev1.Pod) bool in file ctrl/watch/pod.go.

pkg/utils/crtl_utils.go

pkg/controllers/v1alpha1/fluidapp/dataflowaffinity/dataflowaffinity_controller.go

TrafalgarZZZ · 2024-06-12T13:09:09Z

pkg/controllers/v1alpha1/fluidapp/dataflowaffinity/dataflowaffinity_controller.go

+	}
+	requestCtx.job = job
+
+	if !watch.JobShouldInQueue(job) {


I think we can filter out jobs in eventHandler to simplify the logic here?

Do you means there is no need for !watch.JobShouldInQueue(job) ? It's based on the reconcile logic of Fluid app pod, should both be removed as the eventhandler(ood/job) already uses watch.JobShouldInQueue(job) .

My mistake. I've just found that JobShouldInQueue() is already set in dataflow-affinity controller's event-handler. So here in func Reconcile(), it looks like a double check. LGTM now.

Signed-off-by: xliuqq <xlzq1992@gmail.com>

xliuqq · 2024-06-13T02:28:56Z

Sorry for reviewing the code late. I may have a question for the design: what will happen when dataflow controller and dataflow-affinity controller concurrently operates on the given resources? Will there be some concurrency issues?

If a controller updates a outdated resource, an error like "the object has been modified; please apply your changes to the latest version and try again" will be returned, and the controller will retry, the list-watch mechanism ensures to get the latest resource and the controller can successfully update it eventually.

The dataflow-affinity controller only updates the data operation job in two phases:

inject an annotation to indicates that the data operation uses the dataflow affinity feature;
- this is done when the job created, so it is almost impossible when the job is finished but the annotation is not injected,
inject the labels when job succeed
- data operation controller will find job succeed and has the annotation, if the label is not injected, the data operation controller will retry (see the logic in line 48 in pkg/dataflow/helper.go).

TrafalgarZZZ · 2024-06-13T06:09:38Z

pkg/controllers/v1alpha1/fluidapp/dataflowaffinity/dataflowaffinity_controller.go

+	if _, ok := job.Annotations[common.AnnotationDataFlowAffinityInject]; !ok {
+		job.Annotations[common.AnnotationDataFlowAffinityInject] = "true"
+		if err := f.Client.Update(ctx, job); err != nil {
+			requestCtx.Log.Error(err, "Failed to add dataflow affinity enabled label", "AnnotationUpdateError", ctx)
+			return utils.RequeueIfError(err)
+		}
+	}
+	// get job' status, if succeed, add label to job.
+	condition := kubeclient.GetFinishedJobCondition(job)
+	if condition != nil && condition.Type == batchv1.JobComplete {
+		err = f.injectPodNodeLabelsToJob(job)
+		if err != nil {
+			requestCtx.Log.Error(err, "update labels for job failed")
+			return utils.RequeueIfError(err)
+		}
+	}


How about using f.Client.Patch() to patch annotations to the jobs? Current code calls two consecutive f.Client.Update() (one in line 102 and the other inside f.injectPodNodeLabelsToJob), so in my opinion the latter call will definitely get a conflict error. I think we better avoid this.

Updating label (f.injectPodNodeLabelsToJob) occurs only when job succeed(line 109), so it is not possible to call update twice in once reconciler.

sonarcloud · 2024-06-14T06:17:13Z

Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
7.7% Duplication on New Code

See analysis details on SonarCloud

cheyang · 2024-06-17T07:23:57Z

pkg/dataoperation/operation_type.go

+	DataBackupType  OperationType = "DataBackup"
+	DataMigrateType OperationType = "DataMigrate"
+	DataProcessType OperationType = "DataProcess"
+)


Consider renaming OperationType to simply Operation for brevity. Operation alone conveys the same intention, and it's more concise. The constant values can still retain the necessary 'DataLoad', 'DataBackup', 'DataMigrate', 'DataProcess' identifiers for the clarity of usage context. With this change, your code could look like:

type Operation string const ( Load Operation = "DataLoad" Backup Operation = "DataBackup" Migrate Operation = "DataMigrate" Process Operation = "DataProcess" )

another pr will fix this.

cheyang

/lgtm
/approve

cheyang · 2024-06-21T00:54:29Z

/test fluid-e2e

cheyang

/lgtm
/approve

….io/managed-by two labels Signed-off-by: xliuqq <xlzq1992@gmail.com>

xliuqq · 2024-06-21T16:12:31Z

/test fluid-e2e

cheyang · 2024-06-23T15:29:39Z

@TrafalgarZZZ Please check this PR again. Thanks.

TrafalgarZZZ

/lgtm

fluid-e2e-bot · 2024-06-24T02:18:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheyang, TrafalgarZZZ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [TrafalgarZZZ,cheyang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…e#4138) * use fluidapp to support data flow affinity Signed-off-by: xliuqq <xlzq1992@gmail.com> * make fluidapp dataop controller optional Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix check Signed-off-by: xliuqq <xlzq1992@gmail.com> * add unit test for injectPodNodeLabelsToJob Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix: use library.fluid.labels instead of fluid.io/managed-by Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix comment error and reslove conflicts Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix annotation nil error Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix e2e test, serverless pod use serverless.fluid.io/inject and fluid.io/managed-by two labels Signed-off-by: xliuqq <xlzq1992@gmail.com> --------- Signed-off-by: xliuqq <xlzq1992@gmail.com> Signed-off-by: cheyang <cheyang@163.com>

…50932 (fluid-cloudnative#4154) Signed-off-by: cheyang <cheyang@163.com> Bump up the vineyard to v0.22.2 (fluid-cloudnative#4153) Signed-off-by: Ye Cao <caoye.cao@alibaba-inc.com> Bump github.com/agiledragon/gomonkey/v2 from 2.10.1 to 2.11.0 (fluid-cloudnative#4151) Bumps [github.com/agiledragon/gomonkey/v2](https://github.com/agiledragon/gomonkey) from 2.10.1 to 2.11.0. - [Release notes](https://github.com/agiledragon/gomonkey/releases) - [Commits](agiledragon/gomonkey@v2.10.1...v2.11.0) --- updated-dependencies: - dependency-name: github.com/agiledragon/gomonkey/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> date (fluid-cloudnative#4110) Signed-off-by: Lumen002 <1277683286@qq.com> Add apache license_format in fluid_ufs_scheme_test.go (fluid-cloudnative#4120) * Format Apache License to webhook.go Signed-off-by: mave <211220097@smail.nju.edu.cn> * Your commit message Signed-off-by: mave <211220097@smail.nju.edu.cn> * format Signed-off-by: mave <211220097@smail.nju.edu.cn> * format_test Signed-off-by: mave <211220097@smail.nju.edu.cn> * format apache license Signed-off-by: mave <211220097@smail.nju.edu.cn> --------- Signed-off-by: mave <211220097@smail.nju.edu.cn> 111 (fluid-cloudnative#4121) Signed-off-by: lianwg <1494923285@qq.com> Bump golang.org/x/net from 0.25.0 to 0.26.0 (fluid-cloudnative#4156) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.25.0 to 0.26.0. - [Commits](golang/net@v0.25.0...v0.26.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> modified the incorrect copyright time in file pkg/ddc/efc/cache.go (fluid-cloudnative#4125) Signed-off-by: hadoop <scanocdii@gmail.com> --- (fluid-cloudnative#4123) updated-dependencies: - dependency-name: k8s.io/klog/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Build docker images for vineyard 0.22, To #54250932 (fluid-cloudnative#4159) Signed-off-by: cheyang <cheyang@163.com> Bump sigs.k8s.io/yaml from 1.3.0 to 1.4.0 (fluid-cloudnative#4157) Bumps [sigs.k8s.io/yaml](https://github.com/kubernetes-sigs/yaml) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/kubernetes-sigs/yaml/releases) - [Changelog](https://github.com/kubernetes-sigs/yaml/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/yaml@v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: sigs.k8s.io/yaml dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Fix typo of transformer, To #54583165 (fluid-cloudnative#4150) Signed-off-by: cheyang <cheyang@163.com> Build docker images for fixing transformers, To #54583162 (fluid-cloudnative#4161) Signed-off-by: cheyang <cheyang@163.com> change transfromer to transformers in load_data.go (fluid-cloudnative#4118) * change transfromer to transformers Signed-off-by: yayaaaaaaaaa <1435114933@qq.com> * Update load_data.go --------- Signed-off-by: yayaaaaaaaaa <1435114933@qq.com> Co-authored-by: cheyang <cheyang@163.com> addons: add dynamic mount examples (fluid-cloudnative#4148) * Add base image for fluid dynamic mount feature Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add juicefs examples for fluid dynamic mount feature Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * clean up mount point unconditionally Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Dump mount point logs to /var/log/fluid Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Anchor base image for dynamic mount example Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> change copyright year from 2022 to 2020 (fluid-cloudnative#4132) Signed-off-by: FluorescenceLight <1219804366@qq.com> bugfix: check corrupted mount point in utils.RemoveSymlink (fluid-cloudnative#4163) Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Bump github.com/spf13/cobra from 1.7.0 to 1.8.0 (fluid-cloudnative#4158) Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.7.0 to 1.8.0. - [Release notes](https://github.com/spf13/cobra/releases) - [Commits](spf13/cobra@v1.7.0...v1.8.0) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> modify copyright time (fluid-cloudnative#4128) Signed-off-by: liuyun-0002 <2695993128@qq.com> change time from 2022 to 2020 (fluid-cloudnative#4134) Signed-off-by: chxwindows <2216940643@qq.com> changed the time of Copyright in the operation.go file to 2023 (fluid-cloudnative#4145) Signed-off-by: sishijiang <2524798309@qq.com> Bump go.uber.org/zap from 1.24.0 to 1.27.0 (fluid-cloudnative#3727) Bumps [go.uber.org/zap](https://github.com/uber-go/zap) from 1.24.0 to 1.27.0. - [Release notes](https://github.com/uber-go/zap/releases) - [Changelog](https://github.com/uber-go/zap/blob/master/CHANGELOG.md) - [Commits](uber-go/zap@v1.24.0...v1.27.0) --- updated-dependencies: - dependency-name: go.uber.org/zap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> refactor: rename `CreateUpdatePodForSchedulingHandler` to `FluidMutatingHandler` (fluid-cloudnative#4168) * Refactor fluid mutating handler Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Refactor fluid mutating handler Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Refactor fluid mutating handler Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> minorfix: pin libfuse version in dynamic mount example (fluid-cloudnative#4167) * pin libfuse version to fuse-3.16.2 Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * minorfix: add return of line Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Refactor admission webhook handler registration for better error handling and maintainability, To #57240825 (fluid-cloudnative#4170) Signed-off-by: cheyang <cheyang@163.com> Assign pod.namespace before mutation (fluid-cloudnative#4171) Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> minorfix: carefully handle pod.namespace to avoid side effect (fluid-cloudnative#4172) * carefully handle pod.namespace Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * carefully handle pod.namespace Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * carefully handle pod.namespace Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Build docker images for refactoring webhook, To #57240825 (fluid-cloudnative#4173) Signed-off-by: cheyang <cheyang@163.com> Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 (fluid-cloudnative#4164) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.4 to 1.9.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.8.4...v1.9.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> feat: support multiround sidecar injection (fluid-cloudnative#4175) * Prune unused code Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * refactor: move func CollectRuntimeInfosFromPVCs to webhook utils package Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Support multi-round sidecar injection Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Support multi-round sidecar injection Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * fix unit tests Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * fix unit tests Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add unit tests Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add unit tests Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> description （简单描述你的修改） (fluid-cloudnative#4149) Signed-off-by: bc-ace <2336474595@qq.com> docs: Correcting the usage steps of the CONTABUTING.md document (fluid-cloudnative#4176) Signed-off-by: wjp <wjp_199502@163.com> move dataflow affinity logic to fluidapp controller (fluid-cloudnative#4138) * use fluidapp to support data flow affinity Signed-off-by: xliuqq <xlzq1992@gmail.com> * make fluidapp dataop controller optional Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix check Signed-off-by: xliuqq <xlzq1992@gmail.com> * add unit test for injectPodNodeLabelsToJob Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix: use library.fluid.labels instead of fluid.io/managed-by Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix comment error and reslove conflicts Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix annotation nil error Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix e2e test, serverless pod use serverless.fluid.io/inject and fluid.io/managed-by two labels Signed-off-by: xliuqq <xlzq1992@gmail.com> --------- Signed-off-by: xliuqq <xlzq1992@gmail.com> Build docker images for refactoring dataflow controller, To #57240825 (fluid-cloudnative#4181) Signed-off-by: cheyang <cheyang@163.com> wjl (fluid-cloudnative#4039) Signed-off-by: wjlwjlwjlwjlwjlwjlwjl <2931582547@qq.com> Co-authored-by: cheyang <cheyang@163.com> Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 (fluid-cloudnative#4166) Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.8.0 to 1.8.1. - [Release notes](https://github.com/spf13/cobra/releases) - [Commits](spf13/cobra@v1.8.0...v1.8.1) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Update utils.go (fluid-cloudnative#4116) addcopyright Do not skip mutation for fluid pods (fluid-cloudnative#4184) Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Bump github.com/felixge/fgprof from 0.9.3 to 0.9.4 (fluid-cloudnative#4169) Bumps [github.com/felixge/fgprof](https://github.com/felixge/fgprof) from 0.9.3 to 0.9.4. - [Release notes](https://github.com/felixge/fgprof/releases) - [Commits](felixge/fgprof@v0.9.3...v0.9.4) --- updated-dependencies: - dependency-name: github.com/felixge/fgprof dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> bugfix: fix application controller manage pod lifecycle (fluid-cloudnative#4185) Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Bump github.com/go-logr/logr from 1.4.1 to 1.4.2 (fluid-cloudnative#4174) Bumps [github.com/go-logr/logr](https://github.com/go-logr/logr) from 1.4.1 to 1.4.2. - [Release notes](https://github.com/go-logr/logr/releases) - [Changelog](https://github.com/go-logr/logr/blob/master/CHANGELOG.md) - [Commits](go-logr/logr@v1.4.1...v1.4.2) --- updated-dependencies: - dependency-name: github.com/go-logr/logr dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> cm1 (fluid-cloudnative#4114) dataset add storageSize (fluid-cloudnative#4178) Signed-off-by: wangshulei098 <850732903@qq.com> Build docker images for specifying pvc storage size, To #54250932 (fluid-cloudnative#4187) Signed-off-by: cheyang <cheyang@163.com> docs: dev guide for en missing GOPATH setup (fluid-cloudnative#4188) Signed-off-by: backspace <backspace@backblog.me> Update operation_helm.go (fluid-cloudnative#4129) * Update operation_helm.go modify copyright time * Update operation_helm.go modify copyright time. Signed-off-by: rodestexas <1985056713@qq.com> --------- Signed-off-by: rodestexas <1985056713@qq.com> Bump k8s.io/klog/v2 from 2.120.1 to 2.130.1 (fluid-cloudnative#4183) Bumps [k8s.io/klog/v2](https://github.com/kubernetes/klog) from 2.120.1 to 2.130.1. - [Release notes](https://github.com/kubernetes/klog/releases) - [Changelog](https://github.com/kubernetes/klog/blob/main/RELEASE.md) - [Commits](kubernetes/klog@v2.120.1...v2.130.1) --- updated-dependencies: - dependency-name: k8s.io/klog/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> bugfix: allow users to override juicefs attr-cache and entry-cache options (fluid-cloudnative#4194) * bugfix: allow users override attr-cache and entry-cache in JuiceFS Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * go fmt Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Bump google.golang.org/grpc from 1.64.0 to 1.65.0 (fluid-cloudnative#4191) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.65.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.64.0...v1.65.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Update go.mod to comply with Go 1.21 toolchain syntax (fluid-cloudnative#4196) * Update go mod to 1.21.10, To #57240825 Signed-off-by: cheyang <cheyang@163.com> * Update go mod to 1.21.10, To #57240825 Signed-off-by: cheyang <cheyang@163.com> --------- Signed-off-by: cheyang <cheyang@163.com> Build docker images for fixing juicefs timeout, To #57240825 (fluid-cloudnative#4197) Signed-off-by: cheyang <cheyang@163.com> bugfix: fix early break when cleaning mount point in NodeUnpublishVolume (fluid-cloudnative#4198) * Clean up corrupted mount points in NodeUnpublishVolume Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Clean up corrupted mount points in NodeUnpublishVolume Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> This pr is to add the translation of the Apache License notice (fluid-cloudnative#4078) * Update thin.go Signed-off-by: cheyang <cheyang@163.com> * Update thin.go Signed-off-by: cheyang <cheyang@163.com> * Update thin.go license header, To #54250932 Signed-off-by: cheyang <cheyang@163.com> * Update thin.go license header, To #54250932 Signed-off-by: cheyang <cheyang@163.com> --------- Signed-off-by: cheyang <cheyang@163.com> Co-authored-by: cheyang <cheyang@163.com> Bump github.com/agiledragon/gomonkey/v2 from 2.11.0 to 2.12.0 (fluid-cloudnative#4199) Bumps [github.com/agiledragon/gomonkey/v2](https://github.com/agiledragon/gomonkey) from 2.11.0 to 2.12.0. - [Release notes](https://github.com/agiledragon/gomonkey/releases) - [Commits](agiledragon/gomonkey@v2.11.0...v2.12.0) --- updated-dependencies: - dependency-name: github.com/agiledragon/gomonkey/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> bugfix: fix jindoruntime name cannot contain jindofs (fluid-cloudnative#4202) * bugfix: JindoRuntime cannot use a name containing "jindofs" Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * todo: add todo for handling fullnameOverride Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> revert: "upgrade jindocache to 6.4.0 version (fluid-cloudnative#4126)" and downgrade JindoCache to 6.2.0 (fluid-cloudnative#4205) Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Build docker images for reverting jindoruntime from 6.4 to 6.2, To #57978385 (fluid-cloudnative#4206) Signed-off-by: cheyang <cheyang@163.com> Bump golang.org/x/net from 0.26.0 to 0.27.0 (fluid-cloudnative#4200) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.26.0 to 0.27.0. - [Commits](golang/net@v0.26.0...v0.27.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Prepare 1.0.2, To #57978385 (fluid-cloudnative#4207) Signed-off-by: cheyang <cheyang@163.com> Delete the unused rbac roles for vineyard runtime. (fluid-cloudnative#4208) Signed-off-by: Ye Cao <caoye.cao@alibaba-inc.com> Update docker image to fluid 1.0.2, To #54250932 (fluid-cloudnative#4210) Signed-off-by: cheyang <cheyang@163.com> Bump github.com/golang/glog from 1.2.1 to 1.2.2 (fluid-cloudnative#4204) Bumps [github.com/golang/glog](https://github.com/golang/glog) from 1.2.1 to 1.2.2. - [Release notes](https://github.com/golang/glog/releases) - [Commits](golang/glog@v1.2.1...v1.2.2) --- updated-dependencies: - dependency-name: github.com/golang/glog dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> bugfix: support sync ak/sk secret key-values in JindoRuntime (fluid-cloudnative#4212) * bugfix: support sync ak/sk secret key-values in JindoRuntime Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Update CHANGELOG.md Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> del create pod role of juicefs runtime (fluid-cloudnative#4215) Signed-off-by: zwwhdls <zww@hdls.me> refactor: separate image and tag in helm values (fluid-cloudnative#4216) * Separate images in values.yaml into repo, image and tag Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Separate images in values.yaml into imagePrefix, imageName and imageTag Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * refactor: transform control plane images with helper function Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * refactor: transform runtime images with helper function Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Refactor fluid helm chart, To #54583162 (fluid-cloudnative#4218) Signed-off-by: cheyang <cheyang@163.com> Add no cache option to Makefile, To #54583162 (fluid-cloudnative#4217) Signed-off-by: cheyang <cheyang@163.com> update scorecard link (fluid-cloudnative#4220) Signed-off-by: fsl <1171313930@qq.com> feat: pass extra args when building fuse sidecar mutators (fluid-cloudnative#4223) * Rename `MutatorBuildOpts` to `MutatorBuildArgs` Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Pass extraArgs when building platform-specific mutators Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add unit tests Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add unit tests Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> enhance: add github actions workflow e2e tests (fluid-cloudnative#4224) * Add integration.yml Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add github actions e2e for alluxio Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add testcase for juicefsruntime & jindoruntime Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Gracefully clean up resources Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Dump e2e environment info with diagnose scripts and upload it to github action artifacts Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Rename github actions workflow names Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Purge out-of-date testcases Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> minorfix: set read-all permission for e2e functionality check workflow (fluid-cloudnative#4226) * Minorfix: set read-all permission for e2e functionality check workflow Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Add empty line EOF Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> bugfix: fix kind e2e test images (fluid-cloudnative#4232) * Fix version replace in gha-e2e Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Optimize diagnose shell scripts Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> fix Variable Naming Convention for DEFAULT_MASTER_RPC_PORT in CamelCase (fluid-cloudnative#4234) * Change Constant Variable Naming Convention for DEFAULT_MASTER_RPC_PORT in camelCase format defined in pkg/ddc/jindo/const.go and its reference in /pkg/ddc/jindo/transform.go Signed-off-by: SouthWest7 <1403572259@qq.com> * Change Constant Variable Naming Convention for DEFAULT_MASTER_RPC_PORT in camelCase format defined in pkg/ddc/jindo/const.go and its reference in /pkg/ddc/jindo/transform.go Signed-off-by: SouthWest7 <1403572259@qq.com> * Change Constant Variable Naming Convention for DEFAULT_MASTER_RPC_PORT in camelCase format defined in pkg/ddc/jindo/const.go and its reference in /pkg/ddc/jindo/transform.go Signed-off-by: SouthWest7 <1403572259@qq.com> * Change Constant Variable Naming Convention for DEFAULT_MASTER_RPC_PORT in camelCase format defined in pkg/ddc/jindo/const.go and its reference in /pkg/ddc/jindo/transform.go Signed-off-by: SouthWest7 <1403572259@qq.com> --------- Signed-off-by: SouthWest7 <1403572259@qq.com> Change Constant Variable Naming Convention for CSI_DRIVER in camelCase format (fluid-cloudnative#4229) * Update const.go * Update const.go bugfix: fix version replace in gha-e2e (fluid-cloudnative#4235) * Fix version replace in gha-e2e Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> * Fix version replace in gha-e2e Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> --------- Signed-off-by: trafalgarzzz <trafalgarz@outlook.com> Update docker images for refactoring sidecar mutator (fluid-cloudnative#4230) * Update docker images for refactoring sidecar mutator, To #54583162 Signed-off-by: cheyang <cheyang@163.com> * Update docker images for refactoring sidecar mutator, To #54583162 Signed-off-by: cheyang <cheyang@163.com> * Update docker images for refactoring sidecar mutator, To #54583162 Signed-off-by: cheyang <cheyang@163.com> --------- Signed-off-by: cheyang <cheyang@163.com> juicefs: use exec in juicefs pod to avoid ignore signal (fluid-cloudnative#4236) * feat: use exec in juicefs pod to avoid ignore signal Signed-off-by: zwwhdls <zww@hdls.me> * feat: use exec in juicefs pod to avoid ignore signal Signed-off-by: zwwhdls <zww@hdls.me> * fix unittest Signed-off-by: zwwhdls <zww@hdls.me> --------- Signed-off-by: zwwhdls <zww@hdls.me> update juicefs default imaghe (fluid-cloudnative#4237) Signed-off-by: zwwhdls <zww@hdls.me> Bump ossf/scorecard-action from 2.3.3 to 2.4.0 (fluid-cloudnative#4239) Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.3 to 2.4.0. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@dc50aa9...62b2cac) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Update controller-runtime to v1.17.5, To #57240825 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #55315018 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Update controller-runtime to v1.17.5, To #57240825 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Update controller-runtime to v1.17.5, To #57240825 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Refactoring the client cache mode, To #54583162 Signed-off-by: cheyang <cheyang@163.com> Fix compiling issue, To #57978385 Signed-off-by: cheyang <cheyang@163.com>

xliuqq force-pushed the affinity_app branch from 2b40e54 to e9de674 Compare May 28, 2024 06:09

cheyang requested a review from zwwhdls May 28, 2024 11:50

TrafalgarZZZ reviewed Jun 5, 2024

View reviewed changes

charts/fluid-dataloader/alluxio/templates/job.yaml Outdated Show resolved Hide resolved

TrafalgarZZZ requested changes Jun 12, 2024

View reviewed changes

xliuqq added 6 commits June 13, 2024 09:22

use fluidapp to support data flow affinity

833a657

Signed-off-by: xliuqq <xlzq1992@gmail.com>

make fluidapp dataop controller optional

645ad60

Signed-off-by: xliuqq <xlzq1992@gmail.com>

fix check

9f5921d

Signed-off-by: xliuqq <xlzq1992@gmail.com>

add unit test for injectPodNodeLabelsToJob

74c86f8

Signed-off-by: xliuqq <xlzq1992@gmail.com>

fix: use library.fluid.labels instead of fluid.io/managed-by

3ba1a07

Signed-off-by: xliuqq <xlzq1992@gmail.com>

fix comment error and reslove conflicts

e415bc1

Signed-off-by: xliuqq <xlzq1992@gmail.com>

xliuqq force-pushed the affinity_app branch from 9b4d719 to e415bc1 Compare June 13, 2024 01:29

fix annotation nil error

23185b4

Signed-off-by: xliuqq <xlzq1992@gmail.com>

TrafalgarZZZ reviewed Jun 13, 2024

View reviewed changes

Merge branch 'master' into affinity_app

bd58e9e

cheyang reviewed Jun 17, 2024

View reviewed changes

cheyang requested a review from TrafalgarZZZ June 18, 2024 02:55

cheyang approved these changes Jun 20, 2024

View reviewed changes

fluid-e2e-bot bot assigned cheyang Jun 20, 2024

fluid-e2e-bot bot added lgtm approved labels Jun 20, 2024

cheyang self-requested a review June 20, 2024 09:43

Merge branch 'master' into affinity_app

c0d48fb

fluid-e2e-bot bot removed the lgtm label Jun 20, 2024

cheyang approved these changes Jun 21, 2024

View reviewed changes

fluid-e2e-bot bot added the lgtm label Jun 21, 2024

fix e2e test, serverless pod use serverless.fluid.io/inject and fluid…

d5dca02

….io/managed-by two labels Signed-off-by: xliuqq <xlzq1992@gmail.com>

fluid-e2e-bot bot removed the lgtm label Jun 21, 2024

cheyang removed request for TrafalgarZZZ and zwwhdls June 22, 2024 00:16

TrafalgarZZZ approved these changes Jun 24, 2024

View reviewed changes

fluid-e2e-bot bot assigned TrafalgarZZZ Jun 24, 2024

fluid-e2e-bot bot added the lgtm label Jun 24, 2024

fluid-e2e-bot bot merged commit 620babc into fluid-cloudnative:master Jun 24, 2024
8 checks passed

xliuqq deleted the affinity_app branch June 27, 2024 03:32

TrafalgarZZZ mentioned this pull request Jun 28, 2024

bugfix: fix application controller manage pod lifecycle #4185

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

move dataflow affinity logic to fluidapp controller #4138

move dataflow affinity logic to fluidapp controller #4138

xliuqq commented May 28, 2024 •

edited

Loading

TrafalgarZZZ left a comment

TrafalgarZZZ Jun 12, 2024

xliuqq Jun 13, 2024

TrafalgarZZZ Jun 12, 2024

xliuqq Jun 13, 2024

TrafalgarZZZ Jun 13, 2024

xliuqq commented Jun 13, 2024

TrafalgarZZZ Jun 13, 2024

xliuqq Jun 14, 2024

sonarcloud bot commented Jun 14, 2024

cheyang Jun 17, 2024

xliuqq Jun 20, 2024

cheyang left a comment

cheyang commented Jun 21, 2024

cheyang left a comment

xliuqq commented Jun 21, 2024

cheyang commented Jun 23, 2024

TrafalgarZZZ left a comment

fluid-e2e-bot bot commented Jun 24, 2024

move dataflow affinity logic to fluidapp controller #4138

move dataflow affinity logic to fluidapp controller #4138

Conversation

xliuqq commented May 28, 2024 • edited Loading

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

TrafalgarZZZ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xliuqq commented Jun 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Jun 14, 2024

Quality Gate passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheyang left a comment

Choose a reason for hiding this comment

cheyang commented Jun 21, 2024

cheyang left a comment

Choose a reason for hiding this comment

xliuqq commented Jun 21, 2024

cheyang commented Jun 23, 2024

TrafalgarZZZ left a comment

Choose a reason for hiding this comment

fluid-e2e-bot bot commented Jun 24, 2024

xliuqq commented May 28, 2024 •

edited

Loading