configmap item with order #5308

charles-chenzz · 2023-09-02T03:18:06Z

this PR tend to solve #4292
since #5210 remain stagnant, so I took it up.
Also, I borrow some of the code in #5210

…ey order

k8s-ci-robot · 2023-09-02T03:18:08Z

This PR has multiple commits, and the default merge method is: merge.
You can request commits to be squashed using the label: tide/merge-method-squash

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2023-09-02T03:18:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: charles-chenzz
Once this PR has been reviewed and has the lgtm label, please assign knverey for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…eive changes in interface Generate

…n interface Generate

kustomize/command/edit api/testutil/kusttest plugin/example.someteam.com receive changes arg in the function

charles-chenzz · 2023-09-05T14:23:30Z

the lint checks failed, but wasn't about the mismatch of args. have no clues why it would failed. would defer till I fix the Test Linux and Test MacOS CI jobs.

Error: api/resmap/factory.go:105:2: Consider pre-allocating `resources` (prealloc)
	var (
	^
Error: api/internal/generators/configmap.go:41:20: error returned from external package is unwrapped: sig: func (*sigs.k8s.io/kustomize/kyaml/yaml.RNode).LoadMapIntoConfigMapData(m map[string]string) error (wrapcheck)
		return nil, nil, err
		                 ^
Error: api/internal/generators/secret.go:47:20: error returned from external package is unwrapped: sig: func (*sigs.k8s.io/kustomize/kyaml/yaml.RNode).Pipe(functions ...sigs.k8s.io/kustomize/kyaml/yaml.Filter) (*sigs.k8s.io/kustomize/kyaml/yaml.RNode, error) (wrapcheck)
		return nil, nil, err
		                 ^
Error: api/internal/generators/secret.go:54:20: error returned from external package is unwrapped: sig: func (*sigs.k8s.io/kustomize/kyaml/yaml.RNode).LoadMapIntoSecretData(m map[string]string) error (wrapcheck)
		return nil, nil, err
		                 ^
Error: api/internal/generators/utils.go:50:21: error returned from interface method should be wrapped: sig: func (sigs.k8s.io/kustomize/api/ifc.Validator).ErrIfInvalidKey(string) error (wrapcheck)
			return nil, nil, err
			                 ^
Error: api/resource/factory.go:281:20: error returned from external package is unwrapped: sig: func sigs.k8s.io/kustomize/api/internal/generators.MakeConfigMap(ldr sigs.k8s.io/kustomize/api/ifc.KvLoader, args *sigs.k8s.io/kustomize/api/types.ConfigMapArgs) (rn *sigs.k8s.io/kustomize/kyaml/yaml.RNode, orderKeys []string, err error) (wrapcheck)
		return nil, nil, err
		                 ^
Error: api/resource/factory.go:290:20: error returned from external package is unwrapped: sig: func sigs.k8s.io/kustomize/api/internal/generators.MakeSecret(ldr sigs.k8s.io/kustomize/api/ifc.KvLoader, args *sigs.k8s.io/kustomize/api/types.SecretArgs) (rn *sigs.k8s.io/kustomize/kyaml/yaml.RNode, orderKeys []string, err error) (wrapcheck)
		return nil, nil, err
		                 ^
Error: api/resmap/factory.go:96:20: error returned from external package is unwrapped: sig: func (*sigs.k8s.io/kustomize/api/resource.Factory).MakeConfigMap(kvLdr sigs.k8s.io/kustomize/api/ifc.KvLoader, args *sigs.k8s.io/kustomize/api/types.ConfigMapArgs) (*sigs.k8s.io/kustomize/api/resource.Resource, []string, error) (wrapcheck)
		return nil, nil, err
		                 ^
Error: api/resmap/factory.go:127:20: error returned from external package is unwrapped: sig: func (*sigs.k8s.io/kustomize/api/resource.Factory).MakeSecret(kvLdr sigs.k8s.io/kustomize/api/ifc.KvLoader, args *sigs.k8s.io/kustomize/api/types.SecretArgs) (*sigs.k8s.io/kustomize/api/resource.Resource, []string, error) (wrapcheck)
		return nil, nil, err
		                 ^
make[1]: *** [../Makefile-modules.mk:18: lint] Error 1
make[1]: Leaving directory '/home/runner/work/kustomize/kustomize/api'
make: *** [Makefile:127: lint] Error 2
Error: Process completed with exit code 2.

natasha41575 · 2023-09-05T14:53:53Z

error returned from external package is unwrapped means you need to wrap the error, like

return nil, nil, fmt.Errorf("failed to xxxx: %w", err)

charles-chenzz · 2023-09-08T15:01:35Z

I think I might need some here. the CI jobs failed messages showing that it's about assignment values mismatch 3 but g.generate() return 2 values, while I execute the same test function in my local, I got diff error. will it be embed issue? or am I missing something here

in my local:

in CI jobs:

charles-chenzz · 2023-09-08T15:02:26Z

I convert to ready for review for gathering some helps and reviews

charles-chenzz · 2023-09-08T15:02:50Z

/cc @stormqueen1990

…rface Generate

k8s-ci-robot · 2023-09-09T05:33:53Z

@charles-chenzz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
kustomize-presubmit-master	`5675ab9`	link	true	`/test kustomize-presubmit-master`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

stormqueen1990

Hi there, @charles-chenzz! 👋

I left some suggestions. I can also reproduce the error you're seeing about mismatched arguments locally after changing the signature of Generate in converter_test.go, so I'm assuming the change in the method signature might require more steps to pass tests. Perhaps @natasha41575 or @annasong20 can give us some help to understand the issue?

stormqueen1990 · 2023-09-10T16:38:31Z

api/internal/generators/configmap.go

 	}
 	if err = rn.LoadMapIntoConfigMapData(m); err != nil {
-		return nil, err
+		return nil, nil, fmt.Errorf("failed to LoadMapIntoConfigMapData, error:%w", err)


Minor suggestion: add a space between the colon and %w for readability

Suggested change

return nil, nil, fmt.Errorf("failed to LoadMapIntoConfigMapData, error:%w", err)

return nil, nil, fmt.Errorf("failed to LoadMapIntoConfigMapData, error: %w", err)

stormqueen1990 · 2023-09-10T16:39:47Z

api/internal/generators/configmap_test.go

@@ -206,7 +206,7 @@ immutable: true
 	for n := range testCases {
 		tc := testCases[n]
 		t.Run(n, func(t *testing.T) {
-			rn, err := MakeConfigMap(kvLdr, &tc.args)
+			rn, _, err := MakeConfigMap(kvLdr, &tc.args)


I think this test should be updated to validate the orderKeys return value.

stormqueen1990 · 2023-09-10T16:41:29Z

api/internal/generators/secret.go

Same suggestion to add a space between : and %w in this file.

stormqueen1990 · 2023-09-10T16:41:59Z

api/internal/generators/secret_test.go

@@ -214,7 +214,7 @@ immutable: true
 	for n := range testCases {
 		tc := testCases[n]
 		t.Run(n, func(t *testing.T) {
-			rn, err := MakeSecret(kvLdr, &tc.args)
+			rn, _, err := MakeSecret(kvLdr, &tc.args)


Same suggestion as in configmap_test.go here.

stormqueen1990 · 2023-09-10T16:46:01Z

api/internal/target/kusttarget.go

+		subRa, _, err = subKt.accumulateTarget(ra)
 		ra = accumulator.MakeEmptyAccumulator()
 	} else {
 		// Child Kustomizations create a new accumulator which resolves their kustomization directives, which will later
 		// be merged into the current accumulator.
-		subRa, err = subKt.AccumulateTarget()
+		subRa, _, err = subKt.AccumulateTarget()


Was discarding the new output here the goal?

stormqueen1990 · 2023-09-10T16:46:31Z

api/internal/target/kusttarget_test.go

I also think these tests should validate the new output in some way.

stormqueen1990 · 2023-09-10T16:50:01Z

api/internal/generators/utils.go

 				"configmap %s illegally repeats the key `%s`", name, p.Key)
 		}
 		knownKeys[p.Key] = p.Value
+		orderKeys = append(orderKeys, p.Key)


I'm a little confused by this part. Where is the ordering happening?

was trying to using a slice of string to preserve the order when for range the pairs, then return it. I was based on the issue and the former PR as my context about this feature. but what I commit in this PR might have details that wrong or misunderstood.

I think there might be some extra steps needed. Just adding the keys to a slice will likely not be enough to guarantee a predictable order. Another thing I noticed while investigating this code is that the output from this method is discarded, so what ends up being written to the file is exactly the user input:

kustomize/kustomize/commands/edit/add/configmap.go

Lines 114 to 116 in 7c36ed2

args.Options = types.MergeGlobalOptionsIntoLocal(

args.Options, k.GeneratorOptions)

_, err := rf.MakeConfigMap(ldr, args)

natasha41575 · 2023-09-15T18:34:33Z

api/resmap/resmap.go

@@ -30,7 +30,7 @@ type TransformerWithProperties struct {

 // A Generator creates an instance of ResMap.
 type Generator interface {
-	Generate() (ResMap, error)
+	Generate() (ResMap, []string, error)


So unfortunately, this is a public interface and we can't change the signature (this is how users built legacy style extensions and the new KRM plugin extensions and we can't break that).

I think it also isn't quite sensical for this Generator to return an array of keys just for the ConfigMap issue. That type of data is specific to the ConfigMap generator and is not generalizable to all generators, so should instead be attached to the ConfigMapGenerator and methods - and only to things that are specific to the ConfigMapGenerator.

A note on the issue is that maintaining the original order of ConfigMaps is not strictly needed. We can also output the ConfigMap items in alphabetical order. As long as the output is deterministic, we can consider the issue as resolved. If there isn't a good way to do this without modifying the generator struct, then we will have to mark the issue as not feasible and explain why.

I would think about other solution to see if we change limit the changes only to configmapgenerator

k8s-ci-robot · 2023-11-02T12:53:21Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2024-01-31T13:37:10Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-03-01T13:48:18Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-03-31T14:10:04Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2024-03-31T14:10:08Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

charles-chenzz added 3 commits September 2, 2023 11:10

api/internal/generators: add a list of string to preserve configmap k…

116b520

…ey order

api/resource/factory.go: receive orderKeys in Factory's method

e6a02a1

api/resmap/factory.go: receive orderKeys in Factory's method

6b22bd7

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 2, 2023

k8s-ci-robot requested review from annasong20 and yuwenma September 2, 2023 03:18

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 2, 2023

charles-chenzz mentioned this pull request Sep 2, 2023

Config map key order #5210

Closed

charles-chenzz marked this pull request as draft September 2, 2023 03:25

charles-chenzz added 3 commits September 5, 2023 20:55

api/internal/builtins: use plugin/builtin to re-generate files to rec…

3093f19

…eive changes in interface Generate

cmd/pluginator/internal/krmfunction/funcwrappersrc: receive changes i…

b5f1511

…n interface Generate

api/internal/target

e0fc92c

kustomize/command/edit api/testutil/kusttest plugin/example.someteam.com receive changes arg in the function

charles-chenzz force-pushed the configmap_order branch 2 times, most recently from 62f17ca to 92f1f86 Compare September 8, 2023 14:07

charles-chenzz marked this pull request as ready for review September 8, 2023 15:02

k8s-ci-robot requested a review from natasha41575 September 8, 2023 15:02

k8s-ci-robot requested a review from stormqueen1990 September 8, 2023 15:02

charles-chenzz force-pushed the configmap_order branch 2 times, most recently from ac9b4ac to 73e762a Compare September 9, 2023 02:26

charles-chenzz changed the title ~~WIP: configmap item with order~~ configmap item with order Sep 9, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2023

charles-chenzz force-pushed the configmap_order branch from 73e762a to 206afc7 Compare September 9, 2023 05:07

charles-chenzz force-pushed the configmap_order branch from 206afc7 to ce9f793 Compare September 9, 2023 05:20

api/: wrap error returned to external package,receive changes in inte…

5675ab9

…rface Generate

charles-chenzz force-pushed the configmap_order branch from ce9f793 to 5675ab9 Compare September 9, 2023 05:28

stormqueen1990 reviewed Sep 10, 2023

View reviewed changes

natasha41575 reviewed Sep 15, 2023

View reviewed changes

charles-chenzz mentioned this pull request Sep 20, 2023

configmap with alphabetical order #5334

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 2, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 31, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 1, 2024

k8s-ci-robot closed this Mar 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

configmap item with order #5308

configmap item with order #5308

charles-chenzz commented Sep 2, 2023

k8s-ci-robot commented Sep 2, 2023

k8s-ci-robot commented Sep 2, 2023

charles-chenzz commented Sep 5, 2023

natasha41575 commented Sep 5, 2023

charles-chenzz commented Sep 8, 2023 •

edited

Loading

charles-chenzz commented Sep 8, 2023

charles-chenzz commented Sep 8, 2023

k8s-ci-robot commented Sep 9, 2023

stormqueen1990 left a comment

stormqueen1990 Sep 10, 2023

stormqueen1990 Sep 10, 2023

stormqueen1990 Sep 10, 2023

stormqueen1990 Sep 10, 2023

stormqueen1990 Sep 10, 2023

stormqueen1990 Sep 10, 2023

stormqueen1990 Sep 10, 2023

charles-chenzz Sep 11, 2023

stormqueen1990 Sep 12, 2023

natasha41575 Sep 15, 2023 •

edited

Loading

charles-chenzz Sep 16, 2023

k8s-ci-robot commented Nov 2, 2023

k8s-triage-robot commented Jan 31, 2024

k8s-triage-robot commented Mar 1, 2024

k8s-triage-robot commented Mar 31, 2024

k8s-ci-robot commented Mar 31, 2024

	return nil, nil, fmt.Errorf("failed to LoadMapIntoConfigMapData, error:%w", err)
	return nil, nil, fmt.Errorf("failed to LoadMapIntoConfigMapData, error: %w", err)

	args.Options = types.MergeGlobalOptionsIntoLocal(
	args.Options, k.GeneratorOptions)
	_, err := rf.MakeConfigMap(ldr, args)

configmap item with order #5308

configmap item with order #5308

Conversation

charles-chenzz commented Sep 2, 2023

k8s-ci-robot commented Sep 2, 2023

k8s-ci-robot commented Sep 2, 2023

charles-chenzz commented Sep 5, 2023

natasha41575 commented Sep 5, 2023

charles-chenzz commented Sep 8, 2023 • edited Loading

charles-chenzz commented Sep 8, 2023

charles-chenzz commented Sep 8, 2023

k8s-ci-robot commented Sep 9, 2023

stormqueen1990 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natasha41575 Sep 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 2, 2023

k8s-triage-robot commented Jan 31, 2024

k8s-triage-robot commented Mar 1, 2024

k8s-triage-robot commented Mar 31, 2024

k8s-ci-robot commented Mar 31, 2024

charles-chenzz commented Sep 8, 2023 •

edited

Loading

natasha41575 Sep 15, 2023 •

edited

Loading