add options to disable unique keys

[xyz-li]

Upgrading to go-yaml.v3 causes a kustomize returns as error when there are duplicate keys in resources.
I think it will be better to provide an option to disable uniqueKeys.

go-yaml/yaml#751
#3480

[k8s-ci-robot]

@xyz-li: This PR has multiple commits, and the default merge method is: merge.
You can request commits to be squashed using the label: tide/merge-method-squash

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Welcome @xyz-li!

It looks like this is your first PR to kubernetes-sigs/kustomize 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kustomize has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @xyz-li. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xyz-li]

/assign @natasha41575

[mikebz]

I am curious if we can make this PR against goyaml first. We don't want to diverge too much from upstream.

[natasha41575]

I am curious if we can make this PR against goyaml first. We don't want to diverge too much from upstream.

+1, we want to diverge as little as possible. We don't want to run into a situation where we take this change in kustomize and the go-yaml maintainer decides against accepting the change upstream.

I'm also hesitant to expose a new flag for kustomize build, as that introduces build time side effects. Rather than make it configurable, it might be better to firmly take a stance on whether or not we allow unique keys.

I see you've already made a PR upstream go-yaml/yaml#752. Thanks for that, we will have to wait to see what the go-yaml maintainer says.

[natasha41575]

/hold

until go-yaml/yaml#751 is closed.

[k8s-ci-robot]

@xyz-li: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
kustomize-presubmit-master	60624f8	link	/test kustomize-presubmit-master

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xyz-li
To complete the pull request process, please ask for approval from natasha41575 after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[KnVerey]

I agree with @natasha41575 that we cannot accept any changes to the internal fork that have not been agreed to upstream, and even then we should be extremely conservative about merging such changes in vs. waiting to remove the fork in favour of a new release that includes them. FWIW go-yaml.v3 is a yaml 1.2 implementation, and the maintainer has correctly pointed out that duplicate keys are not valid per that spec.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xyz-li]

/close

[k8s-ci-robot]

@xyz-li: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[davisford]

@natasha41575 @KnVerey would you reconsider your position. The PR by @xyz-li has been sitting there since June with no comment or review. Currently go-yaml has 69 open PRs and hundreds of open issues, and it doesn't appear like it is being very actively maintained, which is concerning.

I have a scenario I am facing with ingress-nginx where it is failing b/c of the dupe keys in my deployment, and I have no valid workaround.

[natasha41575]

Even if we were willing to diverge from the go-yaml library - which, if we do, I think we would want to make an official fork under kubernetes and not have it in our internal code - I don't think we want to have an option to disable unique keys. Duplicate keys are not valid YAML so throwing an error seems like correct behavior to me. If the spec requires duplicate keys, that sounds like a problem with the spec.

[natasha41575]

Also the error you've cited in the other issue:

Kustomization reconciliation failed: Deployment/ingress-nginx/ingress-nginx-controller dry-run failed, error: failed to create manager for existing fields: failed to convert new object (apps/v1, Kind=Deployment) to smd typed: .spec.template.spec.containers[name="controller"].ports: duplicate entries for key [containerPort=80,protocol="TCP"]

Are you sure that error is coming from kustomize? It sounds more like an error that kubernetes or kubectl is throwing, because the kubernetes specification defines the key for a port entry to be [containerPort, protocol], and expects this to be unique. This is not the same thing as a YAML key and this PR won't fix your issue. If you just run kustomize build on your input, do you get the same error? If not, then this isn't anything we can fix on the kustomize side even if we wanted to.

[MeNsaaH]

@natasha41575 do you think it's okay to port this changes to https://github.com/kubernetes-sigs/yaml/blob/master/goyaml.v3/yaml.go?

This is still a very much needed change and I'll be happy to take continue from where @xyz-li has started

[MeNsaaH]

@natasha41575 I've opened a follow PR for this on go-yaml. I'll create the next PR to implement this once that is merged.