Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Policy Generator is splitting go template functions in yaml scalar strings #55

Closed
fgharo opened this issue Jun 22, 2022 · 6 comments
Closed

Policy Generator is splitting go template functions in yaml scalar strings #55

fgharo opened this issue Jun 22, 2022 · 6 comments

Comments

@fgharo
Copy link
Contributor

fgharo commented Jun 22, 2022 •
edited

Version(s):
PolicyGenerator - v1.7.0
ACM - 2.4.3

Description:
The policy generator splits a go template function located in the same input manifest into multiple lines, creating an invalid yaml strings as output. Subsequently, this output can't be pasted into the ACM console -> Governance -> Create Policy -> editor. The editor has a red squiggly line and when I hover over it, it says:

Malformed inline yaml string ('{{ fromConfigMap "default" "logs-config" #57).

Note: The output can still be fed to oc apply -f - and the go template function can still be evaluated so this is a workaround.

Steps to reproduce:

  1. Create a folder with the following manifests:
    namespace.yaml:
apiVersion: v1
kind: Namespace
metadata:
  name: demo-of-string-formatting-problem
  annotations:
      region: '{{hub (lookup "cluster.open-cluster-management.io/v1" "ManagedCluster" "default" .ManagedClusterName).metadata.labels.region hub}}'

policy-generator-config.yaml:

apiVersion: policy.open-cluster-management.io/v1
kind: PolicyGenerator
metadata:
  name: acm-policy-generator
policyDefaults:
  namespace: default
  placement:
    clusterSelectors:
        name: "local-cluster"
policies:
  - name: demo-of-string-formatting-problem
    manifests:
      - path: namespace.yaml
        complianceType: "musthave"
    severity: high
    remediationAction: enforce

kustomization.yaml:

generators:
  - policy-generator-config.yaml

  1. Run policy generator with kustomize in the folder with the yaml files.
    $ kustomize build --enable-alpha-plugins

  2. The result contains the line breaks like so:

...
            apiVersion: v1
            kind: Namespace
            metadata:
              annotations:
                region: '{{hub (lookup "cluster.open-cluster-management.io/v1"
                  "ManagedCluster" "default" .ManagedClusterName).metadata.labels.region
                  hub}}'
              name: demo-of-string-formatting-problem
...

Expected Result:

...
            apiVersion: v1
            kind: Namespace
            metadata:
              annotations:
                region: '{{hub (lookup "cluster.open-cluster-management.io/v1" "ManagedCluster" "default" .ManagedClusterName).metadata.labels.region hub}}'
              name: demo-of-string-formatting-problem
...

Extra notes:

  1. Not sure if this is a further bug, but breaks also happen with other yaml scalar strings and some yaml scalar string symbols (|- |+ > >- >+) are not respected. According to this website https://yaml-multiline.info/ there are 9 possible yaml multiline strings.

For instance when this configmap gets fed into the policy generator it is translated into the subsequent configmap:

before:

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-of-string-formatting-problem
  namespace: default
data:
  case1_blockScalar_literalClip: |
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case2_blockScalar_literalStrip: |-
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case3_blockScalar_literalKeep: |+
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case4_blockScalar_foldedClip: >
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case5_blockScalar_foldedStrip: >-
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case6_blockScalar_foldedKeep: >+
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case7_flowScalar_doubleQuotes: "{{ fromConfigMap \"default\" \"logs-config\" \"log-file\" }}"
  case8_flowScalar_singleQuotes: '{{ fromConfigMap "default" "logs-config" "log-file" }}'
  # Note the following case has to have a plain text prefix otherwise the following kustomize error will pop up. I chose prefix "the-configmap-is-"
  # error generating policies from the PolicyGenerator file '/var/folders/mp/dlbrhpwj323b7r_sxdsfvnywkspdg2/T/kust-plugin-config-1123374341': failed to decode the manifest file at cm.yaml: yaml: invalid map key: map[string]interface {}{"hub (lookup \"cluster.open-cluster-management.io/v1\" \"ManagedCluster\" \"default\" .ManagedClusterName).metadata.labels.region hub":interface {}(nil)}
  # Error: failure in plugin configured via /var/folders/mp/dlbrhpwj323b7r_sxdsfvnywkspdg2/T/kust-plugin-config-1123374341; exit status 1: exit status 1
  case9_flowScalar_plain: the-configmap-is-{{ fromConfigMap "default" "logs-config" "log-file" }}

after:

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-of-string-formatting-problem
  namespace: default
data:
  case1_blockScalar_literalClip: |
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case2_blockScalar_literalStrip: '{{ fromConfigMap "default" "logs-config"
    "log-file" }}'
  case3_blockScalar_literalKeep: |
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case4_blockScalar_foldedClip: |
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case5_blockScalar_foldedStrip: '{{ fromConfigMap "default" "logs-config"
    "log-file" }}'
  case6_blockScalar_foldedKeep: |
    {{ fromConfigMap "default" "logs-config" "log-file" }}
  case7_flowScalar_doubleQuotes: '{{ fromConfigMap "default" "logs-config"
    "log-file" }}'
  case8_flowScalar_singleQuotes: '{{ fromConfigMap "default" "logs-config"
    "log-file" }}'
  case9_flowScalar_plain: the-configmap-is-{{ fromConfigMap "default"
    "logs-config" "log-file" }}

For simplicity sake it might be better to focus on the line breaks that are created. And delegate the problem of scalar symbols not being respected to another issue. I would of expected
them to appear in the output as opposed to be replaced by | in some cases.

  1. When the scalar strings are english, the line breaks do not appear. I suspect then that the problem is interrelated with the syntax symbols of a go template function. Nevertheless some of the yaml scalar string symbols (|- |+ > >- >+) are still not respected.

For instance this configmap doesn't get any line breaks:
before:

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-of-string-formatting-problem
  namespace: default
data:
  case1_blockScalar_literalClip: |
    hello from harry potter!
  case2_blockScalar_literalStrip: |-
    hello from harry potter!
  case3_blockScalar_literalKeep: |+
    hello from harry potter!
  case4_blockScalar_foldedClip: >
    hello from harry potter!
  case5_blockScalar_foldedStrip: >-
    hello from harry potter!
  case6_blockScalar_foldedKeep: >+
    hello from harry potter!
  case7_flowScalar_doubleQuotes: "hello from harry potter!"
  case8_flowScalar_singleQuotes: 'hello from harry potter!'
  case9_flowScalar_plain: hello from harry potter!

after:

kind: ConfigMap
apiVersion: v1
data:
  case1_blockScalar_literalClip: |
    hello from harry potter!
  case2_blockScalar_literalStrip: hello from harry potter!
  case3_blockScalar_literalKeep: |
    hello from harry potter!
  case4_blockScalar_foldedClip: |
    hello from harry potter!
  case5_blockScalar_foldedStrip: hello from harry potter!
  case6_blockScalar_foldedKeep: |
    hello from harry potter!
  case7_flowScalar_doubleQuotes: hello from harry potter!
  case8_flowScalar_singleQuotes: hello from harry potter!
  case9_flowScalar_plain: hello from harry potter!

Screenshots:
Screen Shot 2022-06-22 at 6 50 44 PM
@mprahl
Copy link
Member

mprahl commented Jun 23, 2022 •
edited

@fgharo thank you for reporting this!

The issue is not related to the Policy Generator but is a bug with Kustomize:
kubernetes-sigs/kustomize#947

Are you able to make it a multi-line string with | like you showed above?

If you can't use a multi-line string, you aren't using any of the Kustomize features, and you aren't executing Kustomize as part of the GitHub workflows, you can execute the Policy Generator binary directly with something like:

/path/to/PolicyGenerator ./policy-generator-config.yaml

This will provide the generated policies without the line breaks.

@fgharo
Copy link
Contributor Author

fgharo commented Jun 23, 2022

@mprahl
Thanks. We are using Kustomize. As I said the oc apply still works and our templates still get rendered with that. So we will continue to use that.

@fgharo
Copy link
Contributor Author

fgharo commented Jun 23, 2022

@mprahl
However, I was able to verify that with PolicyGenerator directly it doesn't create the line breaks so that convinces me that its not PolicyGenerator perse. I can close this.

$ ~/Downloads/darwin-amd64-PolicyGenerator-v-1.7.0 policy-generator-config.yaml
...
                object-templates:
                    - complianceType: musthave
                      objectDefinition:
                        apiVersion: v1
                        data:
                            case1_blockScalar_literalClip: |
                                {{ fromConfigMap "default" "logs-config" "log-file" }}
                            case2_blockScalar_literalStrip: '{{ fromConfigMap "default" "logs-config" "log-file" }}'
                            case3_blockScalar_literalKeep: |
                                {{ fromConfigMap "default" "logs-config" "log-file" }}
                            case4_blockScalar_foldedClip: |
                                {{ fromConfigMap "default" "logs-config" "log-file" }}
                            case5_blockScalar_foldedStrip: '{{ fromConfigMap "default" "logs-config" "log-file" }}'
                            case6_blockScalar_foldedKeep: |
                                {{ fromConfigMap "default" "logs-config" "log-file" }}
                            case7_flowScalar_doubleQuotes: '{{ fromConfigMap "default" "logs-config" "log-file" }}'
                            case8_flowScalar_singleQuotes: '{{ fromConfigMap "default" "logs-config" "log-file" }}'
                            case9_flowScalar_plain: the-configmap-is-{{ fromConfigMap "default" "logs-config" "log-file" }}
                        kind: ConfigMap
                        metadata:
                            name: demo-of-string-formatting-problem
                            namespace: default
                            ...

@fgharo fgharo closed this as completed Jun 23, 2022
@fgharo
Copy link
Contributor Author

fgharo commented Jun 23, 2022

@mprahl
What about the line symbols not being respected? I assume this is related to PolicyGenerator as in my last example above I only used PolicyGenerator and the yaml scalar string symbols (|- |+ > >- >+) were not retained in the output. Is there a reason for this or should I open another issue?

@mprahl
Copy link
Member

mprahl commented Jun 23, 2022

@mprahl What about the line symbols not being respected? I assume this is related to PolicyGenerator as in my last example above I only used PolicyGenerator and the yaml scalar string symbols (|- |+ > >- >+) were not retained in the output. Is there a reason for this or should I open another issue?

This is expected because the YAML is parsed into a Go data-structure and then converted back to YAML in the generated policy. The output is equivalent it's just not the same style as the input.

@fgharo
Copy link
Contributor Author

fgharo commented Jun 23, 2022

@mprahl Gotcha. Thanks for explaining!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fgharo @mprahl