Streaming YAML CLI processor

A CLI tool for querying and transforming YAML stream data:

• Grep matching documents (ie. K8s objects)
• Join multiple YAML files
• Get/add/edit/delete YAML nodes matching given selector
• Loop over documents and/or data arrays
• etc.

[input.yml] => [query or transformations] => [output.yml]

Note: The input YAML documents in a YAML stream are separated by ---.

• One-liner commands
  • yaml get selector
    • yaml get selector --print-key
    • yaml get array[*] --print-key
    • yaml get spec.containers[*].image --no-separator
  • yaml set "selector: value"
  • yaml default "selector: value"
  • yaml delete selector
  • yaml cat file1.yml file2.yml fileN.yml
  • yaml count
  • yaml doc $index
  • yaml len arraySelector
• Grep YAML documents (objects)
  • yaml grep "selector: value" ...
    • Grep k8s deployment object by name
  • yaml grep -v "selector: value" ...
    • Grep all k8s objects that don't create any Pods
    • Print first container's image of linkerd2 deployment objects
• Useful Kubernetes examples
  • Push all non-pod objects to k8s
  • Rollout k8s deployments from desired-state files sequentially
• Known issues
• Working with JSON
  • JSON to YAML
  • YAML to JSON
• Transformation files
  • yaml apply file1.yt file2.yt fileN.yt
    • Examples of transformation YAML files
• Feedback
• License

One-liner commands

yaml get selector

Print value of a YAML node matching the given selector.

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml get spec.containers[0].image
nats-streaming:0.10.0

yaml get selector --print-key

Since we're printing a value, the output might not necesarilly be a valid YAML.

If we're printing value of a primitive type (ie. string) and we need the output in a valid YAML format, so it can be processed further, we can explicitly print the node key in front of the value:

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml get spec.containers[0].image --print-key
image: nats-streaming:0.10.0

We can print multiple values, and they will be printed as separate objects:

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml get spec.containers[0].image spec.containers[1].image --print-key
image: nats-streaming:0.10.0
---
image: sidecar:1.0.1

yaml get array[*] --print-key

We can print all array items at once with a wildcard (array[*]) too:

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml get spec.containers[*].image --print-key
image: nats-streaming:0.10.0
---
image: sidecar:1.0.1

yaml get spec.containers[*].image --no-separator

Need to print values only?

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml get spec.containers[*].image --no-separator
nats-streaming:0.10.0
sidecar:1.0.1

yaml set "selector: value"

Add/overwrite field's value.

$ cat input.yml | yaml set "metadata.labels.environment: staging" > output.yml

yaml default "selector: value"

Set field's value, if no such value exists yet.

$ cat input.yml | yaml default "metadata.labels.environment: staging" > output.yml

yaml delete selector

Delete specific field.

$ cat input.yml | yaml delete "metadata.labels.environment" > output.yml

yaml cat file1.yml file2.yml fileN.yml

Join multiple YAML files into a single file with multiple documents separated by ---.

$ yaml cat k8s-apps/*.yml > output.yml

yaml count

Print number of YAML documents within the input YAML stream.

input.yml

document: this is doc 1
---
document: this is doc 2

$ yaml count input.yml
2

yaml doc $index

Print Nth (index=0..N-1) YAML document from the input YAML stream.

input.yml

document: this is doc 1
---
document: this is doc 2

$ yaml doc 1
document: this is doc 2

yaml len arraySelector

Print number of items within an array matching the given selector.

Useful for ranging over arrays.

pods=$(kubectl get pods -o yaml)
count=$(echo "$pods" | yaml len items)
for ((i=0; i < $count; i++)); do
    echo "$pods" | yaml get items[$i].status.phase
done

Grep YAML documents (objects)

Grep documents/objects matching all of the given selector: value pairs.

If a provided value is an array (ie. selector: [first, second]), the matching value must match at least one of the provided values (logical "OR").

yaml grep "selector: value" ...

Grep k8s deployment object by name

$ cat desired-state.yml | yaml grep "kind: Deployment" "metadata.name: linkerd"

yaml grep -v "selector: value" ...

Inverse grep.

Grep all k8s objects that don't create any Pods

$ cat desired-state.yml | yaml grep -v "kind: [Deployment, Pod, Job, ReplicaSet, ReplicationController]"

Print first container's image of linkerd2 deployment objects

$ cat linkerd.yml | yaml grep "kind: Deployment" | yaml get "spec.template.spec.containers[0].image"
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/web:stable-2.4.0
prom/prometheus:v2.10.0
gcr.io/linkerd-io/grafana:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0

Useful Kubernetes examples

Push all non-pod objects to k8s

$ cat desired-state.yml | yaml grep -v "kind: [Deployment, Pod, Job]"

Rollout k8s deployments from desired-state files sequentially

$ for file in *.yml; do
    out=$(cat $file | yaml grep "kind: Deployment" | kubectl apply -f -)

    for deploy in $(echo "$out" | cut -d' ' -f1); do
        kubectl rollout status --timeout 180s $deploy || {
            kubectl rollout undo $deploy
            exit 1
        }
    done
  done

Known issues

1. Merging complex nodes doesn't work well

set:
    metadata:
        we:
            cant:
                merge:
                    complex: objects
                    such:
                        as: this
                        properly:
                            just: yet

We'll want to fix this later. For now, use explicit paths to the final nodes:

set:
    metadata.we.cant.merge.complex: objects
    metadata.we.cant.merge.such.as: this
    metadata.we.cant.merge.such.properly.just: yet

2. Wildcard array[*] matching doesn't work yet

match:
    spec.template.spec.containers[*].name: prometheus

3. Selectors with . dots in the selector path, ie.

metadata:
    annotations:
        linkerd.io/inject: enabled

Since the match selectors are separated with . dots, we'll have to figure out how to support these selector keys with inner . dots.

We might wanna support

delete:
    - metadata.annotations."linkerd.io/inject"
set:
    metadata.annotations."rbac.authorization.kubernetes.io/autoupdate": true

Working with JSON

JSON to YAML

$ cat file.json | yaml --from=json grep 'kind: Pod'

YAML to JSON

$ cat file.yml | yaml grep 'kind: Pod' --to=json

Transformation files

All of the above examples, and more, can be described in YAML transformation file syntax. Multiple such transformations can be applied at once.

yaml apply file1.yt file2.yt fileN.yt

Apply multiple YAML "transformations", see the .yt file syntax below.

$ yaml cat k8s-apps/*.yml | yaml apply staging.yt enable-linkerd.yt > staging/desired-state.yml

staging.yt:

match:
    # all YAML objects
set:
    metadata.labels.environment: staging
---
match:
    kind: Deployment
    metadata.name: api
set:
    metadata.labels.first: updated-label
    spec.replicas: 3

enable-linkerd.yt:

match:
  kind: [Deployment, Pod]
default:
  metadata.annotations:
    linkerd.io/inject: enabled

Changes applied to the original object:

 apiVersion: apps/v1
 kind: Deployment
 metadata:
     name: api
     labels:
-        first: label
+        first: updated-label
         second: label
+        environment: staging
+    annotations:
+        linkerd.io/inject: enabled
 spec:
-    replicas: 1
+    replicas: 3
 ...

Examples of transformation YAML files

match:
    kind: Deployment
set:
    spec.template.spec.containers[*]:
        imagePullPolicy: IfNotPresent

match:
    kind: Deployment
set:
    spec.template.spec.nodeSelector:
        worker: generic

match:
    kind: Deployment
    metadata.name: api
set:
    spec.replicas: 3

match:
    kind: Ingress
set:
    metadata:
        annotations:
            kubernetes.io/ingress.class: nginx

match:
    kind: Deployment
delete: spec.replicas

match:
    kind: Deployment
    spec.template.spec.containers[0].image: nats-streaming
set:
    spec.template.spec.containers[0].image: nats-streaming:0.15.1

Feedback

Any feedback welcome! Please open issues and feature requests..

License

Licensed under the MIT License.