Cannot output empty yaml file since sops-v3.7.0

[ikedam]

• sops-3.6.1

  $ curl -Lso sops-v3.6.1 https://github.com/mozilla/sops/releases/download/v3.6.1/sops-v3.6.1.linux
  $ chmod 755 sops-v3.6.1
  $ echo '{}' | ./sops-v3.6.1 --pgp FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4 --input-type yaml --output-type yaml -e /dev/stdin 2>/dev/null | ./sops-v3.6.1 --input-type yaml --output-type yaml -d /dev/stdin
  {}
  $
  

• sops-3.7.1 (same to sops-3.7.0)

  $ curl -Lso sops-v3.7.1 https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux
  $ chmod 755 sops-v3.7.1
  $ echo '{}' | ./sops-v3.7.1 --pgp FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4 --input-type yaml --output-type yaml -e /dev/stdin 2>/dev/null | ./sops-v3.7.1 --input-type yaml --output-type yaml -d /dev/stdin
  Error dumping file: Error marshaling to yaml: yaml: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS, but got document end
  $
  

Encoded outputs are same for both versions:

• sops-v3.6.1

  $ echo '{}' | ./sops-v3.6.1 --pgp FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4 --input-type yaml --output-type yaml -e /dev/stdin 2>/dev/null
  sops:
      kms: []
      gcp_kms: []
      azure_kv: []
      hc_vault: []
      lastmodified: '2021-07-23T04:29:26Z'
      mac: ENC[AES256_GCM,data:UH3M6XfWD7emEY7SS7ZS3ix0ZJJaFbbyJBaJNQD4CuTsaCvNcoXtwfAPf/52/SjhEPVi+LwH6lNwexNUILT1pJqR9HA6SaTcVekXmZrsfM1KI8qn8Tg8ZYl5d51LcouwqO5CChFyvK7aWiTy9d0MPJsN/nZ5YABI7gSGuiTxwFU=,iv:zWvwY1E0kcHmlY3DhM00l1DMKq5WsoboO/FvS7eeFgY=,tag:lP6T17vSSf3FPD6JPCVb/g==,type:str]
      pgp:
      -   created_at: '2021-07-23T04:29:25Z'
          enc: |
              -----BEGIN PGP MESSAGE-----
  
              hQEMAyUpShfNkFB/AQf+IQre0LGWgNaNEXdJjcsDB54KLsvR5YV44Zf0eXnQBvvf
              TpBdeMvRYxfsh+rxxZBb6NT8Fhp+rNRgglUBv4MwL8c3qimskjqBcofQuwdU5RUB
              EC2jxQpenhEEup5BWF/IhkcygaWVdDYYDYeVqARmteZcw4w/FIic5P0Rm0AGcrp3
              FvjxqgxJIlYncx//1XjFeqmZvpeLuFNWh+z4K1Ycy2t0ArF1wrKOYseJ4hLTmE0A
              n2e3jVJA0x9UXWkqmMsk0olcbWrZBlHuojMqv+oLZxtHN64tcScB0FL7LmBSoOL+
              eBmYZtOCu9SuSx/r4zl7HxMrvWeTm9ArkCVtsVUF0dJeAQbR6v9OfJvZipuiWAqN
              6sJIpGIBVFZXx90ZQ0f2f29RhPmwA2eng0W/GyNUeBqa6M0NiI54CzgCgSyqCvQF
              oc2pt992TbBbR4ZMUrFGo/celqQOvCygonUgQ8jkrw==
              =Pu21
              -----END PGP MESSAGE-----
          fp: FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
      unencrypted_suffix: _unencrypted
      version: 3.6.1
  $
  

• sops-v3.7.1

  $ echo '{}' | ./sops-v3.7.1 --pgp FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4 --input-type yaml --output-type yaml -e /dev/stdin 2>/dev/null
  sops:
      kms: []
      gcp_kms: []
      azure_kv: []
      hc_vault: []
      age: []
      lastmodified: "2021-07-23T04:29:07Z"
      mac: ENC[AES256_GCM,data:xSr4qYnQFnrbtqBQertx8eW5uE7E2sT00Sv4VegoHptfYPyzh/h4DzgTgN7/1dcAAeEl9UaoEUXc047RFzffuDNg9jLXYgqLtYQ3EKMmve6oMqGaWrmEyRoDLfi7fxwcFclr50YfVav6cc1a3UEwQQfXV/zx0aSQqVgKHXcCVdo=,iv:wJzZ6tFjM/5yKoPPL7GdJFaLVxeJDxpKjZmNqGKix04=,tag:j6LDnnOXFqW1SdAVhV2SEA==,type:str]
      pgp:
          - created_at: "2021-07-23T04:29:05Z"
            enc: |
              -----BEGIN PGP MESSAGE-----
  
              hQEMAyUpShfNkFB/AQgAr7skCg1FfuugFib5AI8vBVagI3rtpBQNslvZoCn8qZ28
              WWJBq9ym9pcJQDO70ja9Y4/mfnfEkD4B0zjpXcth6B2myBST2FlITT97wbiClWjY
              8o2Bzdq5CDFvDgoKNXZM6fEf2kYgSIzTTBw3J1I5R7o0bmYuPEuFVunfgo3Wsk3u
              Qp6DyKtXhgWZs6tNkP06zml6OhGwcrIPejWciZZDhXi0gL/eGGsOOmzkfokDHkwV
              Zzj59JVmQDGmNRLJnPabHqy38UtyA4+CN4A7jYxG+0YPaqkDIU8XIMsaGjC9sokI
              3ryZTha5CBg0HVvkE+iewGMxgpBw2umKCCozIvhrltJeAesExKXBNFpUellDdRqj
              CCFx9j7mE7ixRaJdw+LuaOh+Mzv6bt+YVuQ0tUEapephHafrZ0oxBs7emySBeFVY
              S00SnOAkvmk1IzBR1aUzym88thTzUBCHKXE6TPG9xw==
              =xcLN
              -----END PGP MESSAGE-----
            fp: FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
      unencrypted_suffix: _unencrypted
      version: 3.7.1
  $
  

[ikedam]

This looks caused for here: https://github.com/mozilla/sops/blob/v3.7.1/stores/yaml/store.go#L364
No Content is set and cause the error.
I can't get why this code exists.

[ikedam]

https://github.com/mozilla/sops/blob/v3.7.1/stores/yaml/store_test.go#L168
might be related?

[jarrettprosser]

Switching back this behaviour will be super useful for my application of sops, thanks!

[felixfontein]

@jarrettprosser would you mind describing what you need this behavior for? (@ikedam you as well maybe?) Please also see my comment: #791 (comment).

[jarrettprosser]

@felixfontein sure thing. I'm not using sops directly, but it's part of a toolchain which broke when this change came into v3.7.0.

We deploy resources to Kubernetes clusters using ArgoCD. In the git repos that represent resources, we use helm secrets to encrypt sensitive values so they aren't committed in plaintext. Helm secrets uses sops under the hood to encrypt yaml files.

In order for ArgoCD to render the secret files, we need to include helm secrets as a custom tool. We configure the ArgoCD applications to use a custom tool called helm3-secrets which uses helm secrets template to generate the manifests. This includes the values.yaml file, as well as a secrets.yaml file.

Sometimes, the deployment doesn't actually have sensitive values. Previously, we could have a secrets file with no content, encrypted by sops, as a placeholder so that the template command would run correctly. When sops 3.7.0 was released and we updated our ArgoCD, we found deployments with these placeholder files would fail to render with the Error marshaling to yaml. Our workaround currently is to remove the placeholder file and change the app definition to use the built-in helm tool, without helm secrets. It works fine then, but It'll be nice to be able to upgrade without needing to modify the deployments!

[ikedam]

I have application servers configured with yaml.
I manage configurations with common configurations in base.yaml and specific configurations in branch.yaml. These configurations contains sensitive values like tokens, and encrypted with sops.
Those files are fed into application servers after merging into a single file with:

docker run -v "$(pwd):/workspace" -w /workspace --rm mikefarah/yq:2 yq m -x base.yaml branch.yaml >config.yaml

branch.yaml contains no configurations for some application servers, and should be {}.

[ikedam]

IMO:

• sops doesn't support top-level arrays: https://github.com/mozilla/sops#top-level-arrays . Not tested, but I believe it doesn't support any non-map values, like null or a single value. It doesn't look strange to me that sops doesn't support empty documents as they are also non-map values.
• Referring sops/3.6.0 encrypted yaml file is broken (even to sops) #695 (comment), they seem just to request preserving empty documents, not encrypting empty documents. So it also can be a solution to provide an option to preserve empty documents when encrypting / decrypting.

[andrew-demb]

Fixed with #908
Shipped with v3.7.3

Thanks @ikedam