Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not crash if an empty YAML file is encrypted #1290

Merged
merged 1 commit into from
Sep 21, 2023

Conversation

felixfontein
Copy link
Contributor

An empty YAML file results in no tree branch, which trips over encrypt since it assumes there's at least one.

I'm not sure this is the correct way to fix this, since the encrypted result will be empty as well (zero tree branches).

On the other hand, if you encrypt a YAML file with {} or --- in it (and nothing else), there will be at least one tree branch. Decrypting the resulting file yields {}. Decrypting an empty file yields an sops metadata not found error.

@felixfontein felixfontein requested a review from a team September 16, 2023 13:19
Copy link
Member

@hiddeco hiddeco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if it wouldn't be better to abort the operation if len(branches) <= 0, instead of continuing to execute a variety of code paths.

In addition, wondering if this is the right place to address this issue. As it appears to be YAML specific; what do the other stores do when they run into an empty document? If they e.g. error out, could this behavior then be replicated within the YAML store?

cmd/sops/encrypt.go Outdated Show resolved Hide resolved
@felixfontein
Copy link
Contributor Author

In addition, wondering if this is the right place to address this issue. As it appears to be YAML specific; what do the other stores do when they run into an empty document? If they e.g. error out, could this behavior then be replicated within the YAML store?

YAML is different from other file formats such JSON, INI etc. in that the other file formats always contain exactly one document, so an empty JSON/INI/... results in one empty branch.

But an empty YAML file contains no document (https://yaml.org/spec/1.2.2/#92-streams), so you get zero branches. This is different from a YAML file which contains a single empty document. The following are examples of a YAML file with a single empty document:

---

or

{}

So we could either simply not support zero-document YAML files, or we convert them to single empty document YAML files (which is kind of wrong), or we allow to 'encrypt' them (which is what the PR right now does - the result cannot be decrypted though, though the decryption code could also be adjusted to convert an empty YAML document back to an empty YAML document).

@hiddeco
Copy link
Member

hiddeco commented Sep 19, 2023

Think having the encrypt and decrypt command handle empty documents gracefully is the best option to go with, as I think the behavior most users will be after is running encryption (or decryption) on a range of files expecting success (or an error, but not because the document is empty).

In addition, I think that changing it to {} would yield an unexpected outcome during "decryption", as you would end up with a different file than you started with (in terms of emptiness).

@felixfontein
Copy link
Contributor Author

Hmm, while looking at how to implement that, I found some more problems:

  1. The json, binary, and env output stores will crash if no tree branch is there.
  2. The ini output store treats zero branches the same as one empty branch.
  3. Adjusting the YAML store load an empty "encrypted" file is not so simple. Also LoadEncryptedFile is expected to return the metadata as well.
  4. (Not really related: if there are two or more YAML documents, all stores except YAML will silently ignore all except the first one when emitting. Not sure whether this is a problem, but maybe emitting a warning woudn't be bad.)

For the first three points, it might be easier to disallow empty inputs completely. What do you prefer (given the above points)?

@hiddeco
Copy link
Member

hiddeco commented Sep 20, 2023

With "disallow", you are aiming at erroring out?

@felixfontein
Copy link
Contributor Author

Yes. Which would be similar to the current behavior, except that the message is a nice text and not a stacktrace :-)

@felixfontein
Copy link
Contributor Author

(It would still be possible to change this at some point later in time, as a new feature. But at least the crash would go away for now.)

@hiddeco
Copy link
Member

hiddeco commented Sep 20, 2023

This sounds good to me, we can ask the community for more feedback and use-cases afterwards.

@felixfontein
Copy link
Contributor Author

I completely changed the implementation. There's now a new error which is used when a completely empty document is provided for encryption.

@felixfontein felixfontein marked this pull request as ready for review September 21, 2023 06:09
This only affects empty YAML files, since only these can contain zero documents.

Signed-off-by: Felix Fontein <felix@fontein.de>
@felixfontein felixfontein merged commit b7da2fc into getsops:main Sep 21, 2023
9 checks passed
@felixfontein felixfontein deleted the empty branch September 21, 2023 06:50
@felixfontein
Copy link
Contributor Author

@hiddeco thanks for reviewing this one as well!

@hiddeco hiddeco added this to the v3.8.1 milestone Sep 21, 2023
mergify bot referenced this pull request in dbsystel/cdk-sops-secrets Oct 11, 2023
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/getsops/sops/v3](https://github.com/getsops/sops) | require | patch | `v3.8.0` -> `v3.8.1` |

---

### Release Notes

<details>
<summary>getsops/sops (github.com/getsops/sops/v3)</summary>

### [`v3.8.1`](https://github.com/getsops/sops/releases/tag/v3.8.1)

[Compare Source](https://github.com/getsops/sops/compare/v3.8.0...v3.8.1)

#### Note from the Maintainers

In this release of SOPS, we have focused on landing a variety of bug fixes to improve the overall user experience.

For a comprehensive list of changes, please refer to [`CHANGELOG.rst`](https://github.com/getsops/sops/blob/v3.8.1/CHANGELOG.rst).

##### Important Information for SOPS SDK Users

With the project transitioning from the Mozilla Foundation to the CNCF, the Go module path has been updated to reflect this change in ownership. If you use `go.mozilla.org/sops/v3` as a library, going forward, import the Go Module using `github.com/getsops/sops/v3`. Apart from this small adjustment, the SDK's API remains fully backward compatible.

For a one-liner to quickly implement this change throughout your codebase, please refer to: [https://github.com/getsops/sops/issues/1246#issuecomment-1625526429](https://github.com/getsops/sops/issues/1246#issuecomment-1625526429)

#### Installation

To install `sops`, download one of the pre-built binaries provided for your platform from the artifacts attached to this release.

For instance, if you are using Linux on an AMD64 architecture:

```shell

### Download the binary
curl -LO https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64

### Move the binary in to your PATH
mv sops-v3.8.1.linux.amd64 /usr/local/bin/sops

### Make the binary executable
chmod +x /usr/local/bin/sops
```

##### Verify checksums file signature

The checksums file provided within the artifacts attached to this release is signed using [Cosign](https://docs.sigstore.dev/cosign/overview/) with GitHub OIDC. To validate the signature of this file, run the following commands:

```shell

### Download the checksums file, certificate and signature
curl -LO https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.checksums.txt
curl -LO https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.checksums.pem
curl -LO https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.checksums.sig

### Verify the checksums file
cosign verify-blob sops-v3.8.1.checksums.txt \
  --certificate sops-v3.8.1.checksums.pem \
  --signature sops-v3.8.1.checksums.sig \
  --certificate-identity-regexp=https://github.com/getsops \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com
```

##### Verify binary integrity

To verify the integrity of the downloaded binary, you can utilize the checksums file after having validated its signature:

```shell

### Verify the binary using the checksums file
sha256sum -c sops-v3.8.1.checksums.txt --ignore-missing
```

##### Verify artifact provenance

The [SLSA provenance](https://slsa.dev/provenance/v0.2) of the binaries, packages, and SBOMs can be found within the artifacts associated with this release. It is presented through an [in-toto](https://in-toto.io/) link metadata file named `sops-v3.8.1.intoto.jsonl`. To verify the provenance of an artifact, you can utilize the [`slsa-verifier`](https://github.com/slsa-framework/slsa-verifier#artifacts) tool:

```shell

### Download the metadata file
curl -LO  https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1.intoto.jsonl

### Verify the provenance of the artifact
slsa-verifier verify-artifact <artifact> \
  --provenance-path sops-v3.8.1.intoto.jsonl \
  --source-uri github.com/getsops/sops \
  --source-tag v3.8.1
```

#### Container Images

The `sops` binaries are also available as container images, based on Debian (slim) and Alpine Linux. The Debian-based container images include any dependencies which may be required to make use of certain key services, such as GnuPG, AWS KMS, Azure Key Vault, and Google Cloud KMS. The Alpine-based container images are smaller in size, but do not include these dependencies.

These container images are available for the following architectures: `linux/amd64` and `linux/arm64`.

##### GitHub Container Registry

-   `ghcr.io/getsops/sops:v3.8.1`
-   `ghcr.io/getsops/sops:v3.8.1-alpine`

##### Quay.io

-   `quay.io/getsops/sops:v3.8.1`
-   `quay.io/getsops/sops:v3.8.1-alpine`

##### Verify container image signature

The container images are signed using [Cosign](https://docs.sigstore.dev/cosign/overview/) with GitHub OIDC. To validate the signature of an image, run the following command:

```shell
cosign verify ghcr.io/getsops/sops:v3.8.1 \
  --certificate-identity-regexp=https://github.com/getsops \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  -o text
```

##### Verify container image provenance

The container images include [SLSA provenance](https://slsa.dev/provenance/v0.2) attestations. For more information around the verification of this, please refer to the [`slsa-verifier` documentation](https://github.com/slsa-framework/slsa-verifier#containers).

#### Software Bill of Materials

The Software Bill of Materials (SBOM) for each binary is accessible within the artifacts enclosed with this release. It is presented as an [SPDX](https://spdx.dev/) JSON file, formatted as `<binary>.spdx.sbom.json`.

#### What's Changed

-   build(deps): Bump the ci group with 3 updates by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/getsops/sops/pull/1295](https://github.com/getsops/sops/pull/1295)
-   pgp: improve handling of GnuPG home dir by [@&#8203;hiddeco](https://github.com/hiddeco) in [https://github.com/getsops/sops/pull/1298](https://github.com/getsops/sops/pull/1298)
-   Delete sops encrypted file we don't have keys for by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1288](https://github.com/getsops/sops/pull/1288)
-   Improve handling of errors when binary store handles bad data by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1289](https://github.com/getsops/sops/pull/1289)
-   On macOS, prefer XDG_CONFIG_HOME over os.UserConfigDir() by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1291](https://github.com/getsops/sops/pull/1291)
-   Do not crash if an empty YAML file is encrypted by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1290](https://github.com/getsops/sops/pull/1290)
-   Fix descriptions of unencrypted-regex and encrypted-regex flags, and ensure unencrypted_regex is considered in config validation by [@&#8203;mitar](https://github.com/mitar) in [https://github.com/getsops/sops/pull/1300](https://github.com/getsops/sops/pull/1300)
-   build(deps): Bump the go group with 4 updates by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/getsops/sops/pull/1306](https://github.com/getsops/sops/pull/1306)
-   build(deps): Bump the ci group with 1 update by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/getsops/sops/pull/1301](https://github.com/getsops/sops/pull/1301)
-   Handle return values of dec.Token() to improve error messages by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1307](https://github.com/getsops/sops/pull/1307)
-   pgp: make error the last return value by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1310](https://github.com/getsops/sops/pull/1310)
-   pgp: do not require abs path for SopsGpgExecEnv  by [@&#8203;holiman](https://github.com/holiman) in [https://github.com/getsops/sops/pull/1309](https://github.com/getsops/sops/pull/1309)
-   decrypt: fix dropped error by [@&#8203;alrs](https://github.com/alrs) in [https://github.com/getsops/sops/pull/1304](https://github.com/getsops/sops/pull/1304)
-   Handle errors by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1311](https://github.com/getsops/sops/pull/1311)
-   Report key rotation errors by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1317](https://github.com/getsops/sops/pull/1317)
-   cmd/sops/main.go: make sure to wrap raw errors with toExitError() by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1318](https://github.com/getsops/sops/pull/1318)
-   build(deps): Bump the go group with 7 updates by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/getsops/sops/pull/1319](https://github.com/getsops/sops/pull/1319)
-   Enrich AWS authentication documentation by [@&#8203;nsantiago2719](https://github.com/nsantiago2719) in [https://github.com/getsops/sops/pull/1272](https://github.com/getsops/sops/pull/1272)
-   Better error reporting for missing gpg binary by [@&#8203;makkes](https://github.com/makkes) in [https://github.com/getsops/sops/pull/1286](https://github.com/getsops/sops/pull/1286)
-   Improve RST and MD files by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1320](https://github.com/getsops/sops/pull/1320)
-   Add linting for RST and MD files by [@&#8203;felixfontein](https://github.com/felixfontein) in [https://github.com/getsops/sops/pull/1287](https://github.com/getsops/sops/pull/1287)
-   Update dependencies by [@&#8203;hiddeco](https://github.com/hiddeco) in [https://github.com/getsops/sops/pull/1325](https://github.com/getsops/sops/pull/1325)
-   Prepare v3.8.1 by [@&#8203;hiddeco](https://github.com/hiddeco) in [https://github.com/getsops/sops/pull/1324](https://github.com/getsops/sops/pull/1324)

#### New Contributors

-   [@&#8203;mitar](https://github.com/mitar) made their first contribution in [https://github.com/getsops/sops/pull/1300](https://github.com/getsops/sops/pull/1300)
-   [@&#8203;holiman](https://github.com/holiman) made their first contribution in [https://github.com/getsops/sops/pull/1309](https://github.com/getsops/sops/pull/1309)
-   [@&#8203;alrs](https://github.com/alrs) made their first contribution in [https://github.com/getsops/sops/pull/1304](https://github.com/getsops/sops/pull/1304)
-   [@&#8203;nsantiago2719](https://github.com/nsantiago2719) made their first contribution in [https://github.com/getsops/sops/pull/1272](https://github.com/getsops/sops/pull/1272)
-   [@&#8203;makkes](https://github.com/makkes) made their first contribution in [https://github.com/getsops/sops/pull/1286](https://github.com/getsops/sops/pull/1286)

**Full Changelog**: getsops/sops@v3.8.0...v3.8.1

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/markussiebert/cdk-sops-secrets).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants