Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix for #548 - handle .ini files in decrypt.Data, add other helper #549

Merged
merged 5 commits into from
Oct 26, 2019

Conversation

dnozay
Copy link
Contributor

@dnozay dnozay commented Oct 15, 2019

No description provided.

@dnozay dnozay closed this Oct 15, 2019
@dnozay
Copy link
Contributor Author

dnozay commented Oct 15, 2019

retest

@dnozay dnozay reopened this Oct 15, 2019
@dnozay dnozay force-pushed the pr-548 branch 2 times, most recently from fd5c42d to 392657f Compare October 16, 2019 00:11
@dnozay dnozay closed this Oct 16, 2019
@dnozay dnozay reopened this Oct 16, 2019
@dnozay
Copy link
Contributor Author

dnozay commented Oct 16, 2019

looks like sometimes tests are failing with

$ vault secrets enable -version=1 kv
Error enabling: Post http://127.0.0.1:8200/v1/sys/mounts/kv: dial tcp 127.0.0.1:8200: connect: connection refused

@dnozay dnozay changed the title fix for #548 fix for #548 - handle .ini files in decrypt.Data, add other helper Oct 16, 2019
@dnozay
Copy link
Contributor Author

dnozay commented Oct 16, 2019

@autrilla ^ WDYT?

cmd/sops/common/common.go Outdated Show resolved Hide resolved
@ajvb
Copy link
Contributor

ajvb commented Oct 16, 2019

@dnozay Could you change this PR to be merged against the develop branch?

@dnozay dnozay changed the base branch from master to develop October 16, 2019 17:19
@dnozay dnozay force-pushed the pr-548 branch 2 times, most recently from 321b504 to 5c43cf0 Compare October 16, 2019 18:41
@dnozay
Copy link
Contributor Author

dnozay commented Oct 16, 2019

@ajvb i changed the merge base to develop branch

Copy link
Contributor

@autrilla autrilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should we have two separate implementations for "format" and for "path"? Correct me if I'm wrong, but it seems to me like we have a mapping of file extensions to stores, and then a default of BinaryStore. We could implement this in a single function, and even better, define a map[string]*Store for the extensions.

I can see the reason being that passing an "extension" instead of a file type to sops.Data is a bit weird, but if that's the reason, I'd make the file types an enum-like thing, so they're statically typed and not strings, and then write a PathToFormat function or something like that for conversion of file extensions to file types.

default:
store = &sopsjson.BinaryStore{}
}
func DataWithStore(data []byte, store sops.Store) (cleartext []byte, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any use case for which this should be public?

Copy link
Contributor Author

@dnozay dnozay Oct 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choosing the store is the contentious part. There are several strategies:

  • based on a format
  • based on an extension
  • based on whatever custom logic
    Example: you can use go-bindata to include assets.

if decrypt has the only stable api, then basically that says "trust me i know what i am doing", let me pass in the correct store.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think DataWithFormat would be a more useful API? Considering Store is not really meant to be used by third parties. And I think we should re-export Format in this file as well. Keep in mind only the decrypt code should be public to external projects, and right now we're requiring them to access go.mozilla.org/sops.

@codecov-io
Copy link

codecov-io commented Oct 17, 2019

Codecov Report

Merging #549 into develop will increase coverage by 0.07%.
The diff coverage is 0%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #549      +/-   ##
===========================================
+ Coverage    36.46%   36.54%   +0.07%     
===========================================
  Files           20       20              
  Lines         2863     2857       -6     
===========================================
  Hits          1044     1044              
+ Misses        1725     1719       -6     
  Partials        94       94
Impacted Files Coverage Δ
decrypt/decrypt.go 0% <0%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b5b7ae...a03712f. Read the comment docs.

@dnozay
Copy link
Contributor Author

dnozay commented Oct 17, 2019

I don't have a strong opinion with Data taking a format.

  • Sometimes you don't know the format, so you can't use decrypt.Data; and you need to use non-stable APIs (anything outside of decrypt) to guess.
  • Sometimes you don't have a file to work with (e.g. asset from go-bindata) but you may have a path, so you can't use decrypt.File.

@dnozay dnozay requested review from autrilla and ajvb October 21, 2019 14:58
Copy link
Contributor

@autrilla autrilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tl;dr of my review: my suggestion is to move away from strings into our own Format type ASAP, and then use that to make decisions on what store should be used.

cmd/sops/common/common.go Outdated Show resolved Hide resolved
cmd/sops/common/common.go Outdated Show resolved Hide resolved
@@ -139,19 +170,40 @@ func IsIniFile(path string) bool {
return strings.HasSuffix(path, ".ini")
}

// DefaultStoreForFormat returns the correct format-specific implementation
// of the Store interface given the format string
func DefaultStoreForFormat(format string) Store {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With my suggested changes, this should not be needed anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either StoreConstructors needs to be public or DefaultStoreForFormat needs to be public, so the CLI can consume it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you're right. I think I'd expose Format and the associated way to get a Store from a given format in the go.mozilla.org/sops package directly (so at the root of the repository) since it isn't command-line specific. It's not super important since it's not part of the public API, though, so feel free to just keep it here.

cmd/sops/common/common.go Show resolved Hide resolved
cmd/sops/common/common.go Show resolved Hide resolved
@autrilla
Copy link
Contributor

autrilla commented Oct 22, 2019

By the way, sorry this initially simple PR has turned into a somewhat major refactor of how we handle formats :(. And thanks a lot for all your work and patience!

Copy link
Contributor

@autrilla autrilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks excellent to me! @ajvb could you have a look as well?

@autrilla
Copy link
Contributor

@dnozay looks like you forgot to commit the formats package though

@dnozay
Copy link
Contributor Author

dnozay commented Oct 25, 2019

@autrilla - sorry about that; it is there now.

@autrilla
Copy link
Contributor

No worries! All good from my end, I'll merge now, and if @ajvb has any concerns he can raise them later and we can revert.

@autrilla autrilla merged commit d98bff6 into getsops:develop Oct 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants