Regression: sops 3.3.1 encrypts with expired gpg subkey

[mhyllander]

On my gpg master key, I have three expired 2048-bit subkeys for signing, encryption and authentication. I also have my current three 4096-bit subkeys that replaced the 2048-bit keys. Normally, when I encrypt a file with sops and specify my master key it will automatically find the correct encryption subkey to use (i.e. the 4096-bit current encryption subkey). This worked up until version 3.3.0.

However starting with 3.3.1, sops has started using the expired 2048-bit encryption subkey when it encrypts.

My setup:

22:43:39 sops-test $ cat .sops.yaml
creation_rules:
  - pgp: 'D4131455E3F1C14E9A4B25FCFEE2FB9581EB73A9'

22:43:42 sops-test $ gpg -kv D4131455E3F1C14E9A4B25FCFEE2FB9581EB73A9
pub   rsa4096/0xFEE2FB9581EB73A9 2016-12-11 [SC]
      Key fingerprint = D413 1455 E3F1 C14E 9A4B  25FC FEE2 FB95 81EB 73A9
uid                   [ultimate] Magnus Hyllander <magnus@hyllander.org>
sub   rsa2048/0x71E06E888FECB942 2016-12-11 [S] [expired: 2018-12-12]
sub   rsa2048/0x8FEC8495E06221DE 2016-12-11 [E] [expired: 2018-12-12]
sub   rsa2048/0xEE67AEF887A1B758 2016-12-11 [A] [expired: 2018-12-12]
sub   rsa4096/0x9161439145189377 2018-11-09 [A] [expires: 2020-01-15]
sub   rsa4096/0xB9FF4E94A57F92AF 2018-11-09 [S] [expires: 2020-01-15]
sub   rsa4096/0x86D7255D39F3B027 2018-11-09 [E] [expires: 2020-01-15]

22:45:54 sops-test $ cat test.yaml
foo: bar

Encrypting with sops 3.3.1:

22:46:10 sops-test $ ./sops-3.3.1.linux --version
sops 3.3.1 (latest)
22:46:14 sops-test $ ./sops-3.3.1.linux --encrypt test.yaml
foo: ENC[AES256_GCM,data:NCc/,iv:f9G03C6YdlpP7jYDLMRPCGBvJkXHT/awPEyAf5D7TgE=,tag:aQjrKFpFzIGeGBZAzbqfNA==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-08-23T20:46:25Z'
    mac: ENC[AES256_GCM,data:wAN18jQNOnY2OwvhUKQUOiXWMKnhToHD6qvfd/khVGjYUJQD1OR/CQB3oUjxr9mi7TYynIy3vf1pivKDIvOVRFQ9U3Ws3hOVvP/eowg9ips0fO4CJBFG2ts5cpoBOObzPRBpI2NCcZ3TGxoBCQY/Rj8O52aXj/d/t/sUnyMOyWA=,iv:cczMEv0697M4ufNN4C68ZDH1jgfja/G51sOFT0H6A4U=,tag:OU2uGczuTPKi5v8NcQZA7A==,type:str]
    pgp:
    -   created_at: '2019-08-23T20:46:24Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----

            wcBMA4/shJXgYiHeAQgAqgrMofK/epLouC5FDKWwG+/LSuGzZQmfw5jFFOMmmDjg
            c967dMFliZF27NwFQTWFK9Xb3nnzRyHvYGZ9j0FNJLNydNNTC0p7EDTpWvEK0oZO
            jkTd9mm6GnEbxZ03elbupyPuDwGV7aFh24PTkWs6OjKsyM0UfYn/oPlJLpD2N3jf
            X72mlozIdVnhKYGGnzxyJAusAbtlkAhGoyyFw2NAGFzcnz+zU9tnS1YENxlYFChh
            vattu/8+2IMwMbfaC1RPaNP23xZhr9y2RIzJW3cRCw4ezxW6JowwQLfELX2+LHo4
            Xq12vDf980mECu2VzKjRx93rExN4cImb9+d5mhaZONLgAeRDSWzl1nzofyGAU4dD
            iZIM4evo4BzgVuHINOCV4gm6aozggeUBMHxUzMjC/HHz+uVv7KIAE6Kha45snTGX
            F0SC7VuffOBg5B00f3EXMuQBiq/3ozx/75XijGRZF+FuUwA=
            =9KV4
            -----END PGP MESSAGE-----
        fp: D4131455E3F1C14E9A4B25FCFEE2FB9581EB73A9
    unencrypted_suffix: _unencrypted
    version: 3.3.1

22:46:25 sops-test $ sed -e 's/^ *//' | gpg --verbose -d -o /dev/null
            -----BEGIN PGP MESSAGE-----

            wcBMA4/shJXgYiHeAQgAqgrMofK/epLouC5FDKWwG+/LSuGzZQmfw5jFFOMmmDjg
            c967dMFliZF27NwFQTWFK9Xb3nnzRyHvYGZ9j0FNJLNydNNTC0p7EDTpWvEK0oZO
            jkTd9mm6GnEbxZ03elbupyPuDwGV7aFh24PTkWs6OjKsyM0UfYn/oPlJLpD2N3jf
            X72mlozIdVnhKYGGnzxyJAusAbtlkAhGoyyFw2NAGFzcnz+zU9tnS1YENxlYFChh
            vattu/8+2IMwMbfaC1RPaNP23xZhr9y2RIzJW3cRCw4ezxW6JowwQLfELX2+LHo4
            Xq12vDf980mECu2VzKjRx93rExN4cImb9+d5mhaZONLgAeRDSWzl1nzofyGAU4dD
            iZIM4evo4BzgVuHINOCV4gm6aozggeUBMHxUzMjC/HHz+uVv7KIAE6Kha45snTGX
            F0SC7VuffOBg5B00f3EXMuQBiq/3ozx/75XijGRZF+FuUwA=
            =9KV4
            -----END PGP MESSAGE-----
gpg: public key is 0x8FEC8495E06221DE
gpg: using subkey 0x8FEC8495E06221DE instead of primary key 0xFEE2FB9581EB73A9
gpg: encrypted with 2048-bit RSA key, ID 0x8FEC8495E06221DE, created 2016-12-11
      "Magnus Hyllander <magnus@hyllander.org>"
gpg: AES encrypted data
gpg: original file name=''

Notice that it has encrypted using the expired 0x8FEC8495E06221DE subkey.

Encrypting with sops 3.3.0:

22:47:35 sops-test $ ./sops-3.3.0.linux --version
sops 3.3.0
[info] sops 3.3.1 is available, update with `go get -u go.mozilla.org/sops/cmd/sops`
22:48:27 sops-test $ ./sops-3.3.0.linux --encrypt test.yaml
foo: ENC[AES256_GCM,data:AbJs,iv:9LVjrOTITIUYE3p0RznVBv/UQMJpXJrwsshiIcLLJ30=,tag:P1qUTpx/QJLFHzU+9M3TEw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-08-23T20:48:39Z'
    mac: ENC[AES256_GCM,data:gPDj8L/JEyzWmLAQxvM+w07Kp9vG2Zq5p/YoTJoUoDX40lWlYsox3ZThuDRlYcszgP5+2R7XlvkFjXAAelfAPl82CaqAGQrIkv3UhjuNJjcGyb9HQfj4wteGeiz+PLee/cUutKobF60bsVfbLMYlDgiqbLVMdCBzU/7CWaElCNk=,iv:2F5TyP8OezHmmGZH6ilEHTxkwNnomZVHzEZEVa3EYwo=,tag:j6JhYQdmxrhe83YROr+WKA==,type:str]
    pgp:
    -   created_at: '2019-08-23T20:48:38Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMA4bXJV0587AnAQ//YzuJ9UPmc6BlzUxOx4LpbDG9j6+TFZZ14y6sNNd6WHR3
            1eGPTLxs8Kz0mlGMoRWEbUU46CBSkPR5p8TP9akhUVzSuG0rIoKW23Zk3TZrfvVz
            tf98eRZlzBdgv/+QNMrpLWmG8zaOf1pnLr1hNWp4ClYpK51nIDW+oFNYkIBmgtGp
            AT/9eWEq3G+LNWq0EawGz9yVv98KgYCNb9Am3AlQr1KD+pE0cUdbXSGD+pgQN05W
            wJYv0ORcBqnhM4NxDleP8SbCTAG2fDZRBcmdMmWpVIBshvKS7vNbNN/RvfvPNwL4
            ny/g1x8k3Nd+IBRDN09alag/4+41ggtit5d0RN/vqpcaCCziuk7V2EUlu0/IaAtT
            3MmTJ81A/NjlYdeTuWuZc6HSAHUdMkGvxW/jJOZbBDQIcCN9pg7/jN8oaJnEIdKE
            yf7Ma/fuvAQ3dXWuxtwAqvgJ3Wbz4HYGMM4lnO/4fGjCL0r2n5J1FKdCnw2bn6/w
            2FzJQNxJffoKX1dv87SapUUE7DXoFus9VX3krlH1+mqVk3oDaHWhzovVqZCUNvgJ
            bglI2W33z4PGYJUaG7oVN7YbZ3YnrMqqQL+Sf8+FFl1inyPFkP7VWebCG39nY5w9
            6BwAn6Pcaf7i3wxytOsjotAOdQ+y8ifhl26pCQHHm3N0D9N9qJTb2FpSW1jmPefS
            XgE1SUDYyslYG02YGeW/N4cxPmxgujR00UUU+aOYdQAf8LMvxDCxe2WBgJ/09xWL
            rn7OpH28bhEqAyRiEtdb9yI60pA8hnv/ei0vQfrhqRszGRYKVo5yFXMCChjlOus=
            =xagC
            -----END PGP MESSAGE-----
        fp: D4131455E3F1C14E9A4B25FCFEE2FB9581EB73A9
    unencrypted_suffix: _unencrypted
    version: 3.3.0

22:48:39 sops-test $ sed -e 's/^ *//' | gpg --verbose -d -o /dev/null
            -----BEGIN PGP MESSAGE-----

            hQIMA4bXJV0587AnAQ//YzuJ9UPmc6BlzUxOx4LpbDG9j6+TFZZ14y6sNNd6WHR3
            1eGPTLxs8Kz0mlGMoRWEbUU46CBSkPR5p8TP9akhUVzSuG0rIoKW23Zk3TZrfvVz
            tf98eRZlzBdgv/+QNMrpLWmG8zaOf1pnLr1hNWp4ClYpK51nIDW+oFNYkIBmgtGp
            AT/9eWEq3G+LNWq0EawGz9yVv98KgYCNb9Am3AlQr1KD+pE0cUdbXSGD+pgQN05W
            wJYv0ORcBqnhM4NxDleP8SbCTAG2fDZRBcmdMmWpVIBshvKS7vNbNN/RvfvPNwL4
            ny/g1x8k3Nd+IBRDN09alag/4+41ggtit5d0RN/vqpcaCCziuk7V2EUlu0/IaAtT
            3MmTJ81A/NjlYdeTuWuZc6HSAHUdMkGvxW/jJOZbBDQIcCN9pg7/jN8oaJnEIdKE
            yf7Ma/fuvAQ3dXWuxtwAqvgJ3Wbz4HYGMM4lnO/4fGjCL0r2n5J1FKdCnw2bn6/w
            2FzJQNxJffoKX1dv87SapUUE7DXoFus9VX3krlH1+mqVk3oDaHWhzovVqZCUNvgJ
            bglI2W33z4PGYJUaG7oVN7YbZ3YnrMqqQL+Sf8+FFl1inyPFkP7VWebCG39nY5w9
            6BwAn6Pcaf7i3wxytOsjotAOdQ+y8ifhl26pCQHHm3N0D9N9qJTb2FpSW1jmPefS
            XgE1SUDYyslYG02YGeW/N4cxPmxgujR00UUU+aOYdQAf8LMvxDCxe2WBgJ/09xWL
            rn7OpH28bhEqAyRiEtdb9yI60pA8hnv/ei0vQfrhqRszGRYKVo5yFXMCChjlOus=
            =xagC
            -----END PGP MESSAGE-----
gpg: public key is 0x86D7255D39F3B027
gpg: using subkey 0x86D7255D39F3B027 instead of primary key 0xFEE2FB9581EB73A9
gpg: encrypted with 4096-bit RSA key, ID 0x86D7255D39F3B027, created 2018-11-09
      "Magnus Hyllander <magnus@hyllander.org>"
gpg: AES256 encrypted data
gpg: original file name=''

This is the expected behavior, where expired subkeys are ignored, and the current 0x86D7255D39F3B027 subkey is used.

[autrilla]

I don't see any relevant changes on our side that would cause this. My random guess would be that because of #458, something upstream changed and it broke this.

I'll see if I can figure out how to reproduce this, but it'd be great if you could bisect to get the commit that actually broke things.

[mhyllander]

Hi, thanks! I did as you suggested, and built for each of the commits between tags 3.3.0 and 3.3.1:

16:52:33 {sops::(3.3.1) %} sops $ git lg 3.3.0..3.3.1
* e9e1e8772 - (HEAD, tag: 3.3.1, origin/master, origin/HEAD, master) 3.3.1 (develop -> master) (#478) (2019-06-11) <AJ Bahnken>
* 300927c38 - Update README.rst (#469) (2019-05-27) <Teppei Fukuda>
* ea2a10b7a - Merge pull request #466 from kentso/regex-path (2019-05-16) <Adrian Utrilla>
* 811880337 - expand file path to full path before processing (2019-05-16) <Ken Tso>

I built each commit with go 1.12.5. Everything was OK until I built commit e9e1e87 (#478), then I got the wrong result.

I then repeated this for all the commits in #478. I found that commit 15dfcfa (#472) is the culprit. And that PR is very large...

[mhyllander]

Since this seems to be related to updated vendor packages, and since I noticed that the vendor subfolder has been removed the head of the develop branch, I also built c9e025f (the current develop HEAD). The problem is still there.

[autrilla]

Excellent, thanks! It seems like that vendoring commit updates vendor/golang.org/x/crypto/openpgp, so that's very likely the culprit.

[mhyllander]

It looks like this problem has been fixed along the way. I verified that release 3.5.0 works.