feat: filter changes using go list output

[kuisathaverat]

What does this PR do?

It extracts the PATH dependencies from the list of Go vendor modules

Why is it important?

This approach is automatic when you add new dependencies, also, it is more accurate filtering better the changes.

Related issues

Relates #16953

[v1v]

/LGTM with some NITs

[kuisathaverat]

@urso @jsoriano @andrewkroh
The k8s test is only triggered if you change a file in deploy/kubernetes/ for me looks wrong, it means that your code changes will not be tested on k8s. I'd said that merges to master should trigger those test also.

[jsoriano]

@urso @jsoriano @andrewkroh
The k8s test is only triggered if you change a file in deploy/kubernetes/ for me looks wrong, it means that your code changes will not be tested on k8s. I'd said that merges to master should trigger those test also.

Agree, changes in other places can also affect deployments in kubernetes.

[urso]

I see I'm a little late to the discussion, but I wonder how effective this change will be.

Can we document what exactly getVendorPatterns will return? Just from reading the code it is far from obvious. Does it contain the package names or the vendor paths?

Given that we always add the regular expressions ^libbeat/.* and ^vendor/.* I wonder what the exact effect of goVendorPatterns will be.
If getVendorPatterns creates regular expressions, would it make sense to anchor those with ^?

Running the provided script locally I get paths like this:

github.com/gofrs/uuid/.*
github.com/gogo/protobuf/proto/.*
github.com/gogo/protobuf/sortkeys/.*
github.com/golang/protobuf/proto/.*
github.com/golang/protobuf/ptypes/.*
github.com/golang/protobuf/ptypes/any/.*
github.com/golang/protobuf/ptypes/duration/.*
github.com/golang/protobuf/ptypes/timestamp/.*
github.com/golang/snappy/.*

These are clearly dependencies that should be in vendor. But the path is incorrect.

Only a small set of packages are prefixed with vendor:

vendor/golang.org/x/crypto/chacha20/.*
vendor/golang.org/x/crypto/chacha20poly1305/.*
vendor/golang.org/x/crypto/cryptobyte/.*
vendor/golang.org/x/crypto/cryptobyte/asn1/.*
vendor/golang.org/x/crypto/curve25519/.*
vendor/golang.org/x/crypto/hkdf/.*
vendor/golang.org/x/crypto/internal/subtle/.*
vendor/golang.org/x/crypto/poly1305/.*
vendor/golang.org/x/net/dns/dnsmessage/.*
vendor/golang.org/x/net/http/httpguts/.*
vendor/golang.org/x/net/http/httpproxy/.*
vendor/golang.org/x/net/http2/hpack/.*
vendor/golang.org/x/net/idna/.*
vendor/golang.org/x/sys/cpu/.*
vendor/golang.org/x/text/secure/bidirule/.*
vendor/golang.org/x/text/transform/.*
vendor/golang.org/x/text/unicode/bidi/.*
vendor/golang.org/x/text/unicode/norm/.*

No idea why these packages are prefixed with vendor, but the output actually includes these packages twice:

golang.org/x/text/transform/.*
golang.org/x/text/unicode/bidi/.*
golang.org/x/text/unicode/norm/.*
vendor/golang.org/x/text/transform/.*
vendor/golang.org/x/text/unicode/bidi/.*
vendor/golang.org/x/text/unicode/norm/.*

[kuisathaverat]

Can we document what exactly getVendorPatterns will return? Just from reading the code it is far from obvious. Does it contain the package names or the vendor paths?

It returns the output of go list -mod=vendor adding .* at the end of each line. It also removes the stringgit.luolix.top/elastic/beats/v7 to make the beats paths match.

These are clearly dependencies that should be in vendor. But the path is incorrect.

because not all packages have vendor on the beginning the regexp is generated without '^'

No idea why these packages are prefixed with vendor, but the output actually includes these packages twice:

we can filter them in a follow-up