[Research] Migrate to golang/dep from govendor

[andrewkroh]

Abstract: I investigated migrating from govendor to dep for managing the dependencies of the Beats project. In my opinion the the benefits don't (yet) justify switching to another tool.

Why change tools? - govendor works pretty well for elastic/beats, but it doesn't work so well for third-party projects that are based on elastic/beats (e.g. community beats or apm-server). The problem is that it does not have the concept of transitive dependencies, and so bootstraping a new project with the right set of transitive dependencies is hard. dep will figure out the right transitive dependencies so it's worth consideration.

Downsides of dep are that it's experimental and will probably be fully replaced at some point by Go modules. Several bugs (or missing features) make this migration less than ideal.

• It can't vendor non-go directories. dep cannot vendor dependencies that do not have Go source code golang/dep#1306
  This is the most limiting feature from my point of view because we still have to
  manage build scripts (.py, Makefile, packaging templates) separately and on our own.
  If this were fixed I'd reconsider the migration.
• Symlinks in dependencies are not removed by pruning. Doesn't prune symlinks golang/dep#1625
• Sometimes it runs out of file handles on macOS b/c we have lots of deps. Exceeded available file handles on OSX. Error message should reflect this. golang/dep#1044
• We'd have to update make notice to understand the Gopkg.lock TOML data.

Summary I think we should wait about 6 months and re-evaluate the tool ecosystem. Go modules might be ready for use or the long term plan for dep might become more clear. In the meantime we might be able to improve the developer experience for community beats by adopting some of the ideas from apm-server.

You can see the branch where I converted Beats over to dep at https://github.com/andrewkroh/beats/commits/feature/use-golang-dep. I added a mage syncTools task that mimics the apm-server's rsync script.

[tsg]

+1 on waiting to see if we can use go modules. We can experiment with those already, by using the go 1.11 release candidates.

[cyrilleverrier]

Don't forget vgo (Versioned Go Prototype): https://research.swtch.com/vgo-intro
Yet another official packaging and versioning tool :)

[fearful-symmetry]

We're planning to migrate to modules, closing this.