reorganize scripts + switch to circleCI 2.0 + various new qa scripts and their corresponding fixes

[Dieterbe]

the scripts dir has become a mess and it's time to reorganize.
While we're at it, make scripts more consistent in that they all
start in the root project dir, making it more easy to understand
the file paths used throughout the scripts

• some other stuff

[Dieterbe]

after lots of struggling getting ineffassign and go generate to work reliably on circleCI 1.0 (and ultimately failing), I'm pleased to say not only was upgrade to circleci 2.0 pretty smooth, it also made those things work. I'm not quite sure why. (perhaps something was wrong with our go install on the former, or the symlinking of the checkout dir to the path within $GOPATH, or ...) but anyway. now checkout dir is simply the proper dir, which simplifies a lot, and things just work.

If we were to republish this PR commit by commit you'd see this nice pattern of introducing a qa script which results in a build failure, followed by whatever code fixes that make the build succeed again.
however due to the extensive rebasing that visual pattern got lost, but I think the commits speak for themselves :)

[woodsaj]

The order of the build steps needs to be adjusted so that steps are run in order of complexity. Simple tests should run first, and complex tests last. This ensures that the more resource intensive steps only run if all other steps have passed.

[woodsaj]

end2end takes a long time, we should only run it after all other qa tests have passed.

[Dieterbe]

i've tried to take runtime into account. however some things are more important than others. let's say there's a code change that breaks end2end completely, then it's pointless to first obsess over typo's and minor code linting issues, and only find out after a few iterations of code pushing that the code is wrong and needs to be rewritten completely. I do think most of these code health qa scripts can be run in parallel which is something i want to migrate towards (at some point)

[woodsaj]

I think that is backwards. I think the linting issues and typo's should be resolved first.

By your logic, you will have to wait 20minutes between each code iteration before being told you have a simple typo. It seems insane to be running a full end2end test mulitple times as you fix minor issues that could be reported in a few seconds. With your approach it will take 10x longer to get the same amount of work done.

[Dieterbe]

good point. will fix

[woodsaj]

deployment does not appear to be a supported top level key. Looks like you need to use workflows

[woodsaj]

Your .circleci/config.yml doesnt look to be following the 2.0 spec. https://circleci.com/docs/2.0/configuration-reference/

I think you need to use workflows and break this up into separate jobs. See https://github.com/grafana/worldping-gw/blob/master/.circleci/config.yml as an example.

I would use a Docker image to build and test, then pass the built assets through (persist_to_workspace, attach_workspace) to a second job that uses a VM (machine:true) to build the packages, docker image and run the end2end test. If the branch is master it should also deploy.

[Dieterbe]

at this point the goal is to upgrade to circle 2.0 and do all the script rework stuff.
you're right that the deploy stanza was out of date and i have now set it up as described on
https://circleci.com/docs/2.0/configuration-reference/#deploy
the workflow stuff brings a lot of nice features (parrallelism, scheduled runs instead of on-push, etc) but that can wait.

[Dieterbe]

how does this look now? good to merge? the change in artifacts meant going from 1:25 upload time to 15s

[woodsaj]

LGTM