CI doesn't always run the service tests

[paulmelnikow]

CircleCI builds often don't correctly detect that they are on a PR. Currently the only way to know whether or not the service tests ran is to inspect the build logs.

That's because there is no way to reliably distinguish branch and PR builds in CircleCI.

Somtimes the cause is having CircleCI set up on the fork where the PR originates, triggering a branch build ahead of time, though we've also had cases where this happened on forks without CircleCI. Plus Dependabot branches are never on a fork.

As it turns out, there is a workaround suggested by Danger: configure CircleCI to run only on PRs.

That would necessitate either:

1. A separate CI service – or simply a separate repo – to run the daily tests
2. A PR-triggered bot that triggers the service-test builds on Circle programmatically
3. A cron-triggered bot that triggers the nightly Circle build programmatically

I experimented in #1936 with a radically different alternative, which is to specify the affected services in the commit history. This has an advantage of making development a bit easier, but it didn't really pan out. Maybe that work could be harvested for a dev-only tool.

[chris48s]

A separate CI service – or simply a separate repo – to run the daily tests

This seems like the quickest/simplest thing to do. Also refs #1584

[paulmelnikow]

I went ahead and set up another repo with Circle, and got Coveralls working as well. (Thanks for the reminder about that!)

The reason I chose Circle is because #979 was a long-standing issue solved by Circle, and I didn't want to solve that problem again.

I turned on "Only build PRs" for the main repo. I think the service test runner should start working reliably now, without other changes. Though let's see!

[paulmelnikow]

This is looking good. Coveralls is looking nice, too!

[chris48s]

Just thinking about this is slightly more detail, was the CI_PULL_REQUEST variable the thing that we couldn't rely on?

If so, now that we only run on PR, should we also remove the checks on that e.g:

shields/.circleci/config.yml

Line 176 in b922fca

if [[ ! -z $CI_PULL_REQUEST ]]; then

[paulmelnikow]

Yea, that was the main thing. Yep, for sure, those checks can go.