ci: combine integration tests

[mhofman]

refs: #7629

Description

This combines all longer integration type tests into a single workflow.

It also restores the logic to avoid running these tests on PRs before ready to merge.
It also restores duplicate checks to avoid re-running these tests on the exact same content.

It does not remove the github based concurrency limit for pull request, which makes these tests potentially cancel and re-start on the same content if the test was still in progress. I remember there were issues in the custom concurrent checks logic and I'll look into restoring that later.

This is the first step towards switching back to mergify.

Security Considerations

None

Scaling Considerations

Not production scaling, but this should help reining in how often these expensive CI jobs run.

Documentation Considerations

None for now

Testing Considerations

Will test the behavior on this PR

Upgrade Considerations

None

[mhofman]

Ok I think this is actually ready for review now.

@warner I don't automatically expect a review, just more an FYI
@raphdev I'm not sure why the docker upgrade test was added to the "test all packages". It didn't depend on any of these other jobs, and doesn't seem like the right fit. Just want to make sure that moving it won't break the Datadog reporting.
@michaelfig you're still the most familiar with these CI config changes, please review :)

[mhofman]

Sorry for the churn. I had to remove the run to completion test (it's not compatible with the skip duplicate test, at least not without tweaks). I did add a couple checks for merge strategy and linear history (which we should be able to mark as required once this lands)

[michaelfig]

LGTM! Just one minor comment for consideration.

Thanks for doing this work... looking forward to the next review.

[michaelfig]

FWIW, I'd like to rename automerge:merge to automerge:expert, in accordance with earlier discussions how it might otherwise be overused by people who may not know its nuances.

[michaelfig]

Or automerge:no-update.

[michaelfig]

Or automerge:manual-updates.

[mhofman]

I already updated the label description to indicate it's an expert option, and I'm hoping the new no-merge-commits check will help reduce it's usage (it should bump it off the queue if a merge commit is added to make the PR up to date)

[mhofman]

Out of these I prefer the automerge:no-updates

[michaelfig]

I can speak on behalf of the DataDog test recording integration: it won't break since it's dynamically registered, but may possibly start logging to a different metric than before. As long as @raphdev is aware, I think this is safe to merge.

[michaelfig]

As discussed, the local-npm failure is expected until #8250 lands.