sometimes owl bot doesn't run

[SurferJeffAtGoogle]

On bilge pump duty this morning, I've already seen 5 pull requests where owl bot never ran, so I couldn't merge them.

For example: googleapis/nodejs-service-management#63

These pull requests would already have merged if not for owl bot, so this is slowing us down.

[tmatsuo]

Do we know why? Were there a burst of webhook requests at the time when the PR is created?

[SurferJeffAtGoogle]

More examples:
googleapis/nodejs-dlp#629
googleapis/nodejs-contact-center-insights#3

[SurferJeffAtGoogle]

@bcoe has some ideas why.

[SurferJeffAtGoogle]

More examples:
googleapis/nodejs-dms#24
googleapis/nodejs-life-sciences#9
googleapis/nodejs-cloud-container#461

[bcoe]

So far, whenever I've dug into these issues, I've found a corresponding set of timeouts on GitHub WebHooks as described here:

#2114

I think once we address #2114, we might notice that failures to launch the post processor are rare.

[SurferJeffAtGoogle]

More examples:
googleapis/nodejs-retail#67
googleapis/nodejs-analytics-admin#132
googleapis/nodejs-life-sciences#11

[SurferJeffAtGoogle]

More examples:
googleapis/nodejs-iam-credentials#66
googleapis/nodejs-analytics-data#122

[bcoe]

@SurferJeffAtGoogle we found a bug on some of the repos you linked, which was us failing to report a failed status check when config was invalid (we had slightly off config in some repos).

I believe the other failures to run were a result of us missing Webhooks when traffic is bursty, @tmatsuo has addressed this by splitting apart our frontend for OwlBot from our backend job processing. We've tested and have been able to handle 80 jobs concurrently, without any failures.