bursts of webhooks cause bots to fail

[bcoe]

I noticed this debugging #1714, and I believe it's the root cause of many of the bug reports (CC: @parthea, @Neenu1995 , @SurferJeffAtGoogle).

• Cloud Functions only serve a single request at a time (Node.js can handle multiple requests at once, so this is inefficient.).
• When OwlBot (or another bot) performs an action that creates many concurrent webhook events on GitHub, these WebHook events cause the number of Cloud Functions we're running to scale.

• During this scaling phase, response times are slow from cloud functions, resulting in 5 10s timeouts in webhooks (I believe 10 seconds is what GitHub allocates).

Potential Solutions

• Cloud functions recently added support for "min instances" which would allow us to run a larger pool of warmed up functions -- this could be very expensive, but might be smart for us to use in the short term.
• Cloud Run allows multiple concurrent requests, it would be a better model for us.

[tmatsuo]

We're running an experiment with Cloud Run frontend.

[bcoe]

@SurferJeffAtGoogle @parthea we just stress tested @tmatsuo's experiment with 80 concurrent OwlBot builds, everything worked perfectly, no failed Webhooks.

I think we're in a much healthier place, I suggest we wait to close this until after a week of experimenting though.

[tmatsuo]

We still see occasional webhook delivery failure, but I'd close this for owl-bot.
For other bots, it should still happen.

For a long term solution, I think we should have a shared frontend service for receiving webhook requests.
I also propose that we use the same frontend code for receiving Pub/Sub and Cron requests. We can use the same code, but we'll need a separate deployment because we only want to receive legitimate Pub/Sub and Cron requests from Cloud Scheduler and Pub/Sub.

• Design shared frontend
• Deploy the shared frontend and test it with canary bot
• Capacity planning and load test
• Migrate bots to the new frontend
• Migrate cron and scheduler request to the new frontend
• Ditch serverless scheduler proxy

[bcoe]

@tmatsuo lets keep this open to track work on other bots?

[sofisl]

This issue is going OOSLO soon. Perhaps this would be better labeled as a process, or we demote it to a P3, given that the holidays are coming up.

[bcoe]

@sofisl I think you made the right call downgrading this to p3, we could honestly probably call it a feature request alternatively.

@tmatsuo is working on a new design.

[bcoe]

cross linking #2227.

[chingor13]

We think this is resolved now. Opened #3313 to track progress in rolling out frontend cloud run services.