BigQuery is very flakey on local / self-hosted

[neilkakkar]

Bigquery client takes > 30s for insert requests when run locally. Need to investigate why this is happening, since if we're facing this, other self-hosted clients would be too.

What's mystifying is that it works perfectly fine on cloud - there's never* a timeout.

While we don't lose data, thanks to the retry job queues, it would be nice to figure out why this is happening in the first place.

---

More context: See #9

[Twixes]

Could you flesh this issue out a bit more please? I have no idea what the problem is without reading all of the linked issue (which is real ｌｏｎｇ)

[neilkakkar]

Local / self-hosted:

vs Cloud:

---

Note the timing differences, too!

[andrewm4894]

just adding an example from our posthog - lots of variance and some longish looking insert times and some failing it seems once they hit 30s

[neilkakkar]

More data points: I tried the bigquery plugin while setting the TASK_TIMEOUT to 60 seconds, and I still found similar issues (although a bit fewer, since it meant the requests taking between 30 to 60s were ok now).

Also, it probably doesn't have much to do with payload size. (i.e. have noticed smaller payloads taking a lot longer, and then succeeding almost instantly on retry)

Potential places to look for solutions:

• Tweaking the default cycle, maybe we're calling the insertAll endpoint too frequently?
• Existing client library issues / server issues.
• This doesn't inspire confidence: https://www.codementor.io/tips/0817274533/google-bigquery-slow-streaming-inserts-performance