fix: use dagster retry mechanism

[Jabolol]

This PR updates failed queries to use Dagster's retry mechanism instead of failing silently, adjusts the data fetching to reduce nodes to 500 per query—since the Open Collective API returns 503 errors for larger queries—which seems sensible based on local testing and will require further triage, and changes the write disposition to "merge" to prevent duplicates in the dataset.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
kariba-network	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 23, 2024 10:06pm
oso-www	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 23, 2024 10:06pm

[oso-prs]

Auto-approved! please merge responsibly 😄

[oso-prs]

Test deployment unnecessary, no dbt files have been changed.

[Jabolol]

• Added a debounced retry mechanism to handle rate limiting and data retrieval failures.
• On encountering a 429 status, the mechanism waits 65 seconds for the rate limit to reset before retrying the request.
• For 503 errors (which occur when Open Collective fails to return a large dataset), the mechanism retries with progressively smaller payloads: first with 1000 nodes, then 500, 250, and finally 125.
• If all retry attempts fail, the task is marked as failed, allowing Dagster to retry it later.

[ravenac95]

@Jabolol I don't think this is an issue because we have concurrency limits but this doesn't currently handle issues with concurrent connections to open collective. I think that should be fine for now, but I am stating it as a sanity check because if multiple clients hit a 429 nearly simultaneously and the rate limits are by API key, I'm not sure if their api has issues with all of those clients retrying again at approximately the same time.