-
Notifications
You must be signed in to change notification settings - Fork 10.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(gatsby-source-drupal): Use the collection count from JSON:API extras to enable parallel API requests for cold builds #32883
Conversation
…tras to construct URLs Otherwise, we have to wait to start querying each page until the previous one finishes. This change lets us query all pages in parallel. So instead of fetching one collection page at a time, we can fetch up to the maximum concurrency (default 20). For a test site with ~3200 entities, this PR dropped sourcing time from ~14s to 4s.
@KyleAMathews - should we include an addition to the readme to help people discover this update? ie - push people to enable the proper JSON API Extras settings? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've asked a few questions here
Yeah definitely. I'll add it once we do some more real-world testing to verify this change. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great to me! Just one small comment nit.
…tras to enable parallel API requests for cold builds (#32883) * feat(gatsby-source-drupal): Use the collection count from JSON:API extras to construct URLs Otherwise, we have to wait to start querying each page until the previous one finishes. This change lets us query all pages in parallel. So instead of fetching one collection page at a time, we can fetch up to the maximum concurrency (default 20). For a test site with ~3200 entities, this PR dropped sourcing time from ~14s to 4s. * use new browser-based URL parser * Comment code more * Use the page size the site has set instead of assuming 50 * Use the original type that's set as that's always there * Log out updates while sourcing * Encourage people to enable this setting in the README * Update gatsby-node.js
Otherwise, we have to wait to start querying each page until the previous one finishes. This change lets us query all collection pages in parallel. So instead of fetching one collection page at a time, we can fetch up to the maximum concurrency (default 20).
For a test site with ~3200 entities and a warm Drupal cache (and no CDN cache), this PR dropped sourcing time from ~14s to 4s.
On a very large production Drupal site (~600k entities). Fetching time for a cold build dropped from 2 hours to 30 minutes 🚀
TODOs