Reduce excess file system writes by implementing query queue #3237
Conversation
There are multiple ways that running queries can be triggered. Previous to this PR, Gatsby had done some de-duping of queries but in practice, queries for pages/layouts can often be running multiple times in short succession. This is normally fine-ish but @Gaeron was running into troubles while editing markdown on reactjs.org where webpack would error when it detected a change in a file and before it could read it, Gatsby would start writing to the file again before webpack could finish reading it resulting in a JSON.parse error due to the incomplete JSON. On top of this, Gatsby would write out the result of every query if even the query returned the same result as previously. To address these this PR adds a query queue that deduplicates added pages/layouts. This means we're writing out results far less often. Second we hash each query result and only write to file if the result has changed which means far less work for webpack and far faster hot reloading of query updates to the browser.
ghost
assigned 
|
Deploy preview ready! Built with commit 785b97a |
|
/cc @gaearon |
|
@gaearon I upgraded reactjs.org to 1.9.138 and wasn't able to trigger an error regardless of how fast I saved files so hopefully this problem is gone for good 👍 |
|
Nice, thanks! I'll check on Monday. |
|
awesome this might address some of the issues in v1 (and more so in v2) where webpack is always recompiling, especially as the app starts |
|
@jquense that'd be awesome! What's the status of v2? Back full-time on Gatsby development for the foreseeable future so planning to finish the new testing infrastructure early next week and then jump into helping you with v2. |
|
Should be in reasonable beta quality. There is a webpack v3 PR that just needs validation on the example sites but is limited by the way we install gatsby in them (dev-cli doesn't install updates to node_modules) |
|
sweet!
With the new test infrastructure we can build something custom for this. Hoping to land #3242 + your PRs by end of next week before the holidays. Monday I'll get the roadmap up to date. |
|
haha woah! And this does stop (finally!) the initial double compile when gatsby develop starts!!!
|
|
@KyleAMathews im having a lot of troubles with this "fix"... i assume it was only a problem during development current // page-writer.js
const writeAndMove = (file, data) => {
const destination = joinPath(program.directory, `.cache`, file)
const tmp = `${destination}.${Date.now()}`
return fs
.writeFile(tmp, data)
.then(() => fs.move(tmp, destination, { overwrite: true }))
}works but unconditionally // page-writer.js
const writeAndMove = (file, data) => {
const destination = joinPath(program.directory, `.cache`, file)
const tmp = `${destination}.${Date.now()}`
return fs
.writeFile(destination, data)
}i dont know where can i get the information if running in development or production. is it |
|
@digitalkaoz Can you create a new issue for this? If you can add more info on the errors you're seeing that'll help too! |
There are multiple ways that running queries can be triggered. Previous
to this PR, Gatsby had done some de-duping of queries but in practice,
queries for pages/layouts can often be running multiple times in short
succession. This is normally fine-ish but @Gaeron was running into
troubles while editing markdown on reactjs.org where webpack would error
when it detected a change in a file and before it could read it, Gatsby
would start writing to the file again before webpack could finish
reading it resulting in a JSON.parse error due to the incomplete JSON.
On top of this, Gatsby would write out the result of every query if even
the query returned the same result as previously.
To address these this PR adds a query queue that deduplicates
added pages/layouts. This means we're writing out results far less
often.
Second we hash each query result and only write to file if the result
has changed which means far less work for webpack and far faster hot
reloading of query updates to the browser.