feat(gatsby-source-wordpress): support multiple instances of plugin #38119

ascorbic · 2023-05-26T17:02:51Z

Description

Currently the WordPress plugin can't support multiple instances because it uses a global redux store. This PR instead scopes the store according to the type prefix. It uses AsyncLocalStorage to ensure that each api call uses the correct store. It additionally scopes the cache, by including the prefix in the cache key.

Documentation

Tests

Related Issues

ascorbic · 2023-05-26T17:03:32Z

packages/gatsby-source-wordpress/package.json

-    "@rematch/core": "^1.4.0",
-    "@rematch/immer": "^1.2.0",
+    "@rematch/core": "^2.2.0",
+    "@rematch/immer": "^2.1.3",


The old version of rematch had completely broken TS types

nice! it did indeed. I looked at upgrading around a year ago and went through the whole process only to discover rematch was no longer working and I didn't have more time to figure out why 🤦 hopefully you fared better here!

good thing is, while they're odd, the tests are fairly comprehensive, so if there's any problem it'll definitely show up in the tests

ascorbic · 2023-05-26T17:04:41Z

packages/gatsby-source-wordpress/src/store.ts

The good stuff is in here

ascorbic · 2023-05-26T17:05:28Z

packages/gatsby-source-wordpress/src/store.ts

+    )
+  }
+
+const store = (): Store => asyncLocalStorage.getStore()


We change store from an exported store to a function which returns the correct store for the context

ascorbic · 2023-05-26T17:43:39Z

packages/gatsby-source-wordpress/src/models/develop.ts

+  effects: () => {
+    return {}
+  },


There's a bug in the Rematch types that mean that if there are no effects, it infers the type of reducers incorrectly, so we need to add all these empty effects

ascorbic · 2023-05-26T17:45:44Z

packages/gatsby-source-wordpress/src/store.ts

+    if (!STORE_MAP.has(typePrefix)) {
+      STORE_MAP.set(
+        typePrefix,
+        init({
+          models,
+          plugins: [immerPlugin<IRootModel>()],
+        })
+      )
+    }
+
+    const store = STORE_MAP.get(typePrefix)


We get the instance of the store, keyed by the plugin's type prefix

This is great, I love this implementation. async storage is perfect for this.
Looking at this block now I'm wondering if we should throw an error if there are multiple instances that have the same prefix. What do you think? Having 2 instances with the same prefix will break most things in whichever of those instances runs first

otherwise I think we could possibly get at the plugin instance ID and use that to index each store. In that case multiple instances with the same prefix will work and all their nodes will be mixed together

That's a good idea. I think using the instance ID as the key means we could also check that there if there's a prefix collision

ascorbic · 2023-05-26T17:46:14Z

packages/gatsby-source-wordpress/src/store.ts

+    const store = STORE_MAP.get(typePrefix)
+
+    return asyncLocalStorage.run(store, async () =>
+      hook(helpers, pluginOptions)


Run the hook using AsyncLocalStorage, passing the correct store instance

ascorbic · 2023-05-26T17:47:13Z

packages/gatsby-source-wordpress/src/utils/run-steps.ts

+const runApiSteps = (steps: Array<Step>, apiName: string): IGatsbyApiHook =>
+  wrapApiHook(
+    async (
+      helpers: GatsbyNodeApiHelpers,
+      pluginOptions: IPluginOptions
+    ): Promise<void> => runSteps(steps, helpers, pluginOptions, apiName)
+  )


We wrap the function in wrapApiHook, which adds the handling for AsyncLocalStorage

ascorbic · 2023-06-06T14:46:39Z

packages/gatsby-source-wordpress/src/gatsby-node.ts

 import * as steps from "./steps"

-const pluginInitApiName = findApiName(`onPluginInit`)


As this only checks for one method name it's a noop, so can be removed

ascorbic · 2023-06-06T14:49:12Z

packages/gatsby-source-wordpress/src/steps/create-schema-customization/helpers.js

-  const previousTypeDefinitions = await cache.get(`previousTypeDefinitions`)
+  const previousTypeDefsKey = withPluginKey(`previousTypeDefinitions`)
+
+  const previousTypeDefinitions = await cache.get(previousTypeDefsKey)


Include the plugin type prefix in the cache key

ascorbic · 2023-06-07T11:15:30Z

packages/gatsby/src/redux/__tests__/jobsv2.js

-    expect(jobsManager.enqueueJob).toMatchSnapshot()
+    expect(jobsManager.enqueueJob).toHaveBeenCalledWith({


This test was failing because it was snapshotting a Promise object, and using AsyncLocalStorage in other tests meant that there were added internal fields on global Promise. The object it's snapshotting is a jest mock, so it makes more sense to use the actual mock assertions instead of snapshotting its internal state

TylerBarnes · 2023-06-07T15:14:23Z

integration-tests/gatsby-source-wordpress/__tests__/index.js

-    })()
-  })
+  beforeAll(
+    withGlobalStore(done => {


I think the tests and test utils shouldn't need withGlobalStore since they're int tests that are closer to e2e tests. gatsby develop is run in a subprocess and then tests make assertions against queries to the gql server so any store scoping will happen inside that subprocess.
I could be missing something of course, did this fix an issue?

Oh is it cause the tests weirdly use fetchGraphQL from within source-wordpress?

Yeah, exactly. I've added it in places where we're doing deep imports from the plugin

Maybe it would be better to refactor out those imports 🤔

definitely, I did that for content engine in another PR. I can follow up after this PR and bring my changes over from there

ascorbic · 2023-06-07T16:06:29Z

...ages/gatsby-source-wordpress/src/steps/create-schema-customization/transform-fields/index.js

-    field.type.name = `JSON`
+    return {
+      ...field,
+      type: new GraphQLScalarType({
+        name: `JSON`,
+      }),
+    }


field.type, type.name etc are now immutable in graphql.js, so these were throwing errors in tests.

ascorbic · 2023-06-08T10:44:14Z

integration-tests/gatsby-source-wordpress/test-fns/test-utils/graphql.js

+/**
+ *
+ * @param {Object} options
+ * @param {string} options.url
+ * @param {string} options.query
+ * @param {Object} options.variables
+ * @param {Object} options.headers
+ *
+ * @returns {Promise<Object>}
+ */
+exports.fetchGraphql = async ({ url, query, variables = {}, headers = {} }) => {


Same signature as the one we were previously importing from the plugin

ascorbic · 2023-06-08T10:46:49Z

packages/gatsby-source-wordpress/src/steps/create-schema-customization/helpers.js

+export const buildTypeName = (name, prefix) => {
  if (!name || typeof name !== `string`) {
    return null
  }

-  const {
-    schema: { typePrefix: prefix },
-  } = getPluginOptions()
+  if (!prefix) {
+    prefix = getPluginOptions().schema.typePrefix
+  }


Allowing the prefix to be passed in rather than needing to call getPluginOptions means we can remove the need to use the store at query time.

ascorbic · 2023-06-08T10:48:27Z

packages/gatsby-source-wordpress/src/steps/create-schema-customization/build-types.js

    typeDef.resolveType = node =>
-      node?.__typename ? buildTypeName(node.__typename) : null
+      node?.__typename ? buildTypeName(node.__typename, prefix) : null
  }


During develop, this resolver is called outside of the async scope of the createSchemaCustomization api, so we'll instead pass the prefix in from this scope

ascorbic · 2023-06-08T12:34:47Z

@TylerBarnes do you know what would cause this failure?

https://app.circleci.com/pipelines/github/gatsbyjs/gatsby/90667/workflows/6d61235c-2dfb-48de-9f2c-134c74b52ccd/jobs/1095663?invite=true#step-109-5999

TylerBarnes · 2023-06-08T23:30:14Z

@ascorbic for the yoast test serializes to the same string makes me think somehow the Gatsby resolvers are returning a different order than WPGraphQL for seo.breadcrumbs. Maybe we should switch it for .toMatchObject or something?

Though it also looks like a bunch of the Gatsby schema differences in the second failed test are for Yoast types 🤔 so maybe there's a change in here somewhere that's unexpectedly changing how the schema is generated?

ascorbic · 2023-06-09T09:05:18Z

@TylerBarnes Thanks. The Yoast one was fixed by 58ba99e but I've had less luck with the other test. I'm not sure that one is Yoast-related, because some of the types are unrelated. Can you explain a bit more about the test? What would have changed the schema?

TylerBarnes · 2023-06-09T17:26:30Z

@ascorbic unfortunately as I'm sure you've experienced the codebase for this plugin is complicated to say the least.
It's hard to say what specifically would cause this but I know small changes to how it runs can end up changing the schema drastically.
I ran the tests locally on main and they passed, then I switched to this branch and re-ran and they're failing so it must be some change in here.
It's probably easiest to make a debug branch and then bring over pieces of this PR one commit at a time to figure out what's causing the tests to fail

ascorbic · 2023-06-12T13:26:09Z

@TylerBarnes I think I've solved it! 29dd03e

TylerBarnes

oh heck yes! 🕺 nice work 🙌

ascorbic · 2023-06-13T07:18:52Z

Oh, and thanks for the comprehensive tests that caught this, @TylerBarnes !

ThyNameIsMud · 2023-08-14T22:18:42Z

@ascorbic @TylerBarnes I'm having some async issues after the upgrade to 7.11.0 of gatsby-source-wordpress. I resolved this after downgrading to 7.10.1, and based on the error, this update is the cause.

Error: Store not found

store.ts:56 getStore
[cree-component-library]/[gatsby-source-wordpress]/src/store.ts:56:11

helpers.js:31 getQueryInfoByTypeName
[cree-component-library]/[gatsby-source-wordpress]/src/steps/source-nodes/helpers.js:31:35

transform-object.js:70 resolver
[cree-component-library]/[gatsby-source-wordpress]/src/steps/create-schema-customization/transform-fields/transform-object.js:70:45

resolvers.ts:751 wrappedTracingResolver
[cree-component-library]/[gatsby]/src/schema/resolvers.ts:751:20

async Promise.all

graphql-runner.ts:255 GraphQLRunner.query
[cree-component-library]/[gatsby]/src/query/graphql-runner.ts:255:14

query-runner.ts:140 queryRunner
[cree-component-library]/[gatsby]/src/query/query-runner.ts:140:14

This causes the worker to exit without finishing. Would you happen to have any insights on what's going on here, purhapes a way to debug to understand why it's losing the context of the store?

cavemon · 2024-01-20T00:41:23Z

@ascorbic @TylerBarnes I'm having some async issues after the upgrade to 7.11.0 of gatsby-source-wordpress. I resolved this after downgrading to 7.10.1, and based on the error, this update is the cause.

Error: Store not found

store.ts:56 getStore
[cree-component-library]/[gatsby-source-wordpress]/src/store.ts:56:11

helpers.js:31 getQueryInfoByTypeName
[cree-component-library]/[gatsby-source-wordpress]/src/steps/source-nodes/helpers.js:31:35

transform-object.js:70 resolver
[cree-component-library]/[gatsby-source-wordpress]/src/steps/create-schema-customization/transform-fields/transform-object.js:70:45

resolvers.ts:751 wrappedTracingResolver
[cree-component-library]/[gatsby]/src/schema/resolvers.ts:751:20

async Promise.all

graphql-runner.ts:255 GraphQLRunner.query
[cree-component-library]/[gatsby]/src/query/graphql-runner.ts:255:14

query-runner.ts:140 queryRunner
[cree-component-library]/[gatsby]/src/query/query-runner.ts:140:14

This causes the worker to exit without finishing. Would you happen to have any insights on what's going on here, purhapes a way to debug to understand why it's losing the context of the store?

@ThyNameIsMud Were you able to fix this? I've been running into this issue for months as I'm trying to upgrade from v3 to v5.

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label May 26, 2023

ascorbic commented May 26, 2023

View reviewed changes

packages/gatsby-source-wordpress/src/store.ts

Copy link

Contributor Author

ascorbic May 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The good stuff is in here

ascorbic commented May 26, 2023

View reviewed changes

LekoArts added topic: source-wordpress Related to Gatsby's integration with WordPress topic: source-plugins Relates to the Gatsby source plugins (e.g. -filesystem) and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels May 30, 2023

ascorbic force-pushed the wp-store-context branch from def75f1 to 6e2b9b0 Compare June 6, 2023 12:11

ascorbic commented Jun 6, 2023

View reviewed changes

ascorbic changed the title ~~wip(gatsby-source-wordpress): scope redux store per plugin instance~~ feat(gatsby-source-wordpress): support multiple instances of plugin Jun 7, 2023

ascorbic marked this pull request as ready for review June 7, 2023 10:34

ascorbic commented Jun 7, 2023

View reviewed changes

TylerBarnes reviewed Jun 7, 2023

View reviewed changes

ascorbic commented Jun 7, 2023

View reviewed changes

ascorbic force-pushed the wp-store-context branch from 677f3c9 to d880c0a Compare June 8, 2023 10:34

ascorbic commented Jun 8, 2023

View reviewed changes

ascorbic added 3 commits June 12, 2023 14:25

wip(gatsby-source-wordpress): scope redux store per plugin instance

403ab15

fix: segregate caches too

c8675f2

fix: correct var name

cd7ab4d

ascorbic added 12 commits June 12, 2023 14:25

chore: refresh lockfile

91fe6d2

fix: typos and console errs

40b68b1

chore: add als support to tests

e21d798

chore: scope als store to test suite

47140cd

chore: fix test

dba2307

chore: add store to integration test

dbb17e5

chore: use own graphql client

1bb174f

chore: remove dep on store at query time

7dce814

chore: pass the prefix in more places

feebdaa

chore: constently pass pluginOptions

381e40b

chore: change matcher

9c49d82

fix: types are plain objects

29dd03e

ascorbic force-pushed the wp-store-context branch from fccc74c to 29dd03e Compare June 12, 2023 13:26

ascorbic requested a review from TylerBarnes June 12, 2023 14:14

TylerBarnes approved these changes Jun 12, 2023

View reviewed changes

LekoArts merged commit 77e4e19 into gatsbyjs:master Jun 13, 2023

ascorbic deleted the wp-store-context branch June 13, 2023 07:12

X-oss-byte mentioned this pull request Jul 14, 2023

[Snyk] Upgrade gatsby from 5.0.0 to 5.11.0 X-oss-byte/gatsby-1#2

Open

X-oss-byte mentioned this pull request Aug 12, 2023

[Snyk] Upgrade gatsby-plugin-image from 3.0.0 to 3.11.0 X-oss-byte/gatsby-1#11

Open

Rutam21 mentioned this pull request Nov 27, 2023

[Snyk] Fix for 21 vulnerabilities Rutam21/gatsby#1428

Open

MaxMood96 mentioned this pull request Nov 27, 2023

[Snyk] Fix for 21 vulnerabilities MaxMood96/gatsby#1198

Open

nidhi42 mentioned this pull request Dec 19, 2023

[Snyk] Fix for 30 vulnerabilities nidhi42/gatsby#2844

Open

monizb mentioned this pull request Dec 19, 2023

[Snyk] Fix for 4 vulnerabilities monizb/gatsby#919

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gatsby-source-wordpress): support multiple instances of plugin #38119

feat(gatsby-source-wordpress): support multiple instances of plugin #38119

ascorbic commented May 26, 2023 •

edited

Loading

ascorbic May 26, 2023

TylerBarnes May 26, 2023 •

edited

Loading

TylerBarnes May 26, 2023

ascorbic May 26, 2023

ascorbic May 26, 2023

ascorbic May 26, 2023

ascorbic May 26, 2023

TylerBarnes May 26, 2023

TylerBarnes May 26, 2023 •

edited

Loading

ascorbic May 26, 2023

ascorbic May 26, 2023

ascorbic May 26, 2023

ascorbic Jun 6, 2023

ascorbic Jun 6, 2023

ascorbic Jun 7, 2023

TylerBarnes Jun 7, 2023

TylerBarnes Jun 7, 2023

ascorbic Jun 7, 2023

ascorbic Jun 7, 2023

TylerBarnes Jun 7, 2023

ascorbic Jun 7, 2023

ascorbic Jun 8, 2023

ascorbic Jun 8, 2023 •

edited

Loading

ascorbic Jun 8, 2023

ascorbic commented Jun 8, 2023

TylerBarnes commented Jun 8, 2023 •

edited

Loading

ascorbic commented Jun 9, 2023

TylerBarnes commented Jun 9, 2023

ascorbic commented Jun 12, 2023 •

edited

Loading

TylerBarnes left a comment

ascorbic commented Jun 13, 2023

ThyNameIsMud commented Aug 14, 2023

cavemon commented Jan 20, 2024

		import * as steps from "./steps"

		const pluginInitApiName = findApiName(`onPluginInit`)

		expect(jobsManager.enqueueJob).toMatchSnapshot()
		expect(jobsManager.enqueueJob).toHaveBeenCalledWith({

feat(gatsby-source-wordpress): support multiple instances of plugin #38119

feat(gatsby-source-wordpress): support multiple instances of plugin #38119

Conversation

ascorbic commented May 26, 2023 • edited Loading

Description

Documentation

Tests

Related Issues

Choose a reason for hiding this comment

TylerBarnes May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerBarnes May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ascorbic Jun 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ascorbic commented Jun 8, 2023

TylerBarnes commented Jun 8, 2023 • edited Loading

ascorbic commented Jun 9, 2023

TylerBarnes commented Jun 9, 2023

ascorbic commented Jun 12, 2023 • edited Loading

TylerBarnes left a comment

Choose a reason for hiding this comment

ascorbic commented Jun 13, 2023

ThyNameIsMud commented Aug 14, 2023

cavemon commented Jan 20, 2024

ascorbic commented May 26, 2023 •

edited

Loading

TylerBarnes May 26, 2023 •

edited

Loading

TylerBarnes May 26, 2023 •

edited

Loading

ascorbic Jun 8, 2023 •

edited

Loading

TylerBarnes commented Jun 8, 2023 •

edited

Loading

ascorbic commented Jun 12, 2023 •

edited

Loading