Local images are null on gatsby build, but correct in dev mode (Gatsby v4)

[acidicX]

Hi,

I'm using the 2.7.0-next.1 version of this plugin and gatsby-plugin-image@2.5.0-next.2, but after upgrading to Gatsby v4 there localFile on all image assets that should be processed by gatsby-plugin-sharp is null when using gatsby build. I've set downloadLocalImages: true obviously. It mostly works fine when using gatsby develop though. But I'm also not getting any errors in the gatsby cli.

I dug around a bit and noticed three issues:

1. makeResolveGatsbyImageData returns an async function, but the type GraphQLFieldResolver is not defined as an async function - neither in gatsby-plugin-image, nor in graphql. So that could be a potential source of failure, especially since v4 now runs queries in parallel.
   1.1 code in question: https://github.com/GraphCMS/gatsby-source-graphcms/blob/228ad3711ac559ee9403d299df036d9507c2453d/gatsby-source-graphcms/src/gatsby-node.js#L358
   1.2. gatsby-plugin-image: https://github.com/gatsbyjs/gatsby/blob/gatsby-plugin-image%402.5.0-next.2/packages/gatsby-plugin-image/src/resolver-utils.ts#L64
   1.3. graphQL: https://github.com/graphql/graphql-js/blob/main/src/type/definition.ts#L962

2. Console cli shows the image sharp processing jobs during gatsby develop, but not during gatsby build:

success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 1.067s - 10/10 9.38/s
success Writing page-data.json files to public directory - 0.015s - 1/32 2199.60/s
[============================]   1.494 s 4/4 100% Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs

That might be another hint that something is async and works in develop because there the process is still alive, while it has already terminated on build.

3. The mime-type list in makeResolveGatsbyImageData does not include image/webp - is this correct? I've modified it to include webp because I use webp almost exclusively, but it does not seem to be the source of my problem.
   3.1 see https://github.com/GraphCMS/gatsby-source-graphcms/blob/228ad3711ac559ee9403d299df036d9507c2453d/gatsby-source-graphcms/src/gatsby-node.js#L363

I'm happy to provide more details if you need them :)

---

Gatsby stuff in package.json:

    "gatsby": "^4.4.0",
    "gatsby-plugin-canonical-urls": "^4.4.0",
    "gatsby-plugin-emotion": "^7.4.0",
    "gatsby-plugin-force-trailing-slashes": "^1.0.5",
    "gatsby-plugin-google-gtag": "^4.4.0",
    "gatsby-plugin-image": "^2.5.0-next.2",
    "gatsby-plugin-klaro": "^1.0.1",
    "gatsby-plugin-manifest": "^4.4.0",
    "gatsby-plugin-react-helmet": "^5.4.0",
    "gatsby-plugin-react-svg": "^3.1.0",
    "gatsby-plugin-robots-txt": "^1.6.14",
    "gatsby-plugin-sharp": "^4.4.00",
    "gatsby-plugin-sitemap": "^5.4.0",
    "gatsby-plugin-typescript": "^4.4.0",
    "gatsby-source-filesystem": "^4.4.0",
    "gatsby-source-graphcms": "^2.7.0-next.1",
    "gatsby-transformer-json": "^4.4.0",
    "gatsby-transformer-sharp": "^4.4.0",
    "graphql": "^15.5.1",

[notrab]

HI @acidicX can you try upgrading to 2.7.0-next.4 and see if this helps at all?

[acidicX]

Hi @notrab - just did, but nope, that's worse now - it's broken in Dev Mode as well. Now the localFile is null in develop as it is after build.

fyi: the peer depencency to gatsby-plugin-image is broken as well, so you currently need to use --legacy-peer-deps on install.

npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving: test@1.27.0
npm ERR! Found: gatsby-plugin-image@2.5.0-next.3
npm ERR! node_modules/gatsby-plugin-image
npm ERR!   gatsby-plugin-image@"^2.5.0-next.3" from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer gatsby-plugin-image@"^2.0.0-next" from gatsby-source-graphcms@2.7.0-next.4
npm ERR! node_modules/gatsby-source-graphcms
npm ERR!   gatsby-source-graphcms@"^2.7.0-next.4" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

[notrab]

Ok thanks, I’ll look into it a bit more next week when I’m back in the office. What I pushed is mostly unrelated anyways. I’ll update the dependencies too

[notrab]

@acidicX I'm not sure the root of the issue here. It would be worth us updating the dependencies, and going through the changelogs to see what can be done to resolve these issues.

I'm happy to merge in any PRs you want to submit if you are able to narrow the problem down further. I'll bring this up internally to try get more resources on it.

[notrab]

Can we follow the steps in this comment to see if it makes anything better or worse here? #209 (comment)

[acidicX]

@notrab well, updating deps is always a good idea - but as I stated some points in my initial post, I still think the async calls are in the wrong place in this plugin. I can't find any reference that something inside createResolvers can be async, and the function is not being marked as async as well in this plugin.

See https://www.gatsbyjs.com/docs/reference/config-files/gatsby-node/#createResolvers

In any case, even though you are passing an async function into getGatsbyImageResolver, it will not be awaited anywhere in the plugin image code you are using, thus it probably will not finish before gatsby has finished building.

If you check the tutorial on creating a source plugin, all the async work is done in other places, like sourceNodes or onCreateNodes - the tutorial even has a section "Optimize Remote Images" which handles this use case.

That said, it's just my digging around for a few minutes. I'm by no means an expert on gatsby plugins, but this plugin does things noticeably different than in the tutorial.

I'll try next.6 tomorrow, but I don't see how the changes in 665a5db would affect this bug. I'm happy to give it a try though :)

[notrab]

@acidicX I've not looked at the images stuff today at all, or gone down the rabbit hole yet on this fully. There's a few issues on v4, and some of our GraphCMS schema changes that can cause issues in other areas. If for example you change a rich text field to allow embeds, the fragments will need to be rebuilt, causing builds to fail as soon as that's done.

I'll try to look into this when I get more time. If you're able to submit a PR I would merge that for sure.

Thanks

[acidicX]

@notrab just tried next.6, same behaviour as before on build, but dev mode works again. Sadly I don't really have the time to open a PR with a solution, but as a hint I had a look at the gatsby-source-contentful plugin, which has the same functionality. They also download the images in the sourceNodes step, which can be async:

https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-contentful/src/source-nodes.js#L477

[notrab]

I'll try to find some time to look into this further.

Regarding webp, last time I checked, the resolveGatsbyImageData didn't support webp and only a small subset (as shown in the source).

Thanks!

[acidicX]

Blurred placeholders work with webp (which is what we use), but traced SVG does not (png only, also node-potrace seems to be unmaintained) - not sure if dominant color generation works. Funny though, even when image/webp is not in the list, the blurred placeholder generation still worked for us in v2 and v3.

[notrab]

I can’t remember the specifics, it’s been over a year since I last worked on this plugin. Maybe it was a limitation with us, who knows.

I’ll try to look into this further and get back to you in due course. Feel free to open a PR if you ever do get the time, thanks

[M4ss1ck]

I just updated my plugins

System:
    OS: Windows 10 10.0.19042
    CPU: (4) x64 Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz
  Binaries:
    Node: 16.13.0 - C:\Program Files\nodejs\node.EXE
    npm: 8.3.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.19041.423.0)
  npmPackages:
    gatsby: ^4.4.0 => 4.4.0
    gatsby-plugin-dark-mode: ^1.1.2 => 1.1.2
    gatsby-plugin-disqus: ^1.2.3 => 1.2.3
    gatsby-plugin-image: ^2.5.0-next.3 => 2.5.0-next.3
    gatsby-plugin-manifest: ^4.4.0 => 4.4.0
    gatsby-plugin-mdx: ^3.5.0-next.3 => 3.5.0-next.3
    gatsby-plugin-pnpm: ^1.2.10 => 1.2.10
    gatsby-plugin-postcss: ^5.4.0 => 5.4.0
    gatsby-plugin-react-helmet: ^5.4.0 => 5.4.0
    gatsby-plugin-react-svg: ^3.1.0 => 3.1.0
    gatsby-plugin-sharp: ^4.5.0-next.3 => 4.5.0-next.3
    gatsby-source-filesystem: ^4.4.0 => 4.4.0
    gatsby-source-graphcms: ^2.7.0-next.9 => 2.7.0-next.9
    gatsby-transformer-sharp: ^4.4.0 => 4.4.0

I'm now using gatsby-source-graphcms@2.7.0-next.9 as you can see. But I'm still receiving warnings about images.

...
success Writing page-data.json files to public directory - 0.011s - 9/9 818.57/s
warn [gatsby-plugin-image] Missing image prop
warn [gatsby-plugin-image] Missing image prop
warn [gatsby-plugin-image] Missing image prop
warn [gatsby-plugin-image] Missing image prop
warn [gatsby-plugin-image] Missing image prop
warn [gatsby-plugin-image] Missing image prop
success Building static HTML for pages - 4.031s - 9/9 2.23/s
...

And what is worse, my blogposts are empty now, and there are no error messages. More info about my repo in #213

[notrab]

Hey @M4ss1ck does the same error occur when using 2.7.0 (no next flag) in your project?

[M4ss1ck]

I updated my plugins, including gatsby-source-graphcms@2.7.0

Trying to discard random mistakes, I deleted node_modules, fragments and .cache folders and reinstalled.

Then I ran pnpm build and got the same warn [gatsby-plugin-image] Missing image prop warnings. However, this time it's failing to fetch allGraphCmsPost.nodes.content.markdownNode in my blog/index.js page but everything is fine with each blog post individually.

gatsby info:

System:
    OS: Windows 10 10.0.19042
    CPU: (4) x64 Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz
  Binaries:
    Node: 16.13.0 - C:\Program Files\nodejs\node.EXE
    npm: 8.3.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.19041.423.0)
  npmPackages:
    gatsby: ^4.5.3 => 4.5.3
    gatsby-plugin-dark-mode: ^1.1.2 => 1.1.2
    gatsby-plugin-disqus: ^1.2.3 => 1.2.3
    gatsby-plugin-image: ^2.5.2 => 2.5.2
    gatsby-plugin-manifest: ^4.5.2 => 4.5.2
    gatsby-plugin-mdx: ^3.5.2 => 3.5.2
    gatsby-plugin-pnpm: ^1.2.10 => 1.2.10
    gatsby-plugin-postcss: ^5.5.0 => 5.5.0
    gatsby-plugin-react-helmet: ^5.5.0 => 5.5.0
    gatsby-plugin-react-svg: ^3.1.0 => 3.1.0
    gatsby-plugin-sharp: ^4.5.2 => 4.5.2
    gatsby-source-filesystem: ^4.5.2 => 4.5.2
    gatsby-source-graphcms: ^2.7.0 => 2.7.0
    gatsby-transformer-sharp: ^4.5.0 => 4.5.0

repo: https://github.com/M4ss1ck/wasting-blog/tree/actualizar-dependencias

[notrab]

@M4ss1ck regarding the markdown node issue, are you able to provide an endpoint for your API so I can debug further?

We should discuss this in another issue perhaps.

[M4ss1ck]

@notrab my current GRAPHCMS_ENDPOINT is https://api-ca-central-1.graphcms.com/v2/ckqbyhdoci9cm01z1dt205z8t/master, I hope you find what's wrong and thank you for the quick response.

I don't know if you'll need a token too so lmk

We should discuss this in another issue perhaps.

Since this is so weird I guess I'm doing something wrong and perhaps there's nothing wrong with markdown nodes

[acidicX]

@notrab for the images part I created a fork of this plugin and enabled the functionality inside the demo:

https://github.com/acidicX/gatsby-source-graphcms

How To

1. clone the repo
2. cd demo && nvm use && npm i
3. npm run dev
4. You can now see the local images working on the product page (localFile object is filled and image is correctly served)
5. kill dev server
6. npm run clean
7. npm run build
8. npm run serve
9. You can now see that the localFile object is null (feature not working)

Hint in case you are wondering about the commented out markdown nodes: I had to disable them on the product page demo because as several others pointed out, they seem to be broken as well, but that is unrelated to this issue. I'm not using MDX in my setup.

[notrab]

@acidicX I believe you're in touch with our support who will help triaging this further. Let's try to keep comms in one place (here, or on the chat) so we've a log of everything going on.

I'm not too familiar with Gatsby these days, so it might be worth us trying to speak with them to get further insight on this. I did speak with Kyle Matthews briefly about this, but our implementation of the source plugin (using gatsby graphql toolkit) was different to what he has seen before.

Once local assets are enabled, the asset becomes a File, which itself doesn't have the resolver in the same way assets do. See https://github.com/GraphCMS/gatsby-source-graphcms/blob/main/gatsby-source-graphcms/src/gatsby-node.js#L460

I'm not sure if we need to add, or if the gatsby image plugin should automatically do this to the File type:

 File: {
      gatsbyImageData: {
        ...getGatsbyImageResolver(makeResolveGatsbyImageData(cache), args),
        type: 'JSON',
      },
    },

I've explored as much as I can with this to my limited knowledge. I'll keep looking, but I wouldn't rely on me saving the day here 😓

PS. I tried to run your demo but I run into an error about gatsby-source-filesystem not existing. I was able to follow the same steps in this plugins source code demo workspace though.

[notrab]

I've reached out to Gatsby. I'll let you know if I hear anything back.

[acidicX]

Let's try to keep comms in one place (here, or on the chat) so we've a log of everything going on.

@notrab Sure thing! All the info really is here though. I wasn't sure how actively this repo is monitored by graphcms and I just wanted to make sure that this is getting the proper attention as we are paying customers, so I reached out to the support team as well. Sorry for disturbing your holiday ;)

Once local assets are enabled, the asset becomes a File, which itself doesn't have the resolver in the same way assets do.

Afaik this is done via the the https://www.gatsbyjs.com/plugins/gatsby-transformer-sharp/ plugin, but I can be mistaken. As I said, I'm also no expert on gatsby plugins.

PS. I tried to run your demo but I run into an error about gatsby-source-filesystem not existing. I was able to follow the same steps in this plugins source code demo workspace though.

You probably need to create the assets folder in src, I added a .gitkeep so it should work now.

[notrab]

Thanks @acidicX

I think we potentially should have added a note to the repo saying it wasn't an "official" plugin (way back when), as it was formally created by the community, and then taken over by some Gatsby users internally.

I am trying to get this plugin now part of our official integrations team, so we can offer better support, and functionality. So I'm sure we'll see that happen this year. Gatsby doesn't come up very often with our customers, so it's probably why this project may feel quite stale.

I'll wait for Gatsby and get back to you. I've also linked them to this issue. I also went back to a very early version of the plugin, and this feature didn't seem to work then. So maybe I just picked a bad commit, or my yarn link is broken, but let's hope this can be resolved easily, and doesn't cause any breaking changes for anyone else 😅

Sorry for messing you about on this though. I'll keep you posted.

[acidicX]

I also went back to a very early version of the plugin, and this feature didn't seem to work then.

Well it did work quite well for me on Gatsby v2 and v3, so I still suspect the issue is with the async execution while creating the nodes/resolvers, because the main change in v4 is running queries in parallel.

[notrab]

@acidicX but how come File isn't adjusted to be aware of the gatsby image stuff, this is what I'm scratching my head at 😅

[acidicX]

@notrab If I understand it correctly, the current workflow is like this:

gatsby-source-graphcms -> Asset -> createRemoteFileNode (this adds localFile) -> gatsby-plugin-image (adds placeholder generation) -> gatsby-plugin-sharp (does the actual image processing) -> gatsby-transform-sharp (this adds all the output from the sharp plugin to the resolver)

I'm still unsure how gatsby-source-filesystem plays into that though.

[notrab]

@acidicX it's my understanding that Asset is replaced with the native File when downloadLocalImages: true is set. This is what seems to be throwing things off somewhere, maybe unrelated.

[notrab]

@acidicX can you try version v2.7.1-next.1 and report back?

I'm having more luck with this now than before, but we might not be there yet...

[sbuys]

@acidicX can you try version v2.7.1-next.1 and report back?

I'm having more luck with this now than before, but we might not be there yet...

@notrab tried v2.7.1-next.1 and it appears to have broken buildMarkdownNodes. All of our markdownNodes are now null.

[notrab]

@sbuys can I asked what happened with local assets though?

[sbuys]

@notrab well our images (local assets) are part of the markdown, so was not able to tell.

[notrab]

Can you share the query?

[acidicX]

Could we please try to keep this issue about the local images? I'm not sure the problems are related.

@notrab I've tested it and it seems to work for PNGs again now 🎉, could you please add webp to the list here?
297b99e#diff-8582651aaa3a1a09548e66fe368b79d26b6ae75d40d5e4cc4684baf998095d26R323

I've also updated my demo-fork, works there as well: https://github.com/acidicX/gatsby-source-graphcms

[notrab]

@acidicX great! Could you try v2.8.0-next.1?

[acidicX]

@notrab seems to be working 🎉 thanks for the fix!

[notrab]

Amazing! I'll release next into the main branch 🚀

The markdownNodes is unrelated (but the fix is mostly similar).

[sbuys]

Amazing! I'll release next into the main branch 🚀

The markdownNodes is unrelated (but the fix is mostly similar).

@notrab. Thanks for the image fix. Is a markdownNodes fix in the works? We're struggling to debug / understand if it's something we're doing or an issue with the plugin.

[notrab]

@sbuys can you open a new ticket with replicable steps & endpoint? Thanks