Gatsby-source-wordpress not able to fetch image behind basic authenticate WordPress source

[hazuremon]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Upgrading to latest gatsby-source-wordpress 6.10.0 has led to a build error of unable to fetch image error, with WordPress source that is behind basic authentication password protection.

Build is fine once it's downgraded back to gatsby-source-wordpress 6.9.1

Error message:
UNHANDLED REJECTION Unable to fetch:
[Image URL]

Reason: Response code 401 (Unauthorized)

Reproduction Link

Any basic auth protected WordPress

Steps to Reproduce

1. With latest gatsby-source-wordpress 6.10.0 installed, run a build from a basic authenticated protected wordpress source

Expected Result

gatsby-source-wordpress MediaItem to fetch images and media without error

Actual Result

UNHANDLED REJECTION Unable to fetch:
[Image URL]

Reason: Response code 401 (Unauthorized)

Fetch details:
{
"attempt": 3,
"method": "GET",
"errorCode": "ERR_NON_2XX_3XX_RESPONSE",
"responseStatusCode": 401,
"responseStatusMessage": "Unauthorized",
"requestHeaders": {
"user-agent": "got (https://github.com/sindresorhus/got)",
"accept-encoding": "gzip, deflate, br"
},
"responseHeaders": {
"server": "nginx",
"date": "Wed, 16 Mar 2022 21:55:14 GMT",
"content-type": "text/html",
"content-length": "172",
"connection": "close",
"www-authenticate": "Basic realm="XXXXXXXX""
}
}

Environment

gatsby-source-wordpress 6.10.0
basic auth protected WordPress source

Config Flags

No response

[LekoArts]

Hi!

Sorry to hear you're running into an issue. To help us best begin debugging the underlying cause, it is incredibly helpful if you're able to create a minimal reproduction. This is a simplified example of the issue that makes it clear and obvious what the issue is and how we can begin to debug it.

If you're up for it, we'd very much appreciate if you could provide a minimal reproduction and we'll be able to take another look.

Thanks for using Gatsby! 💜

[hazuremon]

Hi @LekoArts, while I can create a minimal reproduction of the build, what's the best way to include WordPress source and basic authentication requirement to reproduce this bug?

[vannyle]

I have the same issue after upgrading gatsby-source-wordpress to the latest version 6.10.0.

[miraclemaker]

I'm having a similar issue. I'm patching gatsby-source-wordpress to include an Authorization header, it's working fine in versions up to 6.9.0, in 6.9.1 and 6.10.0 it's not working with the same issue given by OP. GraphQL is fetched ok but images are not.

I thought the "update dependency node-fetch to ^2.6.7 for gatsby-source-wordpress" change in 6.7.0 could have been responsible since that is stripping authorization headers under certain circumstances but that is working fine.

[TylerBarnes]

Hey everyone, a reproduction of this will go a long way. I tried to reproduce this recently and wasn't able to so it's possible it's something specific to only some sites.

[hazuremon]

Hi @TylerBarnes,

I have replicated the official gatsby WordPress starter, and tried it with a fresh WP install on WPEngine behind htaccess password protection. Add a placeholder image to the hello-world post and came across with the following error.

https://github.com/hazuremon/gatsby-starter-wordpress
Add wordpress source and htaccess username and password to replicate the error

I have test that it built successfully before adding in password protection, and also built successfully by running gatsby-source-wordpress: 6.9.1 with htaccess password.

The difference I notice is towards image processing step where before 6.10.0, it is using gatsby-plugin-sharp for image process where it switched to gatsby.image_cdn on the newest version.

Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs
Running gatsby.IMAGE_CDN jobs

---

UNHANDLED REJECTION Unable to fetch:
https://[URL]/wp-content/uploads/2022/03/1920x1080.png

Reason: Response code 401 (Unauthorized)

Fetch details:
{
"attempt": 3,
"method": "GET",
"errorCode": "ERR_NON_2XX_3XX_RESPONSE",
"responseStatusCode": 401,
"responseStatusMessage": "Unauthorized",
"requestHeaders": {
"user-agent": "got (https://github.com/sindresorhus/got)",
"accept-encoding": "gzip, deflate, br"
},
"responseHeaders": {
"server": "nginx",
"date": "Tue, 22 Mar 2022 19:17:47 GMT",
"content-type": "text/html",
"content-length": "172",
"connection": "close",
"www-authenticate": "Basic realm="username""
}
}

[WebGuyJeff]

+1 on this error.

I'm running Nginx which is handling all auth, in contrast to WP handling anything or any Apache .ht dir level stuff.

This affects my current project so I can test.

[hazuremon]

Hi @TylerBarnes,

I have replicated the official gatsby WordPress starter, and tried it with a fresh WP install on WPEngine behind htaccess password protection. Add a placeholder image to the hello-world post and came across with the following error.

https://github.com/hazuremon/gatsby-starter-wordpress Add wordpress source and htaccess username and password to replicate the error

I have test that it built successfully before adding in password protection, and also built successfully by running gatsby-source-wordpress: 6.9.1 with htaccess password.

The difference I notice is towards image processing step where before 6.10.0, it is using gatsby-plugin-sharp for image process where it switched to gatsby.image_cdn on the newest version.

Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs Running gatsby.IMAGE_CDN jobs

UNHANDLED REJECTION Unable to fetch: https://[URL]/wp-content/uploads/2022/03/1920x1080.png

Reason: Response code 401 (Unauthorized)

Fetch details: { "attempt": 3, "method": "GET", "errorCode": "ERR_NON_2XX_3XX_RESPONSE", "responseStatusCode": 401, "responseStatusMessage": "Unauthorized", "requestHeaders": { "user-agent": "got (https://github.com/sindresorhus/got)", "accept-encoding": "gzip, deflate, br" }, "responseHeaders": { "server": "nginx", "date": "Tue, 22 Mar 2022 19:17:47 GMT", "content-type": "text/html", "content-length": "172", "connection": "close", "www-authenticate": "Basic realm="username"" } }

Hi @LekoArts, will you be able to connect to a wordpress source with basic authentication to test the minimal built? Otherwise, what will be the best way to provide these information?

[rburgst]

Ok, I am fairly certain that this commit broke it: fe29c01

Previously the files were downloaded like this adding the auth headers

https://github.com/gatsbyjs/gatsby/blob/gatsby-source-wordpress%406.9.0/packages/gatsby-source-wordpress/src/steps/source-nodes/create-nodes/create-remote-file-node/index.js#L191

now the files appear to be downloaded via gatsbyImageResolver

fe29c01#diff-1505effea6d8cc135677ea127439e5777ee2a7e397447f647a3ee3ec797d4920R596

which appears not to accept auth headers.

[rburgst]

@wardpeet @TylerBarnes as both of you were involved in this change and it affects all wordpress users with a htaccess file, I would appreciate a timely response.

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[rburgst]

This is still a problem and no one in gatsby seems to care. Very frustrating.

[KyleAMathews]

@rburgst it seems you've figured out the issue? Could you prepare a PR with a fix?

[rburgst]

Call me old fashioned, but I feel a bit offended if Gatsby itself broke the feature and then rely on the community to pick up the pieces. Its not that this is difficult to reproduce, just grab a new wordpress docker container, add the following .htaccess

# BEGIN WordPress
# Die Anweisungen (Zeilen) zwischen „BEGIN WordPress“ und „END WordPress“ sind
# dynamisch generiert und sollten nur über WordPress-Filter geändert werden.
# Alle Änderungen an den Anweisungen zwischen diesen Markierungen werden überschrieben.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress



AuthType Basic
AuthName "Wordpress"
AuthUserFile  /var/www/html/.htpasswd
order deny,allow
allow from all
Require valid-user

and the following .htpasswd:

user:$apr1$/W2HUfVZ$kvS1czEa58r6rixyfkQdR/

This will give you a wordpress with basic authentication of user:user.

Judging from my past experience trying to get PRs through on the gatsby project, it was a quite slow experience (see, #31652, #28093). I am not sure I have the energy in the coming weeks to attempt something like this again.

[TylerBarnes]

@rburgst @KyleAMathews the integrations team at Gatsby has this bug scheduled as part of our current 2 week sprint. I'll keep yall updated as we make progress.

[KyleAMathews]

@rburgst I apologize you've had a poor experience before writing PRs. That's not our goal. I'll take responsible for getting the remark one in tomorrow. I hope you'll give contributing a go again in the future and have a better experience.

[veryspry]

Hi @rburgst 👋

As you pointed out, the issue is with gatsbyImageResolver which I was able to confirm with this new code.

Unfortunately though, this isn't suitable for production since it would expose credentials directly in image urls which can get logged, etc in hosting. Before I get to a proposed solution, it's important to mention that Gatsby uses gatsbyImageResolver for Image CDN support That said, there are two things needed to fix this:

1. Make gatsbyImageResolver work with Image CDN disabled.
2. Make gatsbyImageResolver work with Image CDN enabled.

The first part is the simplest part and I am planning to implement it this week so that you and other users with WP instances behind http basic auth can get unblocked. The second one is a little trickier and with our other work in flight will probably get started within the next couple weeks.

I'll tag you here when I have a PR and or release out for the first fix and then try and keep you updated about the second one too. Hope this helps!

[rburgst]

I am mainly interested in variant 1 as we currently dont use gatsby image cdn. However, I dont understand, why you cannot simply pass the auth headers to gatsbyImageResolver

[TylerBarnes]

@rburgst you're right that that's what # 1 will entail. Because of that it should be easier to implement than # 2 where we need to figure out where to store auth headers.

[veryspry]

@rburgst What @TylerBarnes said!

While we could just pass the auth as args to gatsbyImageResolver, this resolver function gets used elsewhere "automagically" and not always explicitly like it is in the HTML replace function you pointed out. We need to make sure this fix covers the following cases:

1. when calling the function explicitly in code - like this example
2. when the resolver is called when querying the gatsbyImage field on nodes where this field is automatically attached. For instance, when running this query against the gatsby-source-graphql plugin:
   
3. when other CMS' / source plugins need to use basic auth for image downloads AND making sure that multiple source plugins needing to pass the auth headers can work side by side

That said, if we just pass the auth header data as args to gatsbyImageData, then this will only fix certain cases for gatsby-source-wordpress and will still not work correctly in other cases (e.g. the query from the screenshot above)

[veryspry]

@rburgst Update: we are planning on implementing a fix to make this work when Image CDN is disabled within the next two weeks.

[rburgst]

@veryspry will this also fix #35626 by any chance?

[veryspry]

@veryspry will this also fix #35626 by any chance?

@rburgst I haven't followed the reproduction steps in that one yet - but I'd assume that it isn't trying to fetch images behind basic auth since it's using this WP demo instance: https://github.com/gatsbyjs/gatsby-starter-wordpress-blog/blob/main/gatsby-config.js#L31. That said, I don't think this will fix that other issue unfortunately.

[MyWay]

Any news on this?

[TylerBarnes]

There is indeed! That PR is close to being merged. If you need a fix today you can try these canary versions:

yarn add gatsby@4.15.0-alpha-wp-image-cdn-auth.46 gatsby-source-wordpress@6.15.0-alpha-wp-image-cdn-auth.46

Make sure the versions are exact in your package.json (without a ^ before the version).

Thanks for asking!

EDIT: Looks like those canaries aren't working properly. I'm looking into it now, possibly I published them wrong
EDIT2: Fixed! I updated the versions above with working versions

[joernroeder]

very likely related #34387

[TylerBarnes]

@joernroeder looks related indeed. The original issue was separate but was resolved, the new issue is the same as this one.

[veryspry]

@rburgst: canaries published with a fix for this: #35151 (comment)

The PR that fixes this should be merged soon and then go out with the next stable Gatsby release too

[hazuremon]

For those who landed on this issue in the future, there are some breaking changes with upgrading to GatsbyCDN, it requires an update on graphql fields and settings in gatsby-config.js. You can find out more on the following page: https://support.gatsbyjs.com/hc/en-us/articles/4522338898579