Request blocked or forbidden

[wKovacs64]

haveibeenpwned.com recently (mid-January, 2019) adjusted their API abuse prevention policies and now many API consumers are being blocked. The primary target appears to be the User-Agent string in the API request, but there is more to the story as multiple users with the same UA can receive different results for the same request (e.g. one blocked, one succeeds). IP address and/or region may also be playing a part. It also seems the rules are most strict surrounding the breachedaccount endpoint.

The requirements in the haveibeenpwned API documentation state that a User-Agent must be included in all API requests to avoid being blocked. hibp does this this outside of the browser (i.e. Node.js), but it is not currently possible for hibp to set its own User-Agent when running in the browser because haveibeenpwned.com does not include it in the Access-Control-Allow-Headers header field of a preflight response. (Try it out on test-cors.org and you will see it fail. See MDN's CORS page for more details.) Furthermore, some browsers (e.g. Chrome) will even block requests client-side that attempt to modify the UA as they consider it unsafe, so the problem would still exist for some users even if the CORS rules were adjusted to allow it.

Currently, it seems like most/all browser UA strings are being blocked outright. The test link in the API documentation fails on every browser I've tested from various devices, locations, and networks. I don't know if this is intentional and browser support needs to be dropped from hibp or if the abuse prevention rules are too strict and need adjusted. Edit (2019-01-25): Confirmed with Troy that this is intentional.

---

In versions of hibp prior to 7.4.0, the error would appear as such:

Error: Forbidden - no user agent has been specified in the request.

This is because the haveibeenpwned.com API documentation states a 403 Forbidden response would occur in the case of a missing User-Agent header field in the request. However, this is not the only case where they respond with 403 Forbidden now, so that error message is no longer accurate.

hibp version 7.4.0 includes a more specific error message, indicating if you are being blocked by haveibeenpwned.com when a Cloudflare Ray ID is present in the response header:

Error: Request blocked, contact haveibeenpwned.com if this continues (Ray ID: xxxxxxxx-xyz)

If no Cloudflare Ray ID is present, you will get a more generalized error:

Error: Forbidden - access denied.

Related issues:

#38
wKovacs64/pwned#27
wKovacs64/pwned#30

[wKovacs64]

If anyone has any corrections, additional information, or suggestions, please comment here. I've been communicating with Troy Hunt of haveibeenpwned.com in an attempt to determine a way forward, but I've gotten nowhere.

[kierxn]

Any more updates on this?

I think we're just going to have to move away from the library all together as it seems absolutely fine if I use Fetch/Axios with a custom UA.

[wKovacs64]

Can you provide more details around exactly what you're doing that works? Are you running in Node.js or a browser?

I can add an option to allow setting your own UA, but that is only temporarily side-stepping the problem (you may get blocked again, requiring you to change the UA, which is actually against Troy's acceptable use policy) and would only currently work in Node.js environments (as stated multiple times).

[kierxn]

I wish I could provide more information other than the fact all I'm doing is setting a custom UA. The UA is specific to our product and Troy is aware as we've been in contact with him before regarding this issue and before we integrated it into our platform. If we set the UA, it works, if we don't or use the default in your library, it doesn't work.

I think the option to add your own UA is a useful feature as it's explained on his website that the UA should describe the service using it.

I wish I could on the browser side of things but I have no ideas other than discussing that with Troy?

[wKovacs64]

OK, so you're in Node.js. Which version of hibp are you using? Have you tried that same version from a different network/location? I suspect the blocks are a combination of UA and IP as the same version of hibp (i.e. same UA) works for some people but not everyone (the difference being which Cloudflare region they hit).

As for the browser, what do we do? As I see it, there are only 2 options: Troy relaxes the new rules to allow browser UA's again, or we drop browser support from the library - which would be a bummer. 🙁

[wKovacs64]

I just emailed Troy one more time in hopes of getting an answer with regards to the browser. We need to establish if browser UAs are intentionally and permanently blocked for the breachedaccount endpoint (meaning we drop browser support from hibp) or if he just has something misconfigured/overly strict. 🤞

[wKovacs64]

Troy responded and confirmed browser UAs are intentionally blocked. This policy seems to be currently applied the breachedaccount endpoint only. He has not clarified (yet) if the other endpoints will start blocking browser UAs or not.

For the time being, I suppose I will update hibp to throw an error if you attempt to call the breachedAccount function from within a browser and document that the upstream API has denied that particular action. If it turns out all the endpoints adopt this same policy, I will probably remove browser support from hibp entirely.

@kierxn If you can tell me which version of hibp you're using that gets blocked and confirm that you've tried that same version from multiple networks/locations and still got blocked, that would be quite helpful.

I'm tempted to just expose the Axios instance itself (or similar), so you can customize it however you wish (custom UA, proxy requests through your own server somewhere to get around the browser block, etc.) but that really locks us into that particular implementation detail. Meh. Edit: I'll just expose UA and baseUrl (for proxying) configuration options and leave it at that.

[wKovacs64]

I've exposed some additional options to help alleviate these problems (released in hibp@7.5.0). Closing the issue for now as there is probably nothing else we can do at the library level at this time. Feel free to comment if any new information becomes available.