Issue 403 when connecting to page with redirects

[MarcoPrins]

I'm getting a 403 forbidden with mechanize when visiting this page, however the same page displays fine when using RestClient
What is the issue here? Seems like mechanize is not handling redirects properly?

To replicate, please try the same url as I used below; It's a public url

Version 2.10.1 and 2.8.5 (tried both)

  def agent
    @agent ||= begin
      agent = Mechanize.new
      agent.user_agent = user_agent
      agent
    end
  end

  def user_agent
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) ' \
      'Chrome/92.0.4515.159 Safari/537.36'
  end

  # This does not work
  agent.get('https://right-to-work.service.gov.uk/rtw-view')

  # This works
  RestClient.get('https://right-to-work.service.gov.uk/rtw-view')

[flavorjones]

@MarcoPrins Interesting. I'm not sure I'm going to be able to fully debug this, but let me equip you with some debugging tools so you can explore on your own if you like.

When I visit this URL using curl and telling it to follow all redirects and use cookies:

curl --junk-session-cookies -b cookies.txt --location-trusted -v \
  -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15" \
  https://right-to-work.service.gov.uk/rtw-view

In the output I see the following responses:

• 302 to /rtw-view?hof-cookie-check
• 302 to /rtw-view/checker-details
• 302 to https://user-auth.apply-to-visit-or-stay-in-the-uk.homeoffice.gov.uk/auth/realms/ExternalChecker/protocol/openid-connect/auth?response_type=code&scope=openid%20email%20profile&state=e4fa6275616126a1dc89991043ceba1d&client_id=external-checker-rtwr&nonce=bcf0050c939c136486d96f309fbdd4f3&redirect_uri=https%3A%2F%2Fright-to-work.service.gov.uk%2Frtw-view%2Fchecker-details%2Fredirection
• 200 OK

When I run with Mechanize using the "Windows Chrome" user agent alias:

#! /usr/bin/env ruby

require "bundler/inline"

gemfile do
  source "https://rubygems.org"
  gem "mechanize", path: "."
end

url = "https://right-to-work.service.gov.uk/rtw-view"

agent = Mechanize.new
agent.user_agent_alias = "Windows Chrome"
agent.log = Logger.new(STDOUT)

page = agent.get(url)

I get:

• 302 to /rtw-view?hof-cookie-check
• 302 to /rtw-view/checker-details
• 302 to https://user-auth.apply-to-visit-or-stay-in-the-uk.homeoffice.gov.uk/auth/realms/ExternalChecker/protocol/openid-connect/auth?response_type=code&scope=openid%20email%20profile&state=867cfbb18daf0f20289af396617dc789&client_id=external-checker-rtwr&nonce=16e0c93f4b2f4b7f4f50d3a794874531&redirect_uri=https%3A%2F%2Fright-to-work.service.gov.uk%2Frtw-view%2Fchecker-details%2Fredirection
• 403 Forbidden

This looks like the same sequence of requests and responses, so I'm not sure why the server is responding with a 403. If there's a bug in Mechanize here I would love your help trying to isolate it!

[flavorjones]

It's certainly possible that Mechanize has a bug here, if RestClient and curl are both behaving OK. But I lack the time right now to dig into the details of the logs and figure out what the behavior difference is that's causing the issue.

[tomasz-jasiulek]

I have tested the failing link (the last one) using:

• curl ✔️ (returned 200)
• Net::HTTP::Get ✔️ (returned 200)
• mechanize ➖ (failed with 403)

And observed that mechanize uses Net::HTTP::Get under the hood, so I have started comparing the headers and other elements to find the differences.

It has turned out that this particular header when left to be set by mechanize is causing all those issues. Removing it manually before each request in that service resulted in getting 200 status codes instead of 403. Also it is worth reading that this header is considered obsolete and should not be used https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Charset

So in the end all you have to do is:

agent.request_headers = {
  'Accept-Charset' => nil
}

before making any requests.

@flavorjones not sure why mechanize sets this header by default, maybe it shouldn't as stated here https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Charset

[flavorjones]

@tomasz-jasiulek Thank you for fully debugging this!

not sure why mechanize sets this header

The Accept-Charset header has been sent by Mechanize since 63d4e15 in 2007, so I think the generous assumption should simply be that the code pre-dates the deprecation. Note that as late as 2012 Chrome was still sending this header.

I'll open an issue to take inventory of all the headers Mechanize is sending and remove any that are deprecated including accept-charset. I expect to have to do some surgery on the tests to make it happen.

[flavorjones]

See #647

[flavorjones]

Fixed in https://github.com/sparklemotion/mechanize/releases/tag/v2.11.0