Allow custom authentication (in particular NTLM) to proxies

[hickford]

Requests can do many kinds of authentication—basic, digest, OAuth1. Even better, the great API allows users to specify a custom authentication class

import requests
from requests_ntlm import HttpNtlmAuth

 requests.get("http://ntlm_protected_site.com",auth=HttpNtlmAuth('domain\\username','password'))

However for proxies, the only authentication presently supported is Basic. The API doesn't allow you to specify other or custom classes. http://docs.python-requests.org/en/latest/user/advanced/#proxies

To use HTTP Basic Auth with your proxy, use the http://user:password@host/ syntax:

Please could you expand the API to allow custom authentication for proxies too?

This would be mega useful. NTLM proxies are common in Windows corporate networks. Python development can't get off the ground in my office because the package manager (and everything else) falls over at the proxy. Examples

• Add support for NTLM proxies pypa/pip#1182
• Doesn't work behind proxy in corporate Windows network (NTLM) httpie/cli#159

[hickford]

It's not obvious what the syntax should look like. This is the current API (string based)

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://user:pass10.10.1.10:1080",  # basic authentication
}
requests.get("http://example.org", proxies=proxies)

Will want to preserve that, but also allow user to specify an HtmlAuth object if they want to.

How about, define a new Proxy class, which is permissible as a value in the proxies dictionary

Proxy(url="http://10.10.1.10:3128", auth=HttpNtlmAuth('domain\\username','password'))

[hickford]

Does proxying happen in requests or does this need a fix upstream in https://github.com/shazow/urllib3 first?

Edit: Urllib3 only does Basic auth. Asked there if there's any plan to support custom auth. I don't know if it would be possible since AuthBase is in Requests. urllib3/urllib3#242

[Lukasa]

This is a good idea @matt-hickford. I see three ways of doing this:

1. Use a Proxy class in the Proxies dictionary. Definitely do-able, but adds quite a bit of complication into the API. Possible. +0
2. Use proxy-type Auth handlers. Not good. -1
3. Use Transport Adapters.

I'm generally in favour of TAs. In particular, Transport Adapters are really the only thing that knows anything about HTTPS over proxy (the CONNECT verb), which needs to be differently handled from other kinds of messages. To that end, it seems more natural to provide proxy authentication solutions at the Transport Adapter level.

Potentially TAs could take pluggable auth modules, just like individual requests? /cc @sigmavirus24

[sigmavirus24]

I would think Transport Adapters would take Proxy Adapters which handle using proxies in much the same way Transport Adapters handle the protocols (even though by default we only handle two). That said, Proxy Adapters could contain all the logic for pluggable auth "modules". This would then factor out the Proxy logic from the Transport Adapters and make things far more testable as well and allow people to configure crazy proxies however they like.

[Lukasa]

I don't think we want to be adding a further Transport Adapter layer. Best to just say 'a Transport Adapter knows what to do'.

[Lukasa]

Until urllib3 has this as an option there's not much we can do, so I'll close this until that time.

[hickford]

Cool I've offered a bounty on the upstream issue urllib3/urllib3#242

You could label this 'blocked' or something

[ryandebruyn]

Hello gents. I apologize to wake the sleeping giant. However Ive been trying to track down a viable workaround to request https: with NTLM proxy authentication and have come up short. @Lukasa et al I appreciate all the work to date. As of what is currently available it doesn't look like this is supported in urllib3, requests-ntlm or request. What does one do? Any suggestions?

[Lukasa]

I believe requests-ntlm does have support for this in the code: have you tried it?

[ryandebruyn]

Thanks for the reply. I have tried requests-ntlm if i have my creds correct this is what I get.

import request
from requests_ntlm import HttpNtlmAuth
session = requests.Sessions()
session.auth = HttpNtlmAuth("domain\\myuser","mypassword")
session.get("https://httpbin.org/get")

ConnectionError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /get (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))

[ryandebruyn]

@Lukasa. From what I can tell requests-ntlm will let you do NTLM auth with a proxy when using HTTP, but not with HTTPS. Am I missing something? Thanks again for your time.

[Lukasa]

Ah, yes, @ryandebruyn that's correct. Unfortunately, httplib makes it very difficult to authenticate to proxies when setting up a TLS tunnel because for any non-200 response to the CONNECT it will throw an exception and lose the response data. We can in principle work around that but it's extremely difficult to do and potentially breaks quite a few behaviours.

Sadly, this is just something that is very, very difficult to do.