no_proxy setting ignores the proxies dictionary

[ofek]

These work:

$ python -c "import os,requests;os.environ['HTTPS_PROXY']='https://b.r.o.k.e.n.com';requests.get('https://google.com', proxies={'no_proxy': 'google.com'})"
$ python -c "import os,requests;os.environ['HTTPS_PROXY']='https://b.r.o.k.e.n.com';os.environ['NO_PROXY']='google.com';requests.get('https://google.com')"

These do not work:

$ python -c "import requests;requests.get('https://google.com', proxies={'https': 'https://b.r.o.k.e.n.com', 'no_proxy': 'google.com'})"
$ python -c "import os,requests;os.environ['NO_PROXY']='google.com';requests.get('https://google.com', proxies={'https': 'https://b.r.o.k.e.n.com'})"

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.5"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}

cc @Lukasa @sigmavirus24

[cocobear]

Many issue (#4871, #879, #4600 ) is relative to no_proxy, and this is definitely a bug, even someone provide some patched code, why it's still unresolved?

[AvitalL]

Please fix the issue as it causes a slightly bigger issue - the session.send() method is ignoring the os.environ proxy settings so there is not way to specify proxy settings other than passing proxies explicitly, which doesn't bypass and ignores no_proxy.

[florimondmanca]

A possible hint looking at the logic for "should we bypass proxies" in Requests, is that this logic is only checked on redirects…

requests/requests/sessions.py

Line 292 in 1b41763

bypass_proxy = should_bypass_proxies(url, no_proxy=no_proxy)

When I would expect it to also be called on each request to decide "should we use proxies for this URL". Eg in .get_adapter(), or whichever place is responsible for saying "use the connection pool, or a proxy":

requests/requests/sessions.py

Lines 718 to 730 in 1b41763

	def get_adapter(self, url):
	"""
	Returns the appropriate connection adapter for the given URL.
	:rtype: requests.adapters.BaseAdapter
	"""
	for (prefix, adapter) in self.adapters.items():
	if url.lower().startswith(prefix.lower()):
	return adapter
	# Nothing matches :-/
	raise InvalidSchema("No connection adapters were found for {!r}".format(url))

But I'm not 100% sure at which layer the proxies vs connection pool decision occur, eg if it's in the default HTTPAdapter or elsewhere (haven't dug that far into the code).

For what it's worth, the 2nd failing test case works as expected with HTTPX:

>>> import os, httpx
>>> os.environ['no_proxy'] = 'google.com'
>>> proxies = {'https': 'https://no.where.com'}
>>> c = httpx.Client(proxies=proxies)
>>> c.get('https://google.com')
<Response [200 OK]>

The 1st failing test case doesn't though, because no_proxy keys aren't taken into account yet by HTTPX (see encode/httpx#1043).