Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-91539: improve performance of get_proxies_environment #91566

Merged
merged 26 commits into from
Oct 5, 2022
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
42118e3
improve performance of get_proxies_environment when there are many en…
eendebakpt Apr 15, 2022
37ff8d9
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 15, 2022
b8de962
fix case of short env name
eendebakpt Apr 15, 2022
0f90945
Merge branch 'performance/getproxies_environment' of githubeendebakpt…
eendebakpt Apr 15, 2022
336da07
fix formatting
eendebakpt Apr 15, 2022
cabeb47
fix whitespace
eendebakpt Apr 15, 2022
c581748
whitespace
eendebakpt Apr 15, 2022
4217396
Merge branch 'main' into performance/getproxies_environment
eendebakpt May 11, 2022
0055604
Merge branch 'main' into performance/getproxies_environment
eendebakpt May 13, 2022
869fa9a
Merge branch 'main' into performance/getproxies_environment
eendebakpt May 17, 2022
dba0414
Update Lib/urllib/request.py
eendebakpt May 18, 2022
76f16ca
Update Lib/urllib/request.py
eendebakpt May 18, 2022
d3feb07
Update Lib/urllib/request.py
eendebakpt May 18, 2022
b9ab037
Update Lib/urllib/request.py
eendebakpt May 18, 2022
f961505
whitespace
eendebakpt May 18, 2022
36e5497
Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7W…
eendebakpt May 18, 2022
ca4fbd4
Update Lib/urllib/request.py
eendebakpt May 18, 2022
77cceab
Merge branch 'main' into performance/getproxies_environment
eendebakpt May 31, 2022
9687ebb
Merge branch 'main' into performance/getproxies_environment
eendebakpt Jun 12, 2022
3bf6f63
Merge branch 'main' into performance/getproxies_environment
eendebakpt Jun 24, 2022
fc36faa
Merge branch 'main' into performance/getproxies_environment
eendebakpt Jul 2, 2022
b6d866f
Merge branch 'main' into performance/getproxies_environment
eendebakpt Jul 16, 2022
f40e0dc
Merge branch 'main' into performance/getproxies_environment
eendebakpt Jul 26, 2022
41410da
Merge branch 'main' into performance/getproxies_environment
eendebakpt Aug 2, 2022
41b72c6
Merge branch 'main' into performance/getproxies_environment
eendebakpt Sep 11, 2022
9db1103
Merge branch 'main' into performance/getproxies_environment
eendebakpt Sep 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions Lib/urllib/request.py
Original file line number Diff line number Diff line change
Expand Up @@ -2497,28 +2497,34 @@ def getproxies_environment():
this seems to be the standard convention. If you need a
different way, you can pass a proxies dictionary to the
[Fancy]URLopener constructor.

"""
proxies = {}
# in order to prefer lowercase variables, process environment in
# two passes: first matches any, second pass matches lowercase only
for name, value in os.environ.items():
name = name.lower()
if value and name[-6:] == '_proxy':
proxies[name[:-6]] = value

# select only environment variables which end in (after making lowercase) _proxy
proxies = {}
environment = []
for name in os.environ.keys():
# fast screen underscore position before more expensive case-folding
if len(name) > 5 and name[-6] == "_" and name[-5:].lower() == "proxy":
value = os.environ[name]
proxy_name = name[:-6].lower()
environment.append((name, value, proxy_name))
if value:
proxies[proxy_name] = value
# CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
# (non-all-lowercase) as it may be set from the web server by a "Proxy:"
# header from the client
# If "proxy" is lowercase, it will still be used thanks to the next block
if 'REQUEST_METHOD' in os.environ:
proxies.pop('http', None)
for name, value in os.environ.items():
for name, value, proxy_name in environment:
# not case-folded, checking here for lower-case env vars only
if name[-6:] == '_proxy':
eendebakpt marked this conversation as resolved.
Show resolved Hide resolved
name = name.lower()
if value:
proxies[name[:-6]] = value
proxies[proxy_name] = value
else:
proxies.pop(name[:-6], None)
proxies.pop(proxy_name, None)
return proxies

def proxy_bypass_environment(host, proxies=None):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve performance of ``urllib.request.getproxies_environment`` when there are many environment variables