Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Allow IOErrors when attempting to retrieve default client encoding. #21531

Merged
merged 23 commits into from
Sep 19, 2018
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6c2987d
BUG: Except IOErrors when attempting to retrieve default client encod…
Jun 18, 2018
c31b914
BUG: Allow IOErrors when attempting to retrieve client encoding. (#21…
JayOfferdahl Jun 20, 2018
510f4b1
Merge branch 'pending-branch' into get-client-encoding-on-mod-wsgi
JayOfferdahl Jun 20, 2018
1afda5f
Merge branch 'master' into get-client-encoding-on-mod-wsgi
JayOfferdahl Jun 20, 2018
b37cb75
Fixup tests to add more scenarios. Code review changes.
JayOfferdahl Jun 20, 2018
0b1ca1d
Merge branch 'get-client-encoding-on-mod-wsgi' of https://github.com/…
JayOfferdahl Jun 20, 2018
bf26adb
PEP8 changes.
JayOfferdahl Jun 20, 2018
77a95d2
Mock out stdout/stdin instead of trying to access their properties.
JayOfferdahl Jun 26, 2018
2598595
Merge branch 'master' into get-client-encoding-on-mod-wsgi
JayOfferdahl Jun 26, 2018
59e0993
Use context manager when monkeypatching standard library items.
JayOfferdahl Jul 13, 2018
8b51b34
Merge branch 'get-client-encoding-on-mod-wsgi' of https://github.com/…
JayOfferdahl Jul 13, 2018
984da5c
Merge branch 'master' into get-client-encoding-on-mod-wsgi
JayOfferdahl Jul 13, 2018
4790e10
Fix linter erros/code review changes.
JayOfferdahl Jul 14, 2018
6156825
Move fix to 0.24.0 release; simplify tests.
JayOfferdahl Jul 16, 2018
fb1dc2e
Attempt at trying to get side_effects right without using mock.
JayOfferdahl Jul 24, 2018
7cedfdc
Add TODO(py27) to test helper.
JayOfferdahl Jul 25, 2018
35c0b3b
Fix pylint error.
JayOfferdahl Jul 25, 2018
49ee7b5
Fix silly linting errors. Use pep8.
JayOfferdahl Jul 26, 2018
40b3588
Move raise_or_return to MockEncoding; clean up parameters.
JayOfferdahl Aug 4, 2018
5e3c8af
Merge branch 'master' into get-client-encoding-on-mod-wsgi
JayOfferdahl Aug 16, 2018
f485223
Merge branch 'master' into get-client-encoding-on-mod-wsgi
JayOfferdahl Aug 23, 2018
ab67024
Merge remote-tracking branch 'upstream/master' into get-client-encodi…
JayOfferdahl Sep 18, 2018
81547da
Merge branch 'get-client-encoding-on-mod-wsgi' of https://github.com/…
JayOfferdahl Sep 18, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.24.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -681,7 +681,7 @@ I/O
- :func:`read_html()` no longer ignores all-whitespace ``<tr>`` within ``<thead>`` when considering the ``skiprows`` and ``header`` arguments. Previously, users had to decrease their ``header`` and ``skiprows`` values on such tables to work around the issue. (:issue:`21641`)
- :func:`read_excel()` will correctly show the deprecation warning for previously deprecated ``sheetname`` (:issue:`17994`)
- :func:`read_csv()` will correctly parse timezone-aware datetimes (:issue:`22256`)
-
- Bug in :meth:`detect_client_encoding` where potential ``IOError`` goes unhandled when importing in a mod_wsgi process due to restricted access to stdout. (:issue:`21552`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move to 0.23.5? @jreback

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a regression? If not, I would probably stay with 0.24.0 unless there is a compelling reason to backport-patch this bug.


Plotting
^^^^^^^^
Expand Down
2 changes: 1 addition & 1 deletion pandas/io/formats/console.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ def detect_console_encoding():
encoding = None
try:
encoding = sys.stdout.encoding or sys.stdin.encoding
except AttributeError:
except (AttributeError, IOError):
pass

# try again for something better
Expand Down
74 changes: 74 additions & 0 deletions pandas/tests/io/formats/test_console.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
import pytest

from pandas.io.formats.console import detect_console_encoding


class MockEncoding(object): # TODO(py27): replace with mock
"""
Used to add a side effect when accessing the 'encoding' property. If the
side effect is a str in nature, the value will be returned. Otherwise, the
side effect should be an exception that will be raised.
"""
def __init__(self, encoding):
super(MockEncoding, self).__init__()
self.val = encoding

@property
def encoding(self):
return self.raise_or_return(self.val)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current LINT error is because you are missing a blank line here

@staticmethod
def raise_or_return(val):
if isinstance(val, str):
return val
else:
raise val


@pytest.mark.parametrize('empty,filled', [
['stdin', 'stdout'],
['stdout', 'stdin']
])
def test_detect_console_encoding_from_stdout_stdin(monkeypatch, empty, filled):
# Ensures that when sys.stdout.encoding or sys.stdin.encoding is used when
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can add a reference to the issue for this and the other new tests, i.e. # GH 21552

# they have values filled.
# GH 21552
with monkeypatch.context() as context:
context.setattr('sys.{}'.format(empty), MockEncoding(''))
context.setattr('sys.{}'.format(filled), MockEncoding(filled))
assert detect_console_encoding() == filled


@pytest.mark.parametrize('encoding', [
AttributeError,
IOError,
'ascii'
])
def test_detect_console_encoding_fallback_to_locale(monkeypatch, encoding):
# GH 21552
with monkeypatch.context() as context:
context.setattr('locale.getpreferredencoding', lambda: 'foo')
context.setattr('sys.stdout', MockEncoding(encoding))
assert detect_console_encoding() == 'foo'


@pytest.mark.parametrize('std,locale', [
['ascii', 'ascii'],
['ascii', Exception],
[AttributeError, 'ascii'],
[AttributeError, Exception],
[IOError, 'ascii'],
[IOError, Exception]
])
def test_detect_console_encoding_fallback_to_default(monkeypatch, std, locale):
# When both the stdout/stdin encoding and locale preferred encoding checks
# fail (or return 'ascii', we should default to the sys default encoding.
# GH 21552
with monkeypatch.context() as context:
context.setattr(
'locale.getpreferredencoding',
lambda: MockEncoding.raise_or_return(locale)
)
context.setattr('sys.stdout', MockEncoding(std))
context.setattr('sys.getdefaultencoding', lambda: 'sysDefaultEncoding')
assert detect_console_encoding() == 'sysDefaultEncoding'