Reading log from stdin hits error on str.decode() in Python 3

[twblamer]

Expected behavior

According to the docs and code, mlogvis is designed to read from stdin. However, doing so seems to hit code in logfile.py that works in Python 2, but fails in Python 3.

Actual/current behavior

$ mlogfilter mtools/test/logfiles/wiredtiger.log  | mlogvis
Traceback (most recent call last):
  File "/home/tblamer/repos/mtools/.venv/mtools/bin/mlogvis", line 11, in <module>
    load_entry_point('mtools', 'console_scripts', 'mlogvis')()
  File "/home/tblamer/repos/mtools/mtools/mlogvis/mlogvis.py", line 114, in main
    tool.run()
  File "/home/tblamer/repos/mtools/mtools/mlogvis/mlogvis.py", line 87, in run
    json_docs = self._export(True)
  File "/home/tblamer/repos/mtools/mtools/mlogvis/mlogvis.py", line 45, in _export
    for line_no, logevent in enumerate(self.args['logfile']):
  File "/home/tblamer/repos/mtools/mtools/util/logfile.py", line 248, in __iter__
    le = self.next()
  File "/home/tblamer/repos/mtools/mtools/util/logfile.py", line 215, in next
    line = line.decode('utf-8', 'replace')
AttributeError: 'str' object has no attribute 'decode'

Steps to reproduce the actual/current behavior

cd mtools
mkdir -p .venv
python3 -m venv .venv/mtools-py3
.venv/mtools-py3/bin/activate
pip3 install -r requirements.txt
python3 setup.py develop

mlogfilter mtools/test/logfiles/wiredtiger.log  | mlogvis

Environment

Software	Version
mtools	v1.5.1
MongoDB server	3.4.7
Operating system	Fedora 25

[stennie]

@twblamer Thanks for the bug report! Do you have the same issue piping output to mplotqueries?

Does this occur with all log files? Would you be able to provide a sample snippet of log file to test with?

Thanks,
Stennie

[twblamer]

It happens with mplotqueries too.

I think it happens with any log file. You can use mtools/test/logfiles/wiredtiger.log in the mtools repo as an example, or any of the other log files in that directory.

[stennie]

@twblamer Are you still experiencing this issue or were you able to find a solution?

I wasn't able to trivially reproduce this with Python 3.6 and there haven't been any subsequent bug reports.

If you are still interested in helping investigate, can you provide the output of:

• mlogfilter --version (which includes the Python interpreter version used by the script)
• pip3 list
• python3 -m locale

Thanks,
Stennie

[twblamer]

@stennie, I tried on a fresh install of mtools from pip and I still see the same issue. This is using the python3 package in Fedora 28.

$ virtualenv-3.6 test

$ source test/bin/activate

$ pip3 install mtools
...
Successfully installed mtools-1.5.3 python-dateutil-2.7.0 six-1.12.0

$ cd $HOME/repos/mtools
$ mlogfilter mtools/test/logfiles/wiredtiger.log | mlogvis
Traceback (most recent call last):
  File "/tmp/test/bin/mlogvis", line 10, in <module>
    sys.exit(main())
  File "/tmp/test/lib/python3.6/site-packages/mtools/mlogvis/mlogvis.py", line 114, in main
    tool.run()
  File "/tmp/test/lib/python3.6/site-packages/mtools/mlogvis/mlogvis.py", line 87, in run
    json_docs = self._export(True)
  File "/tmp/test/lib/python3.6/site-packages/mtools/mlogvis/mlogvis.py", line 45, in _export
    for line_no, logevent in enumerate(self.args['logfile']):
  File "/tmp/test/lib/python3.6/site-packages/mtools/util/logfile.py", line 248, in __iter__
    le = self.next()
  File "/tmp/test/lib/python3.6/site-packages/mtools/util/logfile.py", line 215, in next
    line = line.decode('utf-8', 'replace')
AttributeError: 'str' object has no attribute 'decode'

$ mlogfilter --version
mtools version 1.5.3 || Python 3.6.5 (default, Mar 29 2018, 18:20:46) [GCC
8.0.1 20180317 (Red Hat 8.0.1-0.19)]

$ mlogvis --version
mtools version 1.5.3 || Python 3.6.5 (default, Mar 29 2018, 18:20:46) [GCC
8.0.1 20180317 (Red Hat 8.0.1-0.19)]

$ pip3 list
Package         Version
--------------- -------
mtools          1.5.3
pip             19.0.3
python-dateutil 2.7.0
setuptools      40.8.0
six             1.12.0
wheel           0.33.1


$ python3 -m locale
Locale aliasing:

Locale defaults as determined by getdefaultlocale():
------------------------------------------------------------------------
Language:  en_US
Encoding:  UTF-8

Locale settings on startup:
------------------------------------------------------------------------
LC_CTYPE ...
   Language:  en_US
   Encoding:  UTF-8

LC_TIME ...
   Language:  (undefined)
   Encoding:  (undefined)

LC_COLLATE ...
   Language:  (undefined)
   Encoding:  (undefined)

LC_MONETARY ...
   Language:  (undefined)
   Encoding:  (undefined)

LC_MESSAGES ...
   Language:  (undefined)
   Encoding:  (undefined)

LC_NUMERIC ...
   Language:  (undefined)
   Encoding:  (undefined)


Locale settings after calling resetlocale():
------------------------------------------------------------------------
LC_CTYPE ...
   Language:  en_US
   Encoding:  UTF-8

LC_TIME ...
   Language:  en_US
   Encoding:  UTF-8

LC_COLLATE ...
   Language:  en_US
   Encoding:  UTF-8

LC_MONETARY ...
   Language:  en_US
   Encoding:  UTF-8

LC_MESSAGES ...
   Language:  en_US
   Encoding:  UTF-8

LC_NUMERIC ...
   Language:  en_US
   Encoding:  UTF-8


Locale settings after calling setlocale(LC_ALL, ""):
------------------------------------------------------------------------
LC_CTYPE ...
   Language:  en_US
   Encoding:  UTF-8

LC_TIME ...
   Language:  en_US
   Encoding:  UTF-8

LC_COLLATE ...
   Language:  en_US
   Encoding:  UTF-8

LC_MONETARY ...
   Language:  en_US
   Encoding:  UTF-8

LC_MESSAGES ...
   Language:  en_US
   Encoding:  UTF-8

LC_NUMERIC ...
   Language:  en_US
   Encoding:  UTF-8


Number formatting:

123,456,789 is 123456789
3.14 is 3.14