Unable to start AWS CLI with Unicode locale

[chillum]

~> aws --version
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.5.1/libexec/bin/aws", line 15, in <module>
    import awscli.clidriver
  File "/usr/local/Cellar/awscli/1.5.1/lib/python2.7/site-packages/awscli/clidriver.py", line 31, in <module>
    from awscli.help import ProviderHelpCommand
  File "/usr/local/Cellar/awscli/1.5.1/lib/python2.7/site-packages/awscli/help.py", line 20, in <module>
    from docutils.core import publish_string
  File "/usr/local/Cellar/awscli/1.5.1/libexec/lib/python2.7/site-packages/docutils-0.12-py2.7.egg/docutils/core.py", line 20, in <module>
    from docutils import frontend, io, utils, readers, writers
  File "/usr/local/Cellar/awscli/1.5.1/libexec/lib/python2.7/site-packages/docutils-0.12-py2.7.egg/docutils/frontend.py", line 41, in <module>
    import docutils.utils
  File "/usr/local/Cellar/awscli/1.5.1/libexec/lib/python2.7/site-packages/docutils-0.12-py2.7.egg/docutils/utils/__init__.py", line 20, in <module>
    import docutils.io
  File "/usr/local/Cellar/awscli/1.5.1/libexec/lib/python2.7/site-packages/docutils-0.12-py2.7.egg/docutils/io.py", line 18, in <module>
    from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput
  File "/usr/local/Cellar/awscli/1.5.1/libexec/lib/python2.7/site-packages/docutils-0.12-py2.7.egg/docutils/utils/error_reporting.py", line 47, in <module>
    locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 496, in getdefaultlocale
    return _parse_localename(localename)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 428, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8
~> locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
~> uname -mrsv
Darwin 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64
~> python --version
Python 2.7.2

When I unset the LC_CTYPE, it works:

~> env LC_CTYPE= aws --version 
aws-cli/1.5.1 Python/2.7.2 Darwin/12.5.0

Other Python apps work fine with this LC_CTYPE (checked hg and ansible running on the same Python on the same host).

I believe, this LC_CTYPE variable is set by MacOS, I didn't set it manually.

[lonormaly]

+1

For any chefs out there, that recipe saved me:

script "awscli_fix" do
interpreter "bash"
user "root"
code <<-EOH
echo "export LC_CTYPE= " >> /etc/profile.d/awscli_fix.sh
chmod +x /etc/profile.d/awscli_fix.sh
EOH
end

[jamesls]

Marking as a bug for now.

I think this will depend on whether or not "UTF-8" is a valid value for LC_CTYPE. If it is, then this is something we should support. However, if UTF-8 is not a valid value then I think we should just require that people have correct values for LC_CTYPE. However, I think a better error message would help.

I'm on a mac, and the locale output shows LC_CTYPE set to en_us.UTF-8.

[itskingori]

Dunno if this helps but I added this to my .bashrc ...

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

Output of locale is ...

LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Also on a Mac.

[jamesls]

Was able to find more information on this. Seems to be a bug in python. We'll need to figure out what, if anything, we can do to workaround this issue: http://bugs.python.org/issue18378

[jamesls]

So after looking into the bug I've linked above as well as the traceback in the initial post, this happens literally on the first line of code in bin/aws which is where we import our main module awscli.clidriver. The only way I think see to work around this is to check if LC_CTYPE=utf-8 before we even import anything and if so, change it to something that will work with getdefaultlocale(), say en_us.UTF-8.

Thoughts? cc @kyleknap @danielgtaylor

[kyleknap]

I think using the way you proposed with an if statement should work fine. I was also thinking we do a try, except for ValueError to cover any other incompatible LC_CTYPE's. However, that may cover up other errors in the future, if importing the driver causes a ValueError but with a different reason/message.

Also how do you propose to change the LC_CTYPE before importing the driver?

[jamesls]

If it was a more specific exception I think the try/except would be ok, but I think ValueError is just too generic to know for sure it's because of this issue.

Also how do you propose to change the LC_CTYPE before importing the driver?

Exactly that. Check and change before importing it:

# bin/aws
import sys

### We'd add this here ###
import os
if os.environ.get('LC_CTYPE', '') == 'UTF-8':
    os.environ['LC_CTYPE'] = 'en_US.UTF-8"
import awscli.clidriver


def main():
    return awscli.clidriver.main()


if __name__ == '__main__':
    sys.exit(main())

The python bug report suggests possibly only doing this check for Mac OS X but I'm not sure that's necessary.

[kyleknap]

That's how I would do it too. I think that should be safe. I am not sure if it will affect calls like sys.getdefaultencoding(), but should not matter because we are keeping it as UTF-8.

[jamesls]

Fixed via #1121