Unicode : change df.to_string() and friends to always return unicode objects

[ghost]

closes #2225

Note: Although all the tests pass with minor fixes, this PR has an above-average chance of
breaking things for people who have relied on broken behaviour thus far.

df.tidy_repr combines several strings to produce a result. when one component is unicode
and other other is a non-ascii bytestring, it tries to convert the latter back to a unicode string
using the 'ascii' codec and fails.

I suggest that _get_repr -> to_string should always return unicode, as implemented by this PR,
and that the force_unicode argument be deprecated everyhwere.

The force_unicode argument in to_string conflates two things:

• which codec to use to decode the string (which can only be a hopeful guess)
• whether to return a unicode() object or str() object,

The first is now no longer necessary since pprint_thing already resorts to the same hack
of using utf-8 (with errors='replace') as a fallback.
I believe making the latter optional is wrong, precisely because it brings about situations
like the test case above.
to_string, like all internal functions , should utilize unicode objects, whenever feasible.

[wesm]

This seems pretty reasonable. Should I take a chance merging this for 0.9.1? I've encountered the bug you fixed here before

[ghost]

I would at least wait a few days before merging this (perhaps @jseabold or someone else would like
to argue their use-case ).

[wesm]

I guess the question is what code will break because the string is coming back as unicode. Obviously if you had df.to_string(force_unicode=True).decode('utf-8') that is going to break. Maybe this should be held off until 0.10 series

[ghost]

it depends whether you consider this a bug fix or a breaking change. I'm fine with 0.10 though.

[changhiskhan]

Let wait 'til 0.10. Let's merge it into master as soon as the release is out though.

[wesm]

Agreed...

[aldanor]

This would be great. As of right now, you have to do something dirty (at least that's the only way I found it works) like DataFrame(series).to_string(force_unicode=True, header=False) to correctly print a Series object with unicode characters to a utf-8 console.

[ghost]

I took this a step further, Realizing that the unicode issue really matters only
when we want to get a string representation of an object.

So:

• I Converted more related functions to work exclusively with unicode.
• Since everything should taper down to pprint_thing at the bottom, any utf-8 bytestrings
  should get silently decoded into unicode.
• If your data is not unicode and not utf-8, it's unreasonable to expect str(df) to do
  the right thing, and so you'll get � (the unicode replacement character), but not exceptions
  (hopefully).
• fixing a couple of corner cases along the way, I added all the boilerplae so that
  str(x)/unicode(x)/bytes(x) work on py2 and py3 for series/df/panel.

Yell if something broke.

[wesm]

@aldanor I see you deleted your comment but I checked that your example works now, at least on my environment...

[aldanor]

@wesm Thanks, sounds good. I just didn't want to confuse everyone cause I wasn't sure this wasn't something specific to my environment. I will try and test it again soon as I can.

[ghost]

Added str/unicode/bytes support for Index,MultiIndex.

[ghost]

takeback