ENH Prefer requests over urllib2

[hayd]

At the moment we use urllib2 for http requests, this branch prefers to use requests if it's installed. For one thing it means you can read gzipped json, which is more difficult in urllib2:

url = 'https://api.stackexchange.com/2.1/search?page=1&pagesize=10&order=desc&sort=activity&tagged=pandas&site=stackoverflow'
result = pd.read_json(url)  # in master this raises ValueError: Expected object or value

cc #3804 @jreback

Thoughts?

[hayd]

Probably needs some testing with and without requests installed (quite a few io tests skip for me travis).

[hayd]

As discussed, could put this as a helper function, also I seem to have missed out the bytes_to_str which is presumably there for a reason...

[jreback]

maybe a mention in install.rst of the requests module as an optional dep

[cpcloud]

might want to note also that read_html hasn't been tested with requests as well...

[hayd]

I had a go with html, but got some weird lxml error... (removed them in 2nd commit).

[cpcloud]

u have to catch Exception since you must be able to catch exceptions that are not due to invalid HTML and propagate them, even when lxml is not installed, the ValueError clause u changed probably caused the XMLSyntaxError to not be caught

[cpcloud]

it's a bit annoying since i actually have to call isinstance(e, XMLSyntaxError) but only when i know that lxml is installed (since that's where that is defined) thus all of the kludge around parsing...

[hayd]

Will have another play later in the week.

I actually had a similar thing in the request, checking for exceptions which the user may not have installed (I hid it behind an ImportError: pass). I definitely got into a muddle on that particular Exception...

[cpcloud]

i think there needs to be a mother-of-all-read-functions in io.common since there seems to be lots of repetition of the function with very tiny variations...i have an issue somewhere

[jreback]

yep...including the options NOT to read (e.g. I think html and read_csv want to read line-by-line), most others (e.g. json), io.data, etc. just need the buffer already read

[cpcloud]

html reads the buffer or punts the file-like object to the backend if possible. i think csv is the only iterable one so far

[hayd]

Do I need to worry about testing in Travis both with and without the requests module?

I'm concerned this might have some weird issues either way so push to 0.12?

[cpcloud]

there's the FULL_DEPS variable...check out the build matrix on a past run i don't remember which runs it's used on.

[jreback]

you could add to all if u want
update ci/install.sh
and ci/print_versions

[jreback]

see also #2426

[jreback]

related #3811

[jreback]

@hayd going to resurrect this for primary web access (with fallback to urllib2)
?

and put it io/common if not already there

then need to change uses of web acces across io to use this

and then docs!

[jreback]

see #4140 Google BigQuery is dependent on httplib2 is that a viable alternative here?

[hayd]

@jreback will have a loook

[hayd]

Not sure what to do with this one (certainly a lot has changed in io.common)