Reading from mongodb (BSON)

[hayd]

Now we have a neat read_json function (#3804), it'd be really neat if we could read in from mongo using this i.e. without have to create an intermediary python object.

I wasn't even sure how to return a json (bson) string from a mongo query, though surely this ought to be relatively easy...

I've no idea how much of an issue the BSON parts are, or whether these can be converted after the fact...

http://stackoverflow.com/questions/17805304/how-can-i-load-data-from-mongodb-collection-into-pandas-dataframe

[jreback]

comparing BSON/JSON/msgpack.....
http://stackoverflow.com/questions/6355497/performant-entity-serialization-bson-vs-messagepack-vs-json

[hayd]

I think I'm missing something... Can I use msgpack to read from mongo? (I didn't think you could...)

[jreback]

no....its just a competing format

[ghost]

Has anyone here used Monary for doing this. I was looking for a fast method for hooking up pandas and mongo and came across this:

https://bitbucket.org/djcbeach/monary/wiki/Home

I found it via this blog post:
http://alexgaudio.com/2012/07/07/monarymongopandas.html

[hayd]

@dam5h Thanks for linking to that, it's shame it's not on pip (I posted on google groups to ask author about that, if no response I might just set it up myself :s). It's under the Apache so potentially we could migrate it... easier (at least for now) just to add to the cookbook?

[ghost]

@hayd Sounds good to me! I haven't tried using it yet but it looks interesting.

[lJoublanc]

Just thought I would share my experience here: I've managed to re-use the msgpack code to create very crude serialization to MongoDB (deserialization not tried yet). This involved calling the encode method of packers.py recursively on a dataframe to produe a dict, and then calling pymongo The only other change I made was to monkey-patch pandas.io.packers.convert to wrap any encoded numpy arrays in bson.binary.Binary (currently msgpack stores these as encoded strings in latin-1 base - MongoDB doesn't like this).

[jreback]

interesting

can I benchmark vs monary? see if it's close?

[lJoublanc]

As I mentioned it's very crude. Still trying to get to grips with publishing notebooks on github with nbviewer, see if this works: https://gist.github.com/lJoublanc/c8591cd8e918024d505a#file-testpickle-ipynb. You'll need python-blosc and pymongo installed.

nbviewer : http://nbviewer.ipython.org/gist/lJoublanc/c8591cd8e918024d505a

[femtotrader]

Any news about this ?

[lJoublanc]

@femtotrader was about to suggest you look at arctic as it has BSON (de)serialization into pandas frames implemented, but I see from your profile you already seem to be using it. Does that implementation serve your purpose well?

[femtotrader]

Thanks @lJoublanc I'm giving a try to Arctic... I wondered if Monary wasn't more efficient... but road is not so straight than using Arctic https://bitbucket.org/djcbeach/monary/issues/19/use-pandas-series-dataframe-and-panel-with

[drafter250]

Does the bson_numpy project close this? http://bson-numpy.readthedocs.io/en/latest/#

[jreback]

is this a conda package / wheel?

[gfyoung]

Doesn't appear to be the case from the docs, but I think there is a pip package:

https://pypi.python.org/pypi/BSON-NumPy/0.1

[jreback]

source only are not very useful when a c library is needed

[gfyoung]

Ah, that's a good point.

[wesm]

Closing; this might be better addressed in an external package