Handle ObjectId property value (and arrays of ObjectIds) properly

[johnymontana]

Originally reported on StackOverflow here:

2016-08-02 18:43:07,881 [ERROR] mongo_connector.oplog_manager:282 - Unable to process oplog document {u'h': 82402292097390737L, u'ts': Timestamp(1470143587, 1), u'o': {u'deviceName': u'iphone', u'countryCode': u'+91', u'degreeIds': [ObjectId('56f22b6b3ec80d233fb0f45d')], u'userType': u'DOCTOR', u'practiceIds': [ObjectId('576566288006599059170496')], u'noShowCount': 0, u'timezone': u'asia/kolkata', u'categoryIds': [], u'originalProfilePicture': u'http://bucketname.s3.amazonaws.com/doctor/profilePicture/Profile_57a0986720820b161c932287.png?timestamp=1470142568822', u'patientsRated': 0, u'collegeIds': [ObjectId('56f22b573ec80d233fb0f45c')], u'loginCount': 1, u'clinics': [], u'blockedByDoctors': [], u'phoneNumber': u'1113040410', u'blockedDoctors': [], u'doctorRefNum': 1600001722L, u'deviceType': u'IOS', u'totalRating': 0, u'email': u'myemail.id@gmail.com', u'registrationDate': datetime.datetime(2016, 8, 2, 12, 56, 8, 765000), u'thumbProfilePicture': u'http://bucketname.s3.amazonaws.com/doctor/profilePicture/thumb/Thumb_Profile_57a0986720820b161c932287.png?timestamp=1470142568822', u'profileStatus': u'PROFILE_DETAILS', u'consultationCharges': 200, u'usersRated': 0, u'testIds': [], u'servingLocations': [ObjectId('56f22bcaf9a8f73a3f9e99aa'), ObjectId('576647ff7777044e7387d4b5')], u'fullName': u'ER. FULL NAME', u'specialityIds': [ObjectId('5765662880065990591703ef'), ObjectId('576566288006599059170416'), ObjectId('57656628800659905917041d')], u'_id': ObjectId('57a0986720820b161c932287')}, u't': 36L, u'v': 2, u'ns': u'clinic_world_local.doctors', u'o2': {u'_id': ObjectId('57a0986720820b161c932287')}, u'op': u'u'}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/mongo_connector/oplog_manager.py", line 268, in run
    ns, timestamp)
  File "/usr/local/lib/python2.7/site-packages/mongo_connector/util.py", line 38, in wrapped
    reraise(new_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 101, in update
    tx.commit()
  File "/usr/local/lib/python2.7/site-packages/py2neo/cypher/core.py", line 306, in commit
    return self.post(self.__commit or self.__begin_commit)
  File "/usr/local/lib/python2.7/site-packages/py2neo/cypher/core.py", line 248, in post
    rs = resource.post({"statements": self.statements})
  File "/usr/local/lib/python2.7/site-packages/py2neo/core.py", line 307, in post
    response = self.__base.post(body, headers, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 983, in post
    rq = Request("POST", self.uri, body, headers)
  File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 382, in __init__
    self.__body = json.dumps(body, cls=JSONEncoder, separators=",:")
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 251, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/jsonencoder.py", line 37, in default
    return json.JSONEncoder.default(self, obj)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
Neo4jOperationFailed: ObjectId('5765662880065990591703ef') is not JSON serializable

ObjectId values stored as properties are not handled property by neo4j_doc_manager. When an ObjectId property value is encountered neo4j_doc_manager should:

1. convert the ObjectId to its string representation and store as a property on the Node in Neo4j
2. create a relationship to the node referenced by the ObjectId
3. do the above for arrays of ObjectIds

[johnymontana]

Commit 165e28e addresses 1 and 2 listed in previous comment

[mromanello]

Hi! do you know if there is any plan to address also point 3 (i.e. convert arrays of ObjectIds into properties)? That's precisely the issue I'm having.

[akarsh]

@johnymontana

pip packages installed on conda

• mongo-connector version: 2.5.1
• neo4j-doc-manager version: 1.0.0.dev11
• py2neo version: 2.0.7
• pymongo version: 3.4.0

i am using keystonejs cms template for the university project. Upon creating a blog. I see the following issue with json serialization.

Exception in thread Thread-2: Traceback (most recent call last): File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/mongo_connector/util.py", line 104, in wrapped func(*args, **kwargs) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/mongo_connector/oplog_manager.py", line 274, in run ns, timestamp) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 81, in update tx.commit() File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/py2neo/cypher/core.py", line 306, in commit return self.post(self.__commit or self.__begin_commit) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/py2neo/cypher/core.py", line 248, in post rs = resource.post({"statements": self.statements}) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/py2neo/core.py", line 307, in post response = self.__base.post(body, headers, **kwargs) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 983, in post rq = Request("POST", self.uri, body, headers) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 382, in __init__ self.__body = json.dumps(body, cls=JSONEncoder, separators=",:") File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/json/__init__.py", line 251, in dumps sort_keys=sort_keys, **kw).encode(obj) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/json/encoder.py", line 207, in encode chunks = self.iterencode(o, _one_shot=True) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/json/encoder.py", line 270, in iterencode return _iterencode(o, 0) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/site-packages/py2neo/packages/httpstream/jsonencoder.py", line 37, in default return json.JSONEncoder.default(self, obj) File "/Volumes/ADDITIONAL/anaconda/anaconda/lib/python2.7/json/encoder.py", line 184, in default raise TypeError(repr(o) + " is not JSON serializable") TypeError: ObjectId('595032fddb7a6e1ce1b38345') is not JSON serializable

Help is appreciated & thank you