-
Notifications
You must be signed in to change notification settings - Fork 215
defer bulk channel init for mongo node reuse #211
Conversation
I'm having a hard time figuring out how the bulkWriter close blocks, causing the app to hang. From what I understand, the only exit from that bulkWriter is through the quit channel and we're skipping over that here because it presumably blocks. Is the final data flush hanging? If so, I think it's more natural to leave the quit/cleanup in and attacking the bug from the bulk writer end. If you could a quick execution path summary of the blocking state, that'd be awesome. I don't doubt this works, I just want to make sure I understand the code path properly. |
nodes:
my_mongo:
type: mongo
uri: mongodb://127.0.0.1:27017/my_database
bulk: true
Source({name:"my_mongo", namespace: 'my_database.SomeCollection'})
.transform({filename: 'changenamespace_and_log.js', namespace: 'my_database.SomeCollection'})
.save({name:"my_mongo", namespace: 'my_database.OtherCollection'});\
module.exports = function(doc) {
console.log("transformer: " + JSON.stringify(doc));
doc.ns = "my_database.OtherCollection";
return doc
} load up the
then run transporter:
you'll see the documents get transferred properly to the new collection but transporter never completes. if you add some print statements in the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for this @jipperinbham !
ccb59a1
to
1e5fa38
Compare
* defer bulk channel creation until Listen func * update CHANGELOG
When a mongodb adaptor is defined in the
config.yaml
withbulk=true
and is used for theSource
andSink
in theapplication.js
, transporter will never shutdown properly.The following change adds a new field to the adaptor and sets it when the
Listen
func is called and checked withStop
is called.fixes #152