Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add import_filter to MongoDBRiverDefinition. #157

Closed
wants to merge 1 commit into from

Conversation

bernd
Copy link

@bernd bernd commented Oct 8, 2013

This allows filtering of the initial import.

I noticed that the initial import ignores the defined filter in the MongoDBRiverDefinition. I first tried to feed the configured filter to slurpedCollection.find(), but that does not work because the oplog filter needs the o. prefix. And that does not work with the regular find() on the collection. So I created another filter option. (import_filter)

Not sure if you want to merge this as it is. Maybe it would be nicer to only have one filter definition that works for the initial import and the oplog filter. (by prepending the o. prefix automatically or something like that)

It works for me and helps me to filter the initial import so I wanted to share it.

Attention: I don't know Java, so please double check this code. ;)

This allows filtering of the initial import.
@bernd
Copy link
Author

bernd commented Oct 8, 2013

Usage example:

curl -XPUT "localhost:9200/_river/foo-de/_meta" -d '
{
  "type": "mongodb",
  "mongodb": { 
    "servers":
    [
      { "host": "localhost", "port": 27017 }
    ],
    "db": "foo",
    "collection": "data",
    "import_filter": "{\"lang\":\"de\"}",
    "filter": "{\"o.lang\":\"de\"}"
  }, 
  "index": { 
    "name": "foo", 
    "type": "de"
  }
}'

@richardwilly98
Copy link
Owner

@bernd Thank you for reporting the issue.
I will take a look this week at your PR and will se if we can use only one filter definition.

@benmccann
Copy link
Collaborator

I agree one filter definition would be nicer. Should be able to just check if the filter starts with "o." then strip off the first two characters

richardwilly98 added a commit that referenced this pull request Oct 11, 2013
- mongodb.filter support with or without "o." prefix
@richardwilly98
Copy link
Owner

@bernd it is now possible to set the filter like this:

curl -XPUT "localhost:9200/_river/foo-de/_meta" -d '
{
  "type": "mongodb",
  "mongodb": { 
    "servers":
    [
      { "host": "localhost", "port": 27017 }
    ],
    "db": "foo",
    "collection": "data",
    "filter": "{\"o.lang\":\"de\"}"
  }, 
  "index": { 
    "name": "foo", 
    "type": "de"
  }
}'

or

curl -XPUT "localhost:9200/_river/foo-de/_meta" -d '
{
  "type": "mongodb",
  "mongodb": { 
    "servers":
    [
      { "host": "localhost", "port": 27017 }
    ],
    "db": "foo",
    "collection": "data",
    "filter": "{\"lang\":\"de\"}"
  }, 
  "index": { 
    "name": "foo", 
    "type": "de"
  }
}'

The river definition will provide 2 BDObjects for filtering one without the o. for the initial import and one with the prefix for oplog.rs

@bernd
Copy link
Author

bernd commented Oct 12, 2013

Awesome, thank you very much! :)

I'm closing this one.

@bernd bernd closed this Oct 12, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants