Full text search support #3747

ranhsd · 2017-04-24T09:52:33Z

Hi, I wanted to know if parse-server support full text search. The full text search is available since MongoDB version 2.6 and it is possible to create text index (currently only one) which provides efficient search capabilities on text fields.

From the code it looks like that this capability has not been implemented ...

In case we need to add this capability i think we should perform the following tasks:

Write the server side code (probably modify the MongoDB storage adapter)
Update Parse rest api endpoints
Update all client SDK's (at least iOS/Android/Javascript)

Thanks.

flovilmart · 2017-05-06T23:24:58Z

Can you point out the documentation for full text search? If it requires a special operator?

ranhsd · 2017-05-07T08:29:04Z

Hi @flovilmart . Yes it actually required 2 things and there is also one limitation

Create text index on one or more columns (multiple column is the compound index)
Use the following command the run the search:

db.{collection_name}.find({$text: {$search: "{search string}"}}

Full text search is supported since v2.6 of Mongo and currently you can create only one text index per collection.

There is a great user guide + tutorial which explain what it is and to use it:

https://code.tutsplus.com/tutorials/full-text-search-in-mongodb--cms-24835

flovilmart · 2017-05-07T11:59:44Z

Ok that's something worth considering!

ranhsd · 2017-05-07T12:08:47Z

Yes it is! until now the solution was to create the full text search indexes in elastic search via mongoostatic.
The idea is to replicate all the text indexes column data into elastic search and when the client sending a search query the library execute the query in front of elastic instead of MongoDB. The big advantage of this approach is that elastic search engine is much more powerful than MongoDB native search capabilities and also you can create multiple full text search indexes with this approach. Unfortunately we cannot integrate this solution because parse-server not use Mongoose behind the scenes so we must use the native MongoDB capabilities

BTW! how much effort is to use Mongoose for executing the queries to the DB instead of the native MongoDB driver? (I am not talking about replacing the whole solution but maybe use it as optional dependency)

hhanesand · 2017-05-12T16:14:18Z

Looks like it has already implemented this at https://github.com/seriph/parse-server

@seriph Could you throw together a pull request? 😄

seriph · 2017-05-12T21:37:14Z

Sure, I put this together right around the time that this thread was started. Code works great but I consider it beta. Note that it requires changes to both the server and the API. I'll submit a pull soon.

Until then, if you have urgent need in your projects, feel free to use the npm packages I submitted. Just drop and replace with parse and parse -server.

https://www.npmjs.com/package/parse-server-text
https://www.npmjs.com/package/parse-text

I've only updated the node/js API so far. Would love to see someone update for the other platforms.

flovilmart · 2017-05-12T21:40:41Z

@seriph nice, we'll take care of the SDK's once the feature lands in parse-server (no need to rush those). @rogerhu and I will take care of the android and iOS SDK's.

ranhsd · 2017-05-14T07:02:19Z

We need also to update the docs about this new capability. Thanks a lot !

SebC99 · 2017-06-01T16:31:32Z

Any news on this? It would be awesome! :)

dplewis · 2017-06-03T00:05:10Z

@flovilmart I can do a PR for this as Postgres also supports full text.

Should the tests go in ParseQuery.spec?
What would we call this query.fullTextSearch()?

flovilmart · 2017-06-03T00:08:18Z

YES! Amazing
Ok! That's fine for me

For the REST format, let's stick with something close to the mongo query format (as the GeoPoints do)

dplewis · 2017-06-03T00:11:54Z

Should I include all endpoints? Or do 1 at a time

{
  $text:
    {
      $search: <string>,
      $language: <string>,
      $caseSensitive: <boolean>,
      $diacriticSensitive: <boolean>
    }
}

flovilmart · 2017-06-03T00:20:41Z

The REST format ultimately should support all payloads, you can put validation on all at first, and write failing tests that are xit'ed this way we know this support is ongoing if you prefer spliting the work in multiple PR

seriph · 2017-06-03T00:27:48Z

How does postgres apply?

…

On Jun 2, 2017 6:20 PM, "Florent Vilmart" ***@***.***> wrote: The REST format ultimately should support all payloads, you can put validation on all at first, and write failing tests that are xit'ed this way we know this support is ongoing if you prefer spliting the work in multiple PR — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3747 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUl40u1IR54geM0mn9h2Fv5syJNRwv5ks5sAKbkgaJpZM4NF72e> .

dplewis · 2017-06-03T00:34:45Z

Postgres is another NOSQL database Parse supports like Mongo.
We try to have as much functionality supported by both databases.

https://www.postgresql.org/docs/9.5/static/textsearch.html

seriph · 2017-06-03T06:10:49Z

Pull request submitted @flovilmart @hhanesand @SebC99

seriph · 2017-06-03T06:11:14Z

#3905

nuborian · 2017-07-26T12:47:05Z

Anything new to this topic? :)

SebC99 · 2017-07-26T16:20:53Z

It's available on parse-server since 1.5.0 but only accessible via REST API
The php sdk PR is on hold as they want to include a default index creation with it (I don't know why it has to be done at the same time...) and so are the other sdk. I'm working on the JS one, and will open a PR soon for this.

nuborian · 2017-07-26T18:31:25Z

Thanks for the Update! @SebC99 Are there any docs about this? Couldn't find it on the REST Api Guide

SebC99 · 2017-07-26T18:37:35Z

There's a PR from @dplewis in the docs repo

DIMEGLIOM · 2017-08-09T18:59:00Z

Are there any instructions out there regarding using this feature w/in iOS? This feature looks very promising and I was hoping to use it in a project I am currently working on.

Thanks in advance to anyone who can get back to me.

rihadavid · 2017-10-03T21:56:06Z

@dplewis I am thinking about implementing support for full-text search over multiple fields. Would you be able to help me with this? I made it work on my local MongoDB, so I know it is possible. But I am not familiar with Postgres, REST, how to run tests and don't have much experience with contributing :)

What was needed to make it work on MongoDB:

create index in mongo shell - example: db.getCollection("_User").createIndex( { name: "text", username: "text", about: "text"} )
~~add '$text' to allowConstraints in ClassesRouter.js~~
allow providing null as the key
somehow bypass error 85 in createTextIndexesIfNeeded in MonoStorageAdapter.js (you wrote that function, so maybe you could help with correct solution? Maybe just do not auto-create index if doing query over multiple fields?)
~~define the $text query without the key, like query._extraOptions['$text'] = { '$search': options }; in javascript (cloudcode)~~
call the query with null key query._addCondition(null, '$text', { '$search': options }); in javascript (cloudcode)
got the results! it works!

What do you think? Could we make it, please? :)

dplewis · 2017-10-03T22:21:22Z

@rihadavid We came up with a proposal for handling full-text over multiple fields. Adding indexes via Schema API

#4212 (comment)

I should have some time next week to work on it. If you want to open a PR feel free.

@flovilmart We can provide an immediate solution to this problem or should we wait until the Schema API gets indexes? What do you think?

rihadavid · 2017-10-03T22:33:21Z

@dplewis Actually, I had a bug in my test from previous comment and it is much easier than I thought. Just providing null as the key, allowing the null as the key and bypassing the error 85 makes the query search over the multiple fields.

Like this:
where={"null":{"$text":{"$search":{"$term":"some words"}}}}

This really seems easy, maybe I will try the PR. Do you think it's OK to call it this way, with null? I am surprised it works, but it does!

dplewis · 2017-10-03T22:47:50Z

@rihadavid There are reasons why automatic index creation and catching error 85 are in place. Please see this comment here. #4212 (comment)

Removing error 85 will allow for multiple indexes but we will like to keep it in place for now.

rihadavid · 2017-10-03T22:53:54Z

Ok, sure :) It seems too complicated, I am not a big friend with indexes. So I will rather let you work on it and keep my experiments in my local environment until you guys make it a stable feature :) Thank you for implementing this!

dplewis · 2017-10-03T23:24:05Z

@rihadavid Contributions are always welcomed. Trust me I didn't know about full-text, indexes, postgres, rest when I did this.

grosscorporation · 2017-10-06T21:23:37Z

so will the following work? I tried it aint, or do I have to recreate my Schema

where : { 'null' : { '$text' : { '$search' : { '$term' : val } } } }

dplewis · 2017-10-06T21:36:49Z

@GoGross The REST docs has what you need. You don't need to recreate your schema
http://docs.parseplatform.org/rest/guide/#queries-on-string-values

where : { '$text' : { '$search' : { '$term' : val } } }

grosscorporation · 2017-10-07T13:51:22Z

Do I have to update my schema? this fails

dplewis · 2017-10-07T17:03:03Z

Please update to the latest version of parse server.

https://github.com/parse-community/parse-server/releases

MHX792 · 2017-10-13T09:47:28Z

any way to use fulltext search within the iOS SDK?

rihadavid · 2017-10-13T09:50:10Z

PR for iOS SDK already exists here: parse-community/Parse-SDK-iOS-OSX#1196

montymxb · 2017-11-03T05:31:52Z

Going to go ahead and close this out now that the feature is merged in. Feel free to keep discussing however, if anything significant comes up feel free to open up another issue or let us know to pop this one back open, whichever works.

Also relevant PRs look to be open on the sdks. From here I'll see if we can't get those in place now 👍 .

iOS
JS
PHP(merged)

For Android it's issue #751. If someone has a moment we would appreciate a PR over there as well!

rihadavid · 2018-11-02T14:29:09Z

If anyone is reading this still being interested in the full-text search over multiple fields, I just figured out that it works out of the box - if you had manually created the multi-field text index in the db.

All you need to do is to provide any of the fields defined in the text index (so that the parse-server is not trying to create the index for you). It will search over all of those fields defined in the text index, not just the one you provide 👍

var query = new Parse.Query(BarbecueSauce);
query.fullText('name', 'bbq'); //provide any of the fields in the index, it will search over all of them

Tested on Parse Server 2.8.4 with Mongo.
Maybe this could also be mentioned in the docs, but it would probably require some more testing.

dplewis · 2018-11-02T15:07:30Z

@rihadavid Would you like to submit a PR? Passing in an array to create a compound index automatically.

ranhsd · 2018-11-02T15:14:58Z

Hi @rihadavid Yes I already request it as a feature few days ago here: #5152

@dplewis I don't think that we should replace the current function that we have and use array instead of string because It can break current implementations. We can either create additional function something like: fullTextCompound or check inside the current function if the input is an array or string and create the indexes accordingly.

rihadavid · 2018-11-02T16:01:33Z

@dplewis Nope, sorry, but I am currently buried deeply in heaps of C# code that should have been finished months ago 😭
@ranhsd what a coincidence! :)

aacassandra · 2023-02-26T08:01:16Z

can i try like this in class role?
{{HOST}}{{MOUNTPATH}}/classes/_Role?where={"name":{"$text":{"$search":{"$term":"SUPERADMIN"}}}}
not working

when using $regex is working

mtrezza · 2023-02-26T12:07:05Z

@aacassandra

For help with Parse Platform we recommend our community forum or our community chat
For general coding questions we recommend Stack Overflow using the parse-platform tag

matheusfrozzi · 2023-08-01T18:56:05Z

Hey guys, a question:
fullText is really searching on all fields? in my tests here it doesn't work.

the query produces this JSON:

{
  title: { '$text': { '$search': { '$term': 'pet training' } } }
}

and it only returns items that contain this term on the title key

But in the Atlas search tester, it generates this JSON:

[
  {
    $search: {
      index: "blog-search",
      text: {
        query: "pet training",
        path: {
          wildcard: "*"
        }
      }
    }
  }
]

And it returns great search results, with score counting with all keys.

Is there a way to do this on fullText, or do I have to use aggregate?

Thanks

dplewis mentioned this issue Jun 3, 2017

Full Text Search Support #3904

Merged

rogerhu mentioned this issue Oct 13, 2017

Add full text query search support parse-community/Parse-SDK-Android#751

Closed

montymxb closed this as completed Nov 3, 2017

dplewis mentioned this issue Jan 31, 2018

Full text search for Parse JS SDK and or Parse Cloud code #4490

Closed

allanext mentioned this issue Nov 29, 2020

Aggregate pipeline query doesn't accepts $and operator parse-community/Parse-SDK-JS#1263

Closed

parseplatformorg mentioned this issue Aug 30, 2023

refactor: Security upgrade mongodb from 4.10.0 to 4.17.0 #8723

Closed

skmezanul mentioned this issue Aug 30, 2023

[Snyk] Security upgrade mongodb from 4.10.0 to 4.17.0 skmezanul/parse-server#15

Open

parseplatformorg mentioned this issue Aug 31, 2023

refactor: Security upgrade mongodb from 4.10.0 to 4.17.0 #8729

Closed

Full text search support #3747

Full text search support #3747

Comments

ranhsd commented Apr 24, 2017

flovilmart commented May 6, 2017

ranhsd commented May 7, 2017

flovilmart commented May 7, 2017

ranhsd commented May 7, 2017 • edited Loading

hhanesand commented May 12, 2017 • edited Loading

seriph commented May 12, 2017

flovilmart commented May 12, 2017

ranhsd commented May 14, 2017

SebC99 commented Jun 1, 2017

dplewis commented Jun 3, 2017

flovilmart commented Jun 3, 2017

dplewis commented Jun 3, 2017 • edited Loading

flovilmart commented Jun 3, 2017

seriph commented Jun 3, 2017 via email

dplewis commented Jun 3, 2017

seriph commented Jun 3, 2017

seriph commented Jun 3, 2017

nuborian commented Jul 26, 2017

SebC99 commented Jul 26, 2017

nuborian commented Jul 26, 2017

SebC99 commented Jul 26, 2017

DIMEGLIOM commented Aug 9, 2017

rihadavid commented Oct 3, 2017 • edited Loading

dplewis commented Oct 3, 2017

rihadavid commented Oct 3, 2017 • edited Loading

dplewis commented Oct 3, 2017

rihadavid commented Oct 3, 2017

dplewis commented Oct 3, 2017

grosscorporation commented Oct 6, 2017

dplewis commented Oct 6, 2017

grosscorporation commented Oct 7, 2017

dplewis commented Oct 7, 2017

MHX792 commented Oct 13, 2017

rihadavid commented Oct 13, 2017

montymxb commented Nov 3, 2017

rihadavid commented Nov 2, 2018

dplewis commented Nov 2, 2018

ranhsd commented Nov 2, 2018

rihadavid commented Nov 2, 2018

aacassandra commented Feb 26, 2023

mtrezza commented Feb 26, 2023

matheusfrozzi commented Aug 1, 2023

ranhsd commented May 7, 2017 •

edited

Loading

hhanesand commented May 12, 2017 •

edited

Loading

dplewis commented Jun 3, 2017 •

edited

Loading

rihadavid commented Oct 3, 2017 •

edited

Loading

rihadavid commented Oct 3, 2017 •

edited

Loading