Replacing Solr for full text search

[Elexy]

Would this be worth a try to replace our Solr indexes? We index many large JSON documents in there with many properties and some very big HTML or text properties. We do keep an archive already in S3 so this spiked my interest. The main use case is to find matches for strings in those documents and then return not only the document but preferably metadata on where in the document the match is and perhaps even a snippet.

[fulmicoton]

It depends on whether you update your documents or not.
Quickwit is append-only. Which means that we cannot modify documents once they have been inserted.

You can :

• append documents
• remove documents after they reach their time retention period
• and you can delete documents in batch to deal with something like GDPR
  That's all.

It is usually fine if you work with something that looks like logs or if it is the snapshot of a corpus that you update once a month for instance. In that case, reindexing the world can makes sense.

If you fit that constraint, whether this would be a good or a very good idea depends on your QPS and your retention.

The higher your QPS, the higher the cost of GET requests, and the less it will be a win to use Quickwit.
The higher your retention, the higher the cost of storage, and storing things in S3 will really decrease your cost.

And of course, if your use case fits this bill, your life is likely to be greatly simplified:
storing your live data on S3 simplifies a lot of things.

[Elexy]

Thanks for that. Currently we update documents in Solr, but in reality it is more like a replacement of the document, because a new version of it has come out. So I guess that could still work. Our QPS is quite low, the main problem has always been the ingestion speed and that be solved by just uploading files to S3. Retention is ideally infinite, but we are now obviously having to constrain it because of cost & performance. I think we're going to do an experiment with this. I'll let you know how it goes.

[fmassot]

Note that it’s possible to cache splits (index data files) on the local disk to cope with high QPS use cases.

Happy to help here, we have also a discord server discord.quickwit.io