Allow to search keys

[CharlesNepote]

We will probably have thousands of properties and we need to make them easy to find.

For the end-user, the best option would be to enter a keyword related to the property he wants to add.

Example: searching how to enter the color of the cap:

• the user enter "c", then "o", then "l"
• the UI should provide a list containing properties (or descriptions) containing "co" then "col"

The GET /keys is currently returning all the keys with stats:

• the stats are not useful for our current concern
• returning ALL the keys might be costly when there are thousands of properties

Currently, Open Food Facts website is already using tags autocompletion with a library (tagify) that takes simple JSON arrays in input ["tag1","tag2","free_tag"] (see the explanation here).

Suggestion 1

Create a GET /keys&term=col which returns a simple JSON array: ["color","color_of_the_cap","multicolor"].

Suggestion 2

Create another API route to return only the useful data:

• rename GET /keys into /key_stats
• create GET /keys&term=col which returns a simple JSON array: ["color","color_of_the_cap","multicolor"].

Part of

• 🎯 What can I work on ? (Folksonomy Engine) #70

[stephanegigandet]

Well the current /keys is very similar to what you get on the web site with facets /labels
I do think the stats are very useful, even for end users, to know how popular a key is.

My preference would be to keep /keys as is, and add parameters to filter them ?term=color (word search), and ?autocomplete=col (makes it clear that is searches on starts of words and not full words). But keep the output the same (with counts).

[CharlesNepote]

I don't say we should avoid stats! I also do think stats are useful, but not for an autocompletion service.

To synthetize you suggestion @stephanegigandet :

Suggestion 3

Keep GET /keys results as is but add a &term= (full word search) and &autocomplete= (start of the word) to refine the results.

About &autocomplete=

Currently, Open Food Facts is returning all words containing the term, and not just the words beginning by the searched term. I'm in favor of chosing words containing the searched term as many of the properties will use expressions. Eg:

• all properties containing a date: recalled_product:date, launch:date, seen_on_tv:date, etc.
• multicolor, color_of_the_cap (monochrome should be also found thanks to its description (not implemented yet))

KISS for the backend devs or for the frontend devs?

Suggestions 1 and 2 makes life quite easier for frontend devs: JSON arrays are far easier to use (and it's ready to use in OFF website).

Suggestion 3 probably makes life easier for backend devs but makes frontend devs' work harder.

IMHO, the API should always try to make frontend devs' work easier.

[stephanegigandet]

Currently, Open Food Facts is returning all words containing the term, and not just the words beginning by the searched term. I'm in favor of chosing words containing the searched term as many of the properties will use expressions. Eg:

You need both &term= for word search, and &autocomplete= for autocomplete / suggest as you type search.

What you described here is suggest as you type:

the user enter "c", then "o", then "l"
the UI should provide a list containing properties (or descriptions) containing "co" then "col"

And here it's regular word search:

For the end-user, the best option would be to enter a keyword related to the property he wants to add.

The user types a keyword and hits enter. (then the keyword can match in the property name, but potentially also its metadata, description etc.)

[stephanegigandet]

Suggestion 3 probably makes life easier for backend devs but makes frontend devs' work harder.

Not much I think, and it's useful to have stats/metadata available even for suggest as you type results, or search results, so that you can display it to users:

Search properties: col (user typing)
Properties:

• Color (1532 products)
• Collectible (120 products)
• Bottle cap color (32 products)

I'd argue that it's even simpler for devs if you get the same format of results when you list all keys, or keys containing a specific term, or keys starting with characters. Frontend code can be only one function that covers all use cases.

[Ajay-Nair]

@stephanegigandet So if I understand you correctly, you want to do as follows:
Idea 1: As the user types each letter, the frontend does the &autocomplete query (which does prefix search, else we may have too many hits). When the user enters a space after a word is complete, then do &term query. This &term query does full word search.

For eg: So as I type 'c' it shows ['Color (1532 products)', 'Collectible (120 products)']. If we did full word search with 'c', we will return all items with 'c' in it, thus we only do prefix search for &term queries. As I complete 'color ', it shows ['Color (1532 products)', 'Bottle cap color (32 products)']. The problem here being two different behaviours for the same input field.

Idea 2: Both &term query and &autocomplete query does full word search. But we only give results after second character is typed.

I will think and come back on handling multi-word queries (May have to use some sort of inverted search/search engine, though it may be an overkill).