Filetype when passing contents to index, why is it required?

[ashutoshsaboo]

Hi @jameslittle230 ,

Why does Stork require for filetype even when it's being passed with contents rather than path for index creation? With path, filetype seems fairly intuitive to require for since it needs to determine various parsing strategies etc before tokenizing. But with contents in the input, isn't the first part already done by the client? Then why require for filetype still in input?

This is regarding the issue - reese/gatsby-plugin-stork#12 (comment) - we are trying to make it work with gatsby. But it seems to be failing at the index creation step due to failures from stork (and it seems like it's because it needs filetype specifically when passing contents rather than path in index creation). I also tried with the toml file (you can check for this in the same issue thread linked above) by manually passing to stork --build (rather than using the plugin), and it seems to throw a similar error to the one i mentioned above, i.e, stork wasn't able to determine the file type to use based on contents (rather than path) that were passed.

If possible can you have a look at the linked entire issue thread and possibly help out here? If it's not related to the filetype resolver issue, and if you can point us in the correct direction that'd be extremely helpful! :)

Thanks! :)

[jameslittle230]

Hey @ashutoshsaboo! Thanks for writing in.

I think @reese has answered your question really well in the linked thread:

I think it does make sense that you would need to pass a filetype, because otherwise Stork doesn't know what parser is required. contents isn't a parsed format -- it could be a raw HTML string, or plain markdown, so Stork needs to know what type of contents they are. It can't tokenize something that it can't parse tokens for.

For example, I put raw HTML into the contents field for one Stork index I build, and I need to tell Stork that it should parse the contents as HTML instead of plaintext.

There might be an enhancement here where Stork assumes PlainText unless you specify otherwise, but I’d have to poke at it to make sure that is a fair assumption to make.

(Also: converting to a Github Discussion because this feels more like a “why does Stork do it this way” question than a bug report or feature request)

[ashutoshsaboo]

Umm thanks @jameslittle230 for the clarification, intuitively I seem to think why would clients want to take on the burden of parsing files to contents, when they can let Stork take on that responsibility which it'd be definitely much more suited for as well as more robust (and yes so much more easier as well for the client)! That is just how i think it to be, but you might have better knowledge of use-cases for Stork to take a more informed decision.

Anyways if you think this is not a fair presumption to set a default of filetype = PlainText, possibly bare minimum shouldn't we atleast mention in the docs: https://stork-search.net/docs/config-ref : that with passing contents it is mandatory for clients to pass the filetype param?

Reason I mentioned that was because in SSGs like gatsby more specifically, since it has such a wide array of plugins for sourcing data from various plugins - so data ends up in nodes ; and there i'd presume clients wanting to use the contents field directly for indexing them. But in any case it should be fine, as long as it's called out properly in the docs I think about the requirement to pass filetype when passing contents in input.

Also, thank you so much for building this library! There are some other client side search libs too, but the reason I preferred Stork right away was the direct integration on UI with search bar etc as well - makes it so much more easier to integrate search with only very few configs! :)

[jameslittle230]

Adding a clarification to the docs is a good idea! Thanks 😄