Ignore markdown codeblocks for tags

[elmodor]

Hello,

I took a quick look on how to exclude tags inside markdown codeblocks.
This should help out for #35

I'm using a second regex to remove all markdown codeblocks.
However, this will also remove the markdown from the return value of extract_tags.

I tested the code with various markdown codeblocks. The tag #TagIgnore does not show up when searching for tags:tagignore while tags:tagok shows up.
Tested with following file:

Testing

Testing more
#TagIgnore me please
and this one too #TagIgnore

#TagOk

More to #TagIgnore
One more to #TagIgnore
#TagIgnore

What is going on? #TagOk
EOF

[dullage]

Thanks for giving this a go. When I get some time I'll take a look and run some tests 👍

[dullage]

Hi @elmodor, I've had a look at this this morning and have some thoughts.

My first thought is that MARKDOWN_RE should probably be named CODEBLOCK_RE and content_ex_markdown should be content_ex_codeblocks. A small point but help with the diagram below.

I also think that codeblock content should still be indexed and therefore searchable. Just not as tags. So I think we need to something along these lines:

[elmodor]

I agree, I should have named it codeblock obviously.

Regarding your diagram, I think there should not be a line from tags to content_ex_tags?
As far as I see, we would use content_ex_codeblocks to obtain the tags but use content to obtain content_ex_tags just as before?

content_ex_codeblock = re.sub(cls.CODEBLOCK_RE, '', content)
_, tags = re_extract(cls.TAGS_RE, content_ex_codeblock)
content_ex_tags, _ = re_extract(cls.TAGS_RE, content)

[dullage]

I had in mind to use tags to know what to remove from content to get content_ex_tags. We could, as you propose, just use the TAGS_RE again but this would mean that comments in codeblocks wouldn't get indexed at all (either as content or tags).

Of course, I can imagine the most efficient way of doing all of this would be to update TAGS_RE to simply ignore anything within a codeblock. Then we'd only need to run regex once (as it is now). My concern would be that the TAGS_RE would become unwieldy (even more than it is now). So splitting up the process as proposed above may be the better (more understandable) way of doing things. Even if not the most efficient.

[elmodor]

I wasn't able to get one line TAGS_RE to work so I updated it to my last suggested code change.
If someone can wrap their head around the regex maybe they will be able to figure it out :D

[dullage]

The changes look good. When I get a chance I just want to run a couple of tests on my dataset. I'm interested to see if there's much of a change to the indexing speed and I'm also keen to see the difference in tags before and after this change.

[dullage]

Please bump INDEX_SCHEMA_VERSION constant to 4. This will force an index rebuild when flatnotes is updated.

[dullage]

Index rebuild times seem unchanged and having compared my list of indexed tags before and after the changes, I can confirm the hashtags found in code blocks are no longer indexed. Just one change requested and then this is ok to merge.