Markdown: improved version

[techee]

I fixed the problem with underlined headings from #3233 and in addition cleaned up the code a bit. Also, when running unit tests on the fixed code, there popped up a problem with code blocks using just indenting I fixed, more in the commit message (I had no clue one can do that, I haven't seen this documented anywhere but even github markdown works this way).

[codecov]

Codecov Report

Merging #3236 (78320f2) into master (05e6ab4) will increase coverage by 0.25%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3236      +/-   ##
==========================================
+ Coverage   85.01%   85.27%   +0.25%     
==========================================
  Files         206      207       +1     
  Lines       49127    49304     +177     
==========================================
+ Hits        41765    42043     +278     
+ Misses       7362     7261     -101     

Impacted Files	Coverage Δ
main/nestlevel.c	88.23% <100.00%> (ø)
parsers/markdown.c	100.00% <100.00%> (ø)
parsers/ruby.c	98.22% <100.00%> (+0.39%)	⬆️
main/lregex.c	82.07% <0.00%> (-1.01%)	⬇️
parsers/tex.c	97.47% <0.00%> (-0.32%)	⬇️
main/field.c	92.73% <0.00%> (-0.29%)	⬇️
optlib/man.c	0.00% <0.00%> (ø)
parsers/haxe.c	100.00% <0.00%> (ø)
parsers/gemspec.c	98.71% <0.00%> (ø)
parsers/asm.c	99.32% <0.00%> (+0.04%)	⬆️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05e6ab4...78320f2. Read the comment docs.

[masatake]

Thank you for updating.

[yamato@dev64]~/var/ctags-github% cat input.md                                 
# A

- [a](#a)
  - [b](#b)
- [c](#c)

## B
[yamato@dev64]~/var/ctags-github% ./ctags --options=NONE --sort=no -o - input.md
ctags: Notice: No options will be read from files or environment
A	input.md	/^# A$/;"	c
- [b](#b)	input.md	/^  - [b](#b)$/;"	s	chapter:A
B	input.md	/^## B$/;"	s	chapter:A

I think tagging - [b](#b) is not expected. Could you fix this?

[techee]

I think tagging - b is not expected. Could you fix this?

Oh yeah, forgot about this possibility. Thanks for noticing, should be fixed now.

[masatake]

Thank you for updating.
With using the codebase (universal-ctags/codebase), I compared the difference between the old regex based parser and C parser.

I found 3 bug patterns. Could you try to fix them?

---

input0.md:

# LANGUAGE
## Java
## C++
## C\#

output:

LANGUAGE	/tmp/input0.md	/^# LANGUAGE$/;"	c
Java	/tmp/input0.md	/^## Java$/;"	s	chapter:LANGUAGE
C++	/tmp/input0.md	/^## C++$/;"	s	chapter:LANGUAGE
C\\	/tmp/input0.md	/^## C\\#$/;"	s	chapter:LANGUAGE

expectation:

LANGUAGE	/tmp/input0.md	/^# LANGUAGE$/;"	c
Java	/tmp/input0.md	/^## Java$/;"	s	chapter:LANGUAGE
C++	/tmp/input0.md	/^## C++$/;"	s	chapter:LANGUAGE
C\\#	/tmp/input0.md	/^## C\\#$/;"	s	chapter:LANGUAGE

---

input1.md:

# H1

```STR```

# H2

output:

H1	/tmp/input1.md	/^# H1$/;"	c

expectation:

H1	/tmp/input1.md	/^# H1$/;"	c
H2	/tmp/input1.md	/^# H2$/;"	c

---

The "frontmatter" area should be skipped.
#3031
a679933
#3027 (comment)
#3032

(@masatake must improve Units/parser-markdown.r/frontmatter.d/input.md to be able to detect the bug I will report here.")

input2.md:

---
title: Awesome Geek Podcasts
permalink: /
layout: default
---

The new parser makes a tag for layout: default unexpectedly.
It is part of the frontmatter.

[techee]

Hmm, I just found

https://github.github.com/gfm/

when looking for some formal specification because there are many ambiguities and there are a lot more things to handle. For instance, it's not clear to me how to handle things like

```sh```

I think this isn't a fenced code block

https://github.github.com/gfm/#fenced-code-blocks

but rather a code span

https://github.github.com/gfm/#code-spans

But it's not clear to me how in such a case I could distinguish these when I first encounter the opening ``` and how should I know the sh isn't an info string (i.e. the string representing the parser used as a subparser):

https://github.github.com/gfm/#info-string

Also, is there any formal specification for how the "frontmatter" should look like? This isn't anything markdown-related, I guess it's just used by some tools using markdown - is this always delimited by ---?

[techee]

Also, the markdown standard seems to suport empty headings - should I generate tags for these too? In addition, markdown seems to support multiline headings like

one
two
===

but I think I'll just give up on these.

[masatake]

Also, the markdown standard seems to suport empty headings - should I generate tags for these too?

No, at least this pull request.

If the old regex parser cannot do something, it is o.k. the new parser cannot do them.
However, if the old parser does something well, I would like to expect the new parser does them as well.
I myself will inspect the 3 patterns when I find a time.

[techee]

I myself will inspect the 3 patterns when I find a time.

I'll try to do something myself, it's not a problem to fix the issues you mentioned somehow, I was just wondering if you have some ideas as this "something" may not be the completely correct thing to do.

[techee]

However, if the old parser does something well, I would like to expect the new parser does them as well.

The question is what should happen in situations like ```foo``` - right now I think it's something where it is unspecified what should happen exactly. The fact that the previous parser did something that appeared like a correct output doesn't mean it did it well.

[techee]

@masatake I hope I fixed the problems. With the ``` problem it's more or less a workaround as handling code span and fenced code blocks (and distinguishing which one is which) would be quite complicated. I'll leave this to future generations of ctags coders :-).

[techee]

@masatake I just made minor formatting changes in the code to follow the style used by uctags more closely.

[masatake]

Could you split d37fea4 into smaller ones?

When you made a commit (commit C) changing the behavior of the parser, I would like you to include a test case for the new behavior to the commit C, or made a new commit for testing the new behavior just after the commit C.
It greatly helps me to understand your changes.
git push --force is welcome.

#3241 is an example following the above practice.

[techee]

@masatake Done.

[masatake]

I have tested the update changes.

In the following input:

  `O`
# A

(Two spaces before `O`)

the parser cannot extract A.
I found the root cause in 9e602a6.

diff --git a/parsers/markdown.c b/parsers/markdown.c
index 38e6122ef..6c9ee3d8f 100644
--- a/parsers/markdown.c
+++ b/parsers/markdown.c
@@ -219,7 +219,7 @@ static void findMarkdownTags (void)
 			char c = line[pos];
 			char otherC = c == '`' ? '~' : '`';
 			int nSame;
-			for (nSame = pos + 1; line[nSame] == line[pos]; ++nSame);
+			for (nSame = 1; line[pos + nSame] == line[pos]; ++nSame);
 
 			if (inCodeChar != otherC && nSame >= 3)
 			{

If you are o.k., could you include the above change to 9e602a6?

Could you do mkdir parser-markdown.r/backquote.d?
The directory includes

input.md:

  `O`
# A

args.ctags:

--sort=no

(though there is only one tag entry.)

expected.tags:

A	input.md	/^# A$/;"	c

Could you add this new test case to 9e602a6, too?

[masatake]

The next one is an extra request.
The original regex parser doesn't implement the request. So this is not a must item for merging.

Is it possible to skip <!-- ... --> area?

<!--
# SKIP ME 1
-->

<!-- # SKIP ME 2 -->

# EXTRACT ME 1

[techee]

I found the root cause in 9e602a6.
If you are o.k., could you include the above change to 9e602a6?

Oh, yeah, totally, I didn't pay enough attention when updating the code, thanks for noticing, will fix.

Is it possible to skip area?

It should be no problem for the simple cases you provided but it would be much harder for something like

<!-- # SKIP ME 2 --> # INCLUDE ME <!--
# SKIP ME 3 --> # INCLUDE ME

I'll provide a patch for the simple cases only (it will basically be what is done for ``` right now).

[masatake]

Thank you. I tried the new changes.

It seems that we can expect <!-- is at the beginning of lines.
However, we cannot expect --> is not.

With the following change, tags output became better:

diff --git a/parsers/markdown.c b/parsers/markdown.c
index effd72601..f8b664a3a 100644
--- a/parsers/markdown.c
+++ b/parsers/markdown.c
@@ -256,8 +256,7 @@ static void findMarkdownTags (void)
                        lineProcessed = true;
                }
                /* XML comment end */
-               else if (lineLen >= pos + 3 && line[pos] == '-' && line[pos + 1] == '-' &&
-                       line[pos + 2] == '>')
+               else if (inComment && strstr ((const char *)(line + pos), "-->"))
                {
                        inComment = false;
                        lineProcessed = true;

Could you include this change to 69be461?

[techee]

@masatake Yep, makes sense, done.

[masatake]

Thank you.