Future of CJK characters and CommonMark alignment

[flipacholas]

I understand this issue has been discussed before (#61) and it's even a fundamental problem with the specification (#488), which still hasn't been solved to this day.

I'm now dealing with markdown text written in Chinese Simplified and I found many instances where the ** and * don't work as expected. So, before I start adding hacks to fix my documents, I was wondering if there are plans for goldmark to deviate from commonmark rules and alter its parsing so CJK characters work without these problems. I'm willing to donate if needed.

Regards.

[yuin]

This is the well-known problem. CommonMark is designed by westerner people that uses space-separated languages and they often do not take account into languages like CJK(that are not separated by spaces) and Arabic(that is written from right to left).

This problem has been discussed for a long time, but there are no progress. I suspect that CommonMark authors(mostly westerners) are not interested in languages that they do not use.

I always give compatibilities a high priority, however, ​as a Japanese, I might plan deviating from CommonMark about emphasis future.

I think that introduce "escaped space"(like ReST) is a one good solution for this problem.

This makes 太郎は\ **「こんにちは」**\ と言った <p>太郎は<strong>「こんにちは」</strong>と言った</p>: escaped spaces will not be rendered.

This solution is very easy to implement in CommonMark parser.

[wdscxsj]

I think that introduce "escaped space"(like ReST) is a one good solution for this problem.

This is more than welcome!

[yuin]

@flipacholas @wdscxsj I've implemented a "escaped space"(like ReST) and "east asian line breaks"(like pandoc) and published v1.5.0 package.
I'm a Japanese and am not familiar with other languages like Chinese and Korean. Feedbacks from writers uses these languages are appreciated.

[wdscxsj]

Thanks for your great work! It seems the escaped space following the closing emphasis is not working as expected. I'll check the test code to see what goes wrong.

Update: It turns out this issue happens when WithEscapedSpace is used with Linkify.

By the way, it would be better if WithEastAsianLineBreaks also works with \r\n.

[yuin]

Thanks for your report. I've fixed the bug with linkfiy.

By the way, it would be better if WithEastAsianLineBreaks also works with \r\n.

Hmmm, my implementation do not care about newline code difference. So it should work for '\r\n'.

goldmark/renderer/html/html.go

Lines 654 to 667 in 1dd67d5

	} else if n.SoftLineBreak() {
	if r.EastAsianLineBreaks && len(value) != 0 {
	sibling := node.NextSibling()
	if sibling != nil && sibling.Kind() == ast.KindText {
	if siblingText := sibling.(*ast.Text).Text(source); len(siblingText) != 0 {
	thisLastRune := util.ToRune(value, len(value)-1)
	siblingFirstRune, _ := utf8.DecodeRune(siblingText)
	if !(util.IsEastAsianWideRune(thisLastRune) &&
	util.IsEastAsianWideRune(siblingFirstRune)) {
	_ = w.WriteByte('\n')
	}
	}
	}
	} else {

It just see if a node has a soft line break flag. Soft line break flags is set even if newline code is '\r\n'.

[wdscxsj]

Thanks for the fix! For the 2nd issue, I tried again and found the same culprit: WithEastAsianLineBreaks failed to work with '\r\n' when Linkify was also used. That is:

子曰\n诗云        => <p>子曰诗云</p>\n
子曰\r\n诗云      => <p>子曰\n诗云</p>\n

[yuin]

Thanks, I've fixed the bug.