[DESIGN] Implement changes to bidi to permit non-strict formation

[aphillips]

This PR contains the design document for bidi handling at the syntax level. It contains a fairly comprehensive description of the problem space with examples.

[macchiati]

It is maybe a Pro (but more of a possibly-mitigating factor for a Con)

…

On Tue, Jul 23, 2024 at 10:57 AM Addison Phillips ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In exploration/bidi-usability.md <#811 (comment)> : > -literal-expression = "{" [LRI] [s] literal [s annotation] *(s attribute) [s] [close-isolate] "}" -variable-expression = "{" [LRI] [s] variable [s annotation] *(s attribute) [s] [close-isolate] "}" -annotation-expression = "{" [LRI] [s] annotation *(s attribute) [s] [close-isolate] "}" +**Cons** +- Can be used irresponsibly, including enabling some Trojan Source cases (UAX#55) This would be better as a pro perhaps? "Irresponsibly" might be the wrong word in the 'con', although it smacks of code attacks, which is the idea. This design enables the right thing by also enabling many wrong things. On the other hand, a lot of the time we'll be lucky if tools implement anything or users bother to insert the isolates/marks. — Reply to this email directly, view it on GitHub <#811 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMGGMYDR3VDMY5NUUR3ZN2KP7AVCNFSM6AAAAABJ2W265OVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCOJUGY4DEMJUGY> . You are receiving this because you commented.Message ID: ***@***.***>

[catamorphism]

I can't tell whether to read this as "paired enclosing marks (that the Unicode Bidirectional Algorithm pairs for shaping purposes)" or "uses paired enclosing marks (that the Unicode Bidirectional Algorithm pairs) for shaping purposes."

[aphillips]

It's the former.

Suggested change

	for shaping purposes and these offer a poor person's form of isolation.
	- The syntax uses enclosing marks (specifically curly brackets) which the Unicode Bidirectional Algorithm
	pairs up for shaping purposes, resulting in a weak form of isolation in the syntax itself.

[aphillips]

Done.

[catamorphism]

Should this be part of a bullet list, along with the other "Permit..." imperatives that follow?

[aphillips]

The paragraph on line 331 should probably move below the paragraph "We only permit..." because it has a different meaning and refers to all isolates. Not sure about using a bullet list?

[aphillips]

Done. Created the bullet list and reorganized.

[catamorphism]

I only see one difference?

[aphillips]

I should do the TODO, which makes it clearer. I'll do one here in the comment for clarity and then go back and fix the PR.

The current design has expression thusly:

expression = "{" [LRI] (literal-expression / variable-expression / annotation-expression) [close-isolate] "}"

This alternative would turn that into:

expression = "{" (literal-expression / variable-expression / annotation-expression) "}"
           / "{" LRI (literal-expression / variable-expression / annotation-expression) close-isolate "}"

In this formulation, you cannot have unpaired opening (or closing) isolates without a syntax error, nor can you have multiples of open or close.

Rinse and repeat for markup, option, attribute, and literals.

Make sense?

[aphillips]

Fixed the TODO

[eemeli]

I would be interested in hearing from @catamorphism, @mihnita, @lucacasonato and any other implementers about whether they would have any concerns about including bidi isolation characters within the name ABNF rule, as proposed in the alternative I'm naming here as "Isolate name rather than unquoted-literal".

This would not change the parsed meaning of a name in any way, but only include the bidi isolation or mark characters within the ABNF production so that they do not need to be mentioned in all the places where a name is used, as is the case in the current proposed solution.

@aphillips and I have discussed this, and as our opinions on this diverge, getting comments from other people who'll need to adjust their parsers due to the changes here would be helpful.

[lucacasonato]

Prefix: I am very unfamiliar with bidi - the linked introduction at the start of this doc was a great help. So thanks for linking that.

My impression is that the proposed ABNF seems reasonable. I do not think this would make my parser significantly more complex.

I make this statement on the following assumptions that I have based on my understanding of this document. Please correct them if they are wrong.

• isolation/bidi characters are in no way stored in an AST / data model
• isolation/bidi character placement is not meant to round-trip through parse-serialize
• a serializer can make a fully automatic determination based just on the AST / data model as to where what isolation/bidi chars need to be placed in the output
• a parser does not need to validate the correct pairing of bidi chars

---

I would be interested to understand further how likely it is that by editing a message that contains isolation/bidi chars, a user could produce a message that ends up being either:

• syntactically invalid
• has isolation/bidi chars in the middle of a pattern or quoted literal, only because they backspace / copy pasted etc

I guess what I am asking is how text (and code) editors usually deal with isolation/bidi chars: if I have a message that has an expresion containing a PDI just before the }, and I remove (backspace) the }, then manually move my cursor to the column trailing the last visible character in the expression, and type }, would the PDI now be in the pattern or would it be gone?

[catamorphism]

I would be interested in hearing from @catamorphism, @mihnita, @lucacasonato and any other implementers about whether they would have any concerns about including bidi isolation characters within the name ABNF rule, as proposed in the alternative I'm naming here as "Isolate name rather than unquoted-literal".

This would not change the parsed meaning of a name in any way, but only include the bidi isolation or mark characters within the ABNF production so that they do not need to be mentioned in all the places where a name is used, as is the case in the current proposed solution.

@aphillips and I have discussed this, and as our opinions on this diverge, getting comments from other people who'll need to adjust their parsers due to the changes here would be helpful.

I don't have concerns about this... or maybe I will once I try implementing it, but I can't think of anything right now.

[aphillips]

@catamorphism Thanks. My disagreement with @eemeli about including isolates and directional marks inside names (that are not considered part of the name) is that this would require parsers to process each name to remove these characters. I think this is relatively high impact for very little reward.

We are discussing in another thread potentially doing NFC normalization for comparison purposes, but this doesn't require walking the buffer (and can be fast-checked and otherwise optimized). Prohibiting isolates and strong marks in names will permit a few corner cases in which the namespace and name display awkwardly but no cases in which false matches are produced. Users have a much better workaround ("choose unidirectional names") available.

Note: I would like to merge this PR and then have a technical discussion of the design.

[eemeli]

Note: I would like to merge this PR and then have a technical discussion of the design.

I'd be fine with that, provided that the suggestion from #811 (comment) or something like it is included to document the remaining aspect of this that we disagree on.

[aphillips]

@eemeli I inserted the comment with edits and additions. Please have a look and see that I've captured the pros/cons to your satisfaction or if you have additions.

---

For the record, I think I'm leaning towards the "hybrid approaches" design. It would mean that MF2 messages might have spillover effects, but would encourage implementations to (re)serialize messages in ways that eliminate such effects. Looking forward to merge-and-discuss.

[eemeli]

This looks fine to merge. One line comment below, but it's not a blocker.

[eemeli]

Given how unquoted-literal is defined as

message-format-wg/spec/message.abnf

Line 48 in 01f2880

unquoted-literal = name / number-literal

and number-literal only contains LTR characters

message-format-wg/spec/message.abnf

Line 50 in 01f2880

number-literal = ["-"] (%x30 / (%x31-39 *DIGIT)) ["." 1*DIGIT] [%i"e" ["-" / "+"] 1*DIGIT]

allowing name isolation should cover all the unquoted-literal cases that may need isolation.

[aphillips]

Ur... this is something I don't like, since isolates are "not included" in the name, but are baked into the production. I'd rather do more with the ABNF to keep them separate (the isolates and directional characters are in the same places, but part of the surrounding goo instead of being part of the name-like token).

Note that number-literal contains only neutral and weakly directional characters (except for e, which is strongly LTR). Note that the leading - can switch sides in an RTL context (unless isolated or protected with LRM)