Attempting to make an IDE with an AST (gone wrong)

[Brayconn]

So, I've been working on an IDE for a proprietary scripting language, but I've encountered some issues...

I started work by making an editor backend that stores the document as an Abstract Syntax Tree and handles all editing.
I did this for a few reasons:

1. Separating the backend from the frontend is good practice™
2. I want to have a full AST of the document for more advanced IDE features (error checking, enhanced input suggestions, code analysis, etc.)
3. Due to details of the language, editing the document has a number of side-effects that are very difficult to compensate for without full control over how input is processed.
   • For example, attempting to type a newline character (\n) could result in a new line, or it could result in the text "\x0A" being entered.
   • Similarly, deleting characters could result in other characters after the delete location turning from their regular form to their escaped form (or vice versa).
   • This behavior also extends to unicode/multi-byte characters, non-printable ASCII, etc.

Of course, doing it this way means I can't use any of AvaloniaEdit's built in highlighting options, since there's no control over the input or the text/AST storage, but that's ok, since my next plan was to just assign TextEditor.Document to some wrapper class that redirected input/data lookup to my own class.

...And it was at this point that I realized TextEditor.Document is of type TextDocument, not IDocument, and that TextDocument is sealed, so there's no way to swap anything out at all...

So, my questions are:

1. Is there some alternate way of implementing my class into TextEditor that I missed?
2. Can I use some part(s) of AvaloniaEdit to make a separate frontend for my class, or would I just have to use base Avalonia components? I tried looking into how AvaloniaEdit's presenters worked, but I didn't find much (though, I am very much a noob to Avalonia, so maybe I'm missing something obvious).
3. Is forking the repo and switching TextEditor.Document to use some new interface a reasonable idea? (I actually attempted this already, but it seemed like a huge a mess to do, so I gave up. I will resume it if it's the best/only option though...)

Related note: the frontend of my program is currently stuck on Avalonia 10 due to a dependency, so I'd prefer if any solutions worked on that version, but updating to 11 is inevitable, so it's not a hard requirement.

[Washi1337]

Instead of switching out Document with a custom implementation, there may be an option to make your own subclasses of DocumentColorizingTransformer and VisualLineElementGenerator, that highlight and create new visual elements respectively according to a list of TextSegment (or a custom subclass of it).

For example, for a custom colorizer, you could use the following skeleton setup:

public class MyColorizingTransformer : DocumentColorizingTransformer
{
    public TextSegmentCollection<TextSegment> Highlights { get; set; } = new();

    protected override void ColorizeLine(DocumentLine line)
    {
        foreach (var segment in Highlights.FindOverlappingSegments(line))
        {
            IBrush brush = GetSegmentColor(segment); // TODO: implement coloring logic here.

            if (brush is not null)
            {
                ChangeLinePart(
                    Math.Max(line.Offset, segment.StartOffset),
                    Math.Min(line.EndOffset, segment.EndOffset),
                    element => element.TextRunProperties.SetForegroundBrush(brush));
            }
        }
    }
}

Once you constructed the AST, you could traverse it and determine the parts that needs highlighting. Then, you can populate the Highlights property of this transformer to have it colorize properly. Finally, you can add it as a LineTransformer to your text editor control:

TextEditor editor = ...

var lineTransformer = new MyColorizingTransformer();
lineTransformer.Highlights = ...; // TODO: determine by using the AST.
editor.TextArea.TextView.LineTransformers.Insert(0, lineTransformer);

If you want more interactivity with the highlighted text segments (e.g., tooltips, making them clickable etc.), you will need a custom visual line element generator. Something like the following could work:

public class MyElementGenerator : VisualLineElementGenerator
{
    public TextSegmentCollection<TextSegment> Highlights { get; set; } = new();
    
    public override int GetFirstInterestedOffset(int startOffset) => FindFirstSegmentWithStartAfter(startOffset)?.StartOffset ?? -1;

    public override VisualLineElement? ConstructElement(int offset)
    {
        foreach (var segment in Highlights.FindSegmentsContaining(offset))
        {
            int endOffset = Math.Min(segment.EndOffset, CurrentContext.VisualLine.LastDocumentLine.EndOffset);
            if (offset < endOffset)
                return new MyVisualLineText(CurrentContext.VisualLine, endOffset - offset, ...);
        }

        return null;
    }
}

public class MyVisualLineText : VisualLineText
{
   public MyVisualLineText(VisualLine parent, int length, ...) : base(parent, length) { /* ... */ }

   protected override void OnPointerPressed(PointerPressedEventArgs e) { /* handle click event */ }
}

Similar to the line colorizer, you need to register it as an element generator:

var generator = new MyElementGenerator();
generator.Highlights = ...;
editor.TextArea.TextView.ElementGenerators.Insert(0, generator);