Implement Text Selection handles in TextBox for touch input #13107

emmauss · 2023-10-02T12:33:27Z

What does the pull request do?

Implements text selection handles in textbox that allow changing text selection by dragging. This is activated by non-mouse input, i.e. touch and pen. This provides similar text selection behavior to what users are familiar with on android and iOS.

Recording.2023-10-02.122506.mp4

In addition, the textbox context menu has been simplified and optimized for mobile.

Recording.2023-10-09.091959.mp4

Though what I've planned for this pr is all in, it's still subject to change, so comments on implementation details and recommendations are welcome.

What is the current behavior?

Currently, text can only be be selected by dragging the caret along the text. This makes it very inaccurate on touch devices, and text selection is easily reset by accidental touches.

What is the updated/expected behavior with this PR?

How was the solution implemented (if it's not obvious)?

SelectionHandles must be drawn over all other controls, and though visibility of the handles is affected by the current text cursor, it shouldn't be clipped. As such, a new visual layer with a higher z-index is used to host the handles. This has the side effect of selection handles drawing over hosted popups, which are drawn in the OverlayLayer. It's possible to add text controls to the OverlayLayer, as such, that layer is not suitable to host selection handles.

Checklist

Added unit tests (if possible)?
Added XML documentation to any related classes?
Consider submitting a PR to https://github.com/AvaloniaUI/avalonia-docs with user documentation

Breaking changes

This is a behavioral breaking change.

Obsoletions / Deprecations

Fixed issues

avaloniaui-team · 2023-10-02T13:03:41Z

You can test this PR using the following package version. 11.0.999-cibuild0040386-beta. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

robloo · 2023-10-02T14:16:52Z

Shouldn't this be done with adorners? They are specifically designed for cases like this.

I'm a little uncomfortable with how this was implemented with new canvas controls overlaying everything else. It doesn't seem to match XAML ideas.

The graphical design looks good though and this feature is a big plus for mobile!

emmauss · 2023-10-02T14:30:35Z

Shouldn't this be done with adorners? They are specifically designed for cases like this.

I'm a little uncomfortable with how this was implemented with new canvas controls overlaying everything else. It doesn't seem to match XAML ideas.

The graphical design looks good though and this feature is a big plus for mobile!

Adorners currently can't be used in the any layer above the AdornerLayer, which includes OverlayLayer. It also has a composition bug, #10409 .

robloo · 2023-10-02T15:33:08Z

Adorners currently can't be used in the any layer above the AdornerLayer, which includes OverlayLayer

Why would the selection handles need to appear over the overlay layer? In Android they do the following:

Selection handles are BELOW the overlay layer which contains cut/copy/paste (this can be confirmed by moving the overlay manually). So they accept adorner below overlay layer in concept. This isn't a problem because:
When changing the position of the selection handles, the overlay cut/copy/paste controls hide themselves. They will re-appear after a few millisecond delay again when the selection handles are no longer being moved.
The overlay cut/copy/paste controls are shown in a Y-position below the selection handles. The position of the overlay needs to account for the selection handles (this alone solves most of the issues)

Someone can check iOS as well but simply copying Android would allow Adorner usage.

It also has a composition bug, #10409 .

There a few other issues for adorners as well: #10701 and #11232. However, those bugs should not drive a "hacky" solution for a core feature. The correct path is to fix adorners. We don't want a temporary solution that will stick around in the code base for years I think?

robloo · 2023-10-02T21:24:03Z

A few more comments:

It's possible to add text controls to the OverlayLayer, as such, that layer is not suitable to host selection handles.

I believe that's too severe of a limitation. Several input-type controls have a "popup" that will appear as an overlay on certain platforms. This would not allow modification of text in this scenario using the seleciton handles which seems to be a rather big architecture flaw. This indicates another approach is needed.
The Cut/Copy/Paste menu is going to need to be customized for mobile platforms I think. Instead of a vertical layout of the commands iOS and Android both use a horizontal layout. This is important due to mobile form factor and screen sizes. I think this is fairly easy to do with the new OnPlatform markup extension.
Looking at some iOS documentation it appears they also side-step this issue by placing the overlay cut/copy/paste buttons ABOVE the selection in most cases. Again using a Y-position offset to avoid any intersection.

emmauss · 2023-10-02T23:33:01Z

Adorners currently can't be used in the any layer above the AdornerLayer, which includes OverlayLayer

Why would the selection handles need to appear over the overlay layer? In Android they do the following:
1. Selection handles are BELOW the overlay layer which contains cut/copy/paste (this can be confirmed by moving the overlay manually). So they accept adorner below overlay layer in concept. This isn't a problem because:

2. When changing the position of the selection handles, the overlay cut/copy/paste controls hide themselves. They will re-appear after a few millisecond delay again when the selection handles are no longer being moved.

3. The overlay cut/copy/paste controls are shown in a Y-position below the selection handles. The position of the overlay needs to account for the selection handles (this alone solves most of the issues)
Someone can check iOS as well but simply copying Android would allow Adorner usage.

It also has a composition bug, #10409 .

There a few other issues for adorners as well: #10701 and #11232. However, those bugs should not drive a "hacky" solution for a core feature. The correct path is to fix adorners. We don't want a temporary solution that will stick around in the code base for years I think?

I know of 1 popular avalonia control toolkit that uses the overlay layler for managed dialogs, ie content dialogs. These dialogs can host text boxes. And the overlay layer is above the adorner layer.

emmauss · 2023-10-02T23:38:59Z

@robloo Android uses popups for their copy-paste menu. These are windows on their own, similar to desktop

robloo · 2023-10-03T01:31:36Z

Android uses popups for their copy-paste menu. These are windows on their own, similar to desktop

Ok, even if they are 'windows' rather than an overlay my point stands about z-index. The selection handles are underneath them as shown below. If it works for Android I have to think it would work for Avalonia.... however

I know of 1 popular avalonia control toolkit that uses the overlay layler for managed dialogs, ie content dialogs. These dialogs can host text boxes. And the overlay layer is above the adorner layer.

Yea, true, the adorner layer would still have to render above the overlay layer so the selection handles could appear in all cases like this. I was hoping this could be avoided but it can't - overlay hosted controls need selection handles too in some cases.

This also might be a little bit new territory since UWP didn't have adorners. Perhaps that's why they don't fully work in composition renderer now.

Perhaps the solution is to:

Make adorners work again
Render adorners above overlay layer. Actually, any adorner created in an overlay should just work so there is clearly a missing feature here.
Make cut/copy/paste buttons horizontal on mobile to save Y-axis space
Move cut/copy/paste buttons by:
a. Hiding them when selection handles are being dragged
b. Only show cut/copy/paste overlay above/below the selection box in the Y-axis

TL;DR: I still think adorners should be used. They were designed exactly for things like this.

emmauss · 2023-10-03T07:02:18Z

Android uses popups for their copy-paste menu. These are windows on their own, similar to desktop

Ok, even if they are 'windows' rather than an overlay my point stands about z-index. The selection handles are underneath them as shown below. If it works for Android I have to think it would work for Avalonia.... however

I know of 1 popular avalonia control toolkit that uses the overlay layler for managed dialogs, ie content dialogs. These dialogs can host text boxes. And the overlay layer is above the adorner layer.

Yea, true, the adorner layer would still have to render above the overlay layer so the selection handles could appear in all cases like this. I was hoping this could be avoided but it can't - overlay hosted controls need selection handles too in some cases.

This also might be a little bit new territory since UWP didn't have adorners. Perhaps that's why they don't fully work in composition renderer now.

Perhaps the solution is to:
1. Make adorners work again

2. Render adorners above overlay layer. Actually, any adorner created in an overlay should just work so there is clearly a missing feature here.

3. Make cut/copy/paste buttons horizontal on mobile to save Y-axis space

4. Move cut/copy/paste buttons by:
   a. Hiding them when selection handles are being dragged
   b. Only show cut/copy/paste overlay above/below the selection box in the Y-axis
TL;DR: I still think adorners should be used. They were designed exactly for things like this.

3 and 4 are on my list of things to do for touch. 1 and 2 will need @kekekeks input. I think OverlayLayer was added after Adorners, so it was placed above so as not to break stuff.

avaloniaui-team · 2023-10-09T08:36:55Z

You can test this PR using the following package version. 11.0.999-cibuild0040570-beta. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

emmauss · 2023-10-09T09:22:33Z

@robloo Added mobile themed context flyouts for textbox

avaloniaui-team · 2023-10-09T09:46:27Z

You can test this PR using the following package version. 11.0.999-cibuild0040578-beta. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

src/Avalonia.Controls/Primitives/TextSelectionHandle.cs

src/Avalonia.Controls/Primitives/TextSelectionCanvas.cs

src/Avalonia.Controls/Presenters/TextPresenter.cs

MrJul · 2023-10-09T10:39:27Z

src/Avalonia.Controls/TextBox.cs

@@ -1470,7 +1459,7 @@ protected override void OnPointerPressed(PointerPressedEventArgs e)
            var text = Text;
            var clickInfo = e.GetCurrentPoint(this);

-            if (text != null && clickInfo.Properties.IsLeftButtonPressed &&
+            if (text != null && (e.Pointer.Type == PointerType.Mouse || e.ClickCount >= 2) && clickInfo.Properties.IsLeftButtonPressed &&


I'm not sure what the new condition does, but it seems incompatible with the code inside the if branch: isTouch && e.ClickCount == 1 below can now never be true

Edit: this prevents setting the caret manually with touch without the handles once you have a selection.

Unresolving this as dragging the caret itself doesn't work (see my comment below).

src/Avalonia.Controls/TextBox.cs

src/Avalonia.Themes.Fluent/Controls/TextBox.xaml

MrJul · 2023-10-09T10:58:26Z

There a few other issues for adorners as well: #10701 and #11232. However, those bugs should not drive a "hacky" solution for a core feature. The correct path is to fix adorners. We don't want a temporary solution that will stick around in the code base for years I think?

I completely agree with this statement. We're adding a public layer here to work around adorners issues, but they're really made for this case. Even when adorners get fixed, the current solution might have to stay, which really isn't ideal.

Apart from that, this PR is a great and much needed improvement for mobile!

src/Avalonia.Controls/Primitives/TextSelectionCanvas.cs

avaloniaui-team · 2023-10-13T13:54:31Z

You can test this PR using the following package version. 11.0.999-cibuild0040840-beta. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

emmauss · 2023-10-19T14:50:20Z

2023-10-19-15-42-07.mp4

Here it is. Note that the caret still moves, but only when I'm releasing my finger. In all other apps, the caret moves with the finger (and displays a zoomed text, but that's a different improvement).

Ok, I didn't account for that. There may be issues with scrolling, but I'll try to alleviate it.

emmauss · 2023-10-23T08:10:50Z

To clarify the "with selection" part: once you have a selection, you can unselect/move the caret but it takes two taps: one to dismiss the context menu (that opens automatically) and another one to move the caret elsewhere. One tap should be sufficient to both close the menu and move the caret (at least that's the behavior in all other apps).

That said, I understand if you want that change to be part of another PR. The current behavior is already way, way better than before.

This behavior would need #13351 first, so will update after that's merged.

emmauss · 2023-10-27T08:15:59Z

@MrJul Should be good now. could you try again?

MrJul · 2023-10-28T13:33:13Z

@MrJul Should be good now. could you try again?

Just tried for a few minutes, touch text selection and caret moving feels quite good, kudos!

You even fixed the outer scrollviewer preventing proper touch scrolling inside textboxes after a while (#13351 I presume), which was quite annoying.

robloo · 2023-10-31T11:05:52Z

Awesome work again!

Finally, what was decided about adorners? Will this be adjusted later when adorners work in overlays and the bugs are fixed?

hez2010 · 2023-10-31T11:59:34Z

Several issues after testing on a real Android device:

It's very hard to enter the selection mode, in most cases the selection will be dismissed immediately.
On a multiple line textbox with a scrollbar, you cannot scroll the text area to adjust selection carets and the selection will dismiss after you scroll the textarea.

emmauss · 2023-10-31T12:53:35Z

Awesome work again!

Finally, what was decided about adorners? Will this be adjusted later when adorners work in overlays and the bugs are fixed?

Hopefully, yes.

emmauss · 2023-10-31T12:57:52Z

Several issues after testing on a real Android device:

* It's very hard to enter the selection mode, in most cases the selection will be dismissed immediately.

* On a multiple line textbox with a scrollbar, you cannot scroll the text area to adjust selection carets and the selection will dismiss after you scroll the textarea.

Selection mode is entered by either tapping the text to set the handle, then tapping the handle, or holding on the text. Drag selection for touch has been removed due to how it interferes with gestures.
Scrolling on a multiline textbox is done simply by moving the touch vertically when you press. Moving it horizontally immediately you press will disable scrolling for that pointer and instead allow changing the caret position. For a single line textbox that scrolls horizontally, using the handles to move selection is sufficient.

emmauss force-pushed the text_selector branch from 589f7b5 to b671597 Compare October 9, 2023 09:22