MSC2529: Proposal to use existing events as captions for images #2529

benparsons · 2020-05-07T14:51:13Z

regexident · 2020-05-07T15:28:50Z

FYI: The developers of Ditto and Nio have expressed interest in this feature.

proposals/2529-text-messages-as-captions.md

tulir · 2020-05-07T16:00:41Z

proposals/2529-text-messages-as-captions.md

+
+* Not sure how this relates to the broader questions discussed in MSC1849
+* This is catering to a narrow use-case requirement. There may be a more general solution available
+* Would MSC1767 (extensible events) obsolete this?


Extensible events would definitely include captions, but IMO basic features like captions shouldn't be blocked on MSCs that aren't going to happen any time soon

uhoreg · 2020-05-08T03:14:14Z

proposals/2529-text-messages-as-captions.md

+  },
+```
+
+If a client recognises the `rel_type`, they can render the caption with the


What should a client do if someone tries to caption a message that isn't m.image?

Maybe they should be allowed to, but at render time, there is no change, since the rendering client will ignore the rel_type if the target isn't an image. Not sure.

proposals/2529-text-messages-as-captions.md

Sorunome · 2020-05-08T07:23:52Z

proposals/2529-text-messages-as-captions.md

@@ -0,0 +1,44 @@
+# Use existing m.room.message/m.text events as captions for images


Also, splitting image and caption into separate events will make bridging very complicated. When to send the image? How long until the caption is sent, then?

it'll still be practically impossible to bridge an image+caption away from matrix, the caption would basically have to be rendered as a separate message.

Would that be very bad? For example, bridging to Discord would result in two messages next to each other if they cannot be combined by debouncing.

Yes, as messages can be sent in between the image and the caption the entire caption context is lost.

Soru personally considers this a blocker, TBH.

Easier bridging (and handling captions in general) is one of the main benefits of #2530 btw

+1 to embed the caption in the image event content

turt2live · 2020-07-01T05:17:43Z

proposals/2529-text-messages-as-captions.md

+    "msgtype": "m.text",
+    "m.relates_to": {
+            "event_id": "$(some image event)",
+            "rel_type": "m.caption"


iirc this is a further metadata leak in encrypted rooms.

turt2live · 2020-07-01T05:18:27Z

proposals/2529-text-messages-as-captions.md

+    "msgtype": "m.text",
+    "m.relates_to": {
+            "event_id": "$(some image event)",
+            "rel_type": "m.caption"


not sure this is the right type under MSC1849: there was fairly long discussion at some point about how we shouldn't need to define new relationship types very often, if at all.

matrix-org#2530 and matrix-org#2529

MurzNN · 2020-12-01T10:00:34Z

@benparsons What do you think about merging this MSC with my MSC2881: Message Attachments? MSC2881 have similar implementation like yours, but also allow to group several images into one gallery with one text.

MurzNN · 2020-12-01T10:07:51Z

Also I believe that term "Image caption" in separate event is not so suitable for chatrooms, because most of images in chats are screenshots, memes, gifs, random photos from internet, etc, that need not "caption", but the text comment or description from sender.

The "Caption" term is good for long articles, which have the long text and inline image between text (aligned to left / right), that demonstrate some significant thing, related to some phrase in this long text. In that situation image really need a caption, that describes "What is this?".

But in chat rooms users don't compose long text articles, there are usually short one-two-lined text. So the images, posted between those text lines, don't need the caption, they need the description or comment from sender.

And real caption, that can be needed for chatrooms, is short alt text for images, to understand "what is in this image?" for visually impaired users, but for this task better to send the alt text together with image, like in MSC2530: Body field as media caption, not via separate event.

So maybe better to rename "Caption" to "Description" in your MSC?

uhoreg · 2020-12-01T14:45:36Z

that need not "caption", but the text comment or description from sender.

A "caption" is a "text comment or description from the sender".

MurzNN · 2020-12-01T15:47:40Z

that need not "caption", but the text comment or description from sender.

A "caption" is a "text comment or description from the sender".

This is true, but for chatrooms in most cases, it is the image that is the comment to the text message, and not vice versa (not text is comment to image, which can be named "image caption"), here are examples:

Can anybody help me - when I press "Save" button - I see the popup with error, here is screenshot:
[image.png]

Sorry, I'm late today, I will be at 12:30 in office, please wait!
[meme-i-m-late.gif]

Which bag is looks better for me?
[photo1.jpg] [photo2.jpg]

What is the color of this dress?
[dress-photo.jpg]

In those cases there are too hard to name the text as "Caption to image", because the text is main thing, the image is addition to it.

Also, when chat apps displays those texts as regular caption to image (text with decreased size below the image, inside image border, or even shown only on hover), especially when text have many lines - this looks terrible!

So, naming text field for picture as "Description" or "Text" instead of "Image caption" will prevent problems with implementing bad UI for display and compose interfaces in Matrix clients.

And if we need to keep together with this - also the real "Capture" field for images (for real capture to image, not comment + describing image), better is to have it inline to m.image event, like described in #2530

KitsuneRal · 2022-06-13T09:15:26Z

This MSC doesn't seem to have any future - if we go with a tactical fix before extensible events land, #2530 is considerably lower-impact.

@mscbot fcp close

mscbot · 2022-06-13T09:15:28Z

Team member @KitsuneRal has proposed to close this. The next step is review by the rest of the tagged people:

Once at least 75% of reviewers approve (and there are no outstanding concerns), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for information about what commands tagged team members can give me.

MurzNN · 2022-06-14T06:31:14Z

proposals/2529-text-messages-as-captions.md

+
+## Background
+
+There is a demand to be able to apply a text caption to an image, as is


There are already several MSC about adding text to images (this one MSC2529, MSC2530, MSC2881, MSC2674, maybe more? please point to them), but we can split their ideas to two types:

Classic behavior: media + caption: this is about adding text description, that comments the media. It suit well for articles, where we already have a long text and media (image, video, file) is just an item of the whole story, and it needs a short text description. In that cases it can be named "media caption" or "media description".

Chat behavior: text + media(s): This is a common usage of media in chat style conversations, where we have short text messages and a single media item (or series of several medias), that adds some information to that short text. So actually the text is the main entity, and the media is just an addition to it. The examples can be:

Here is my photos from last weekend [image1, image2]

My payment is rejected, here is the file with the rejection message: [file.pdf]

I've got the "Access denied" error, screenshot is attached. How can I solve this problem?

Also very often we have not a single media, but several ones (two or more photos, several following screenshots, etc) and the "media + caption" behavior doesn't suite for such cases at all!

So I want to discuss the whole idea that the classic media caption implementation does not suit well for Matrix ecosystem (and other chat systems too), and that the text + media(s) behavior suits much better for chat style conversations. I agree that sometimes people really want to post a media and add caption to it (for people with disabilities, or to comment what is displayed on the image), but in most cases the behavior is totally backwards!

Please share your own thoughts about this idea!

And maybe discuss this in some better place, could anyone suggest it?

Seems the better place to discuss it is here element-hq/roadmap#13

mscbot · 2022-07-12T12:28:06Z

🔔 This is now entering its final comment period, as per the review above. 🔔

mscbot · 2022-07-17T12:28:16Z

The final comment period, with a disposition to close, as per the review above, is now complete.

proposal to use existing events as captions for images

f40d175

regexident mentioned this pull request May 7, 2020

Image captions niochat/nio#112

Open

turt2live reviewed May 7, 2020

View reviewed changes

proposals/2529-text-messages-as-captions.md Outdated Show resolved Hide resolved

turt2live added kind:feature MSC for not-core and not-maintenance stuff proposal A matrix spec change proposal proposal-in-review labels May 7, 2020

line-wrapping

720b76f

turt2live self-requested a review May 7, 2020 15:46

tulir reviewed May 7, 2020

View reviewed changes

there are different types of space

531afe8

tulir mentioned this pull request May 7, 2020

MSC2530: Body field as media caption #2530

Merged

uhoreg reviewed May 8, 2020

View reviewed changes

auscompgeek reviewed May 8, 2020

View reviewed changes

proposals/2529-text-messages-as-captions.md Show resolved Hide resolved

Sorunome reviewed May 8, 2020

View reviewed changes

proposals/2529-text-messages-as-captions.md Show resolved Hide resolved

Sorunome reviewed May 8, 2020

View reviewed changes

bmarty mentioned this pull request Jun 4, 2020

Images with "attached" text doesn't show on RiotX (while does on desktop) element-hq/element-android#1454

Open

turt2live reviewed Jul 1, 2020

View reviewed changes

deepbluev7 mentioned this pull request Nov 27, 2020

MSC2881: Message Attachments #2881

Open

MurzNN added a commit to MurzNN/matrix-doc that referenced this pull request Nov 27, 2020

Added mentions of similar MSC

8065221

matrix-org#2530 and matrix-org#2529

MurzNN added a commit to MurzNN/matrix-doc that referenced this pull request Nov 30, 2020

Added proposal for replacement to matrix-org#2530 / matrix-org#2529

2e8536e

Rebeccataylor17 approved these changes Dec 3, 2020

View reviewed changes

MurzNN mentioned this pull request Feb 28, 2021

MSC1767: Extensible event types & fallback in Matrix (v2) #1767

Merged

turt2live added the needs-implementation This MSC does not have a qualifying implementation for the SCT to review. The MSC cannot enter FCP. label Jun 8, 2021

turt2live force-pushed the old_master branch from e895827 to dca99ee Compare August 30, 2021 22:34

auscompgeek mentioned this pull request May 27, 2020

Images with captions matrix-org/matrix-spec#242

Open

turt2live removed the proposal-in-review label May 5, 2022

mscbot added disposition-close proposed-final-comment-period Currently awaiting signoff of a majority of team members in order to enter the final comment period. labels Jun 13, 2022

MurzNN reviewed Jun 14, 2022

View reviewed changes

MurzNN mentioned this pull request Jun 17, 2022

Image galleries & captions element-hq/roadmap#13

Open

mscbot added final-comment-period This MSC has entered a final comment period in interest to approval, postpone, or delete in 5 days. and removed proposed-final-comment-period Currently awaiting signoff of a majority of team members in order to enter the final comment period. labels Jul 12, 2022

mscbot closed this Jul 17, 2022

mscbot added finished-final-comment-period and removed disposition-close final-comment-period This MSC has entered a final comment period in interest to approval, postpone, or delete in 5 days. labels Jul 17, 2022

turt2live added rejected A proposal which has been rejected for inclusion in the spec and removed finished-final-comment-period labels Jul 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSC2529: Proposal to use existing events as captions for images #2529

MSC2529: Proposal to use existing events as captions for images #2529

benparsons commented May 7, 2020 •

edited by turt2live

Loading

regexident commented May 7, 2020

tulir May 7, 2020

uhoreg May 8, 2020

benparsons May 11, 2020

Sorunome May 8, 2020

benparsons May 11, 2020

Sorunome May 11, 2020

tulir May 11, 2020

bmarty Jun 4, 2020

turt2live Jul 1, 2020

turt2live Jul 1, 2020

MurzNN commented Dec 1, 2020

MurzNN commented Dec 1, 2020 •

edited

Loading

uhoreg commented Dec 1, 2020

MurzNN commented Dec 1, 2020 •

edited

Loading

KitsuneRal commented Jun 13, 2022

mscbot commented Jun 13, 2022 •

edited by erikjohnston

Loading

MurzNN Jun 14, 2022 •

edited

Loading

MurzNN Jun 17, 2022

mscbot commented Jul 12, 2022

mscbot commented Jul 17, 2022

		@@ -0,0 +1,44 @@
		# Use existing m.room.message/m.text events as captions for images


		## Background

		There is a demand to be able to apply a text caption to an image, as is

MSC2529: Proposal to use existing events as captions for images #2529

MSC2529: Proposal to use existing events as captions for images #2529

Conversation

benparsons commented May 7, 2020 • edited by turt2live Loading

regexident commented May 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MurzNN commented Dec 1, 2020

MurzNN commented Dec 1, 2020 • edited Loading

uhoreg commented Dec 1, 2020

MurzNN commented Dec 1, 2020 • edited Loading

KitsuneRal commented Jun 13, 2022

mscbot commented Jun 13, 2022 • edited by erikjohnston Loading

MurzNN Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mscbot commented Jul 12, 2022

mscbot commented Jul 17, 2022

benparsons commented May 7, 2020 •

edited by turt2live

Loading

MurzNN commented Dec 1, 2020 •

edited

Loading

MurzNN commented Dec 1, 2020 •

edited

Loading

mscbot commented Jun 13, 2022 •

edited by erikjohnston

Loading

MurzNN Jun 14, 2022 •

edited

Loading