Add conversations API #8832

Gargron · 2018-09-29T23:35:21Z

Replacement for computationally-heavy and unsatisfying direct timelines API. The new API would list DM conversations, rather than individual DMs, which allows for UX that people are more used to.

REST API: GET /api/v1/conversations
Streaming API: New event conversation in the direct stream

ClearlyClaire

So, I'm not completely sure what conversations are supposed to be here. I gather that they are threads of direct messages and are associated with a given number of participants. Multiple conversations with the same set of participants seem possible. Is that on purpose? It seems like a toot can be part of multiple conversations (through adding/removing participants). Is that also on purpose?

Finally, I'm very confused at how participants are computed, there might be a bug, there? To me, participants would be the author of the toot + all mentioned users. It seems that in this case, it's all mentioned users and not the author of the toot? I think there might be a bug there.

(Haven't reviewed front-end or streaming API changes)

ClearlyClaire · 2018-10-03T12:04:59Z

app/models/conversation_account.rb

+    end
+
+    def add_status(recipient, status)
+      conversation = find_or_initialize_by(account: recipient, conversation_id: status.conversation_id, participant_account_ids: participants_from_status(status))


It's somewhat unlikely to happen, but I'm worried about race conditions here.

ClearlyClaire · 2018-10-03T12:12:09Z

app/models/conversation_account.rb

+    private
+
+    def participants_from_status(status)
+      (status.mentions.pluck(:account_id) - [status.account_id]).sort


The toot's author isn't part of the conversation… ?

ClearlyClaire · 2018-10-03T12:14:59Z

app/models/conversation_account.rb

+      while (last_status_id = conversation.status_ids.pop)
+        last_status = Status.find(last_status_id)
+        break if last_status
+      end


I find this loop slightly confusing. Maybe a comment should be added to make clear that it only removes “orphaned” items? Also, why not remove “orphaned” items regardless of whether the last toot was removed?

not sure we even need a loop here? If you're just removing a single status at a time, there's no reason for any other status to be in conversation.status_ids but not actually exist in the database, right?

This whole thing would be less confusing with a normal rails has_many relationship

Because it only removes toots from the array if the removed toot is the last one. This is to track the last status, which should always be the visible one.

Indeed, we don't update the record if a deleted status is not the last status, therefore the array could contain IDs of statuses that no longer exist.

ClearlyClaire · 2018-10-03T19:47:35Z

app/models/conversation_account.rb

+    end
+
+    def add_status(recipient, status)
+      conversation = find_or_initialize_by(account: recipient, conversation_id: status.conversation_id, participant_account_ids: participants_from_status(recipient, status))


(Re-stating my concern about race conditions as github marked my comment as obsolete)

Not sure I understand. Shouldn't find_or_initialize_by be atomic?

find_or_createis not atomic: https://apidock.com/rails/v4.0.2/ActiveRecord/Relation/find_or_create_by
But here, it's even worse, we are using find_or_initialize_by, which means the database record is only created when we call save a few lines further.
There is a possibility multiple incoming toots for a same new conversations would be processed in parallel. This could for instance occur if sidekiq is down while someone sends us a string of DMs, which could then be processed at the same time when sidekiq goes up, resulting in two conversations for the same threads.

nightpool · 2018-10-05T02:43:24Z

app/models/conversation_account.rb

+#  last_status_id          :bigint(8)
+#
+
+class ConversationAccount < ApplicationRecord


I think AccountConversations makes at least a little bit more sense here.

You're right, that makes perfect sense, and I hate how much I'm gonna need to search and replace here oof

nightpool

Incomplete, sorry :(

nightpool · 2018-10-05T02:45:36Z

app/controllers/api/v1/conversations_controller.rb

+  end
+
+  def pagination_params(core_params)
+    params.slice(:limit).permit(:limit).merge(core_params)


I don't think we need both slice and permit here? Just permit should be nearly identical

Slice is required to avoid warnings in the log about unpermitted parameters being passed.

nightpool · 2018-10-05T02:46:17Z

app/controllers/api/v1/conversations_controller.rb

+    set_pagination_headers(next_path, prev_path)
+  end
+
+  def next_path


is all this pagination boilerplate really necessary? can't it be abstracted into a module?

nightpool · 2018-10-05T02:56:48Z

app/javascript/mastodon/features/direct_timeline/components/conversations_list.js

+    this._selectChild(elementIndex);
+  }
+
+  _selectChild (index) {


all the other methods in this class are shorthand lambda types, so this one probably should be as well? Or vice-versa?

Because this method is called from other methods rather being an event handler directly, there are no issues with this binding that the other syntax is meant to solve.

yeah, the lambda form is probably better but we should be consistent throughout the file (and ideally throughout the project), lexical vs function target this is a hard distinction to remember in the moment.

Event handlers use =, everything else uses the normal class method form.

nightpool · 2018-10-05T03:25:32Z

app/models/conversation_account.rb

+  def push_to_streaming_api
+    return unless subscribed_to_timeline?
+
+    if destroyed?


unless destroyed?, remove the comment

It's a stub. There probably should be a delete event for these. The problem is event naming. It would have to be called something ugly like "conversation_delete"

nightpool · 2018-10-05T03:25:42Z

app/javascript/mastodon/reducers/notifications.js

@@ -69,7 +69,7 @@ const expandNormalizedNotifications = (state, notifications, next) => {
    }

    if (!next) {
-      mutable.set('hasMore', true);
+      mutable.set('hasMore', false);


It's a bug I found when copying the code to conversations reducer. I think it's relevant.

nightpool · 2018-10-05T03:26:43Z

app/models/conversation_account.rb

+  belongs_to :last_status, class_name: 'Status'
+
+  def participant_account_ids=(arr)
+    self[:participant_account_ids] = arr.sort


does find_or_initialize_by bypass this?

No, it doesn't

nightpool · 2018-10-05T03:31:56Z

app/models/conversation_account.rb

+    end
+
+    def add_status(recipient, status)
+      conversation = find_or_initialize_by(account: recipient, conversation_id: status.conversation_id, participant_account_ids: participants_from_status(recipient, status))


Not sure I understand. Shouldn't find_or_initialize_by be atomic?

nightpool · 2018-10-05T03:34:51Z

app/models/conversation_account.rb

+      while (last_status_id = conversation.status_ids.pop)
+        last_status = Status.find(last_status_id)
+        break if last_status
+      end


not sure we even need a loop here? If you're just removing a single status at a time, there's no reason for any other status to be in conversation.status_ids but not actually exist in the database, right?

This whole thing would be less confusing with a normal rails has_many relationship

nightpool · 2018-10-05T03:35:58Z

app/models/conversation_account.rb

+#  account_id              :bigint(8)
+#  conversation_id         :bigint(8)
+#  participant_account_ids :bigint(8)        default([]), not null, is an Array
+#  status_ids              :bigint(8)        default([]), not null, is an Array


Not sure I understand why we're using postgres arrays here instead of two has_many tables. It makes the model code somewhat brittle and hard to understand

Not sure about participant_account_ids, but afaik, status_ids makes sense because it's ordered and only used to track the latest status in a thread more efficiently when deleting statuses.
(Although it probably doesn't handle toots received out-of-order that well, hm)

nightpool · 2018-10-05T05:38:53Z

app/models/domain_block.rb

+#  updated_at     :datetime         not null
+#  severity       :integer          default("silence")
+#  reject_media   :boolean          default(FALSE), not null
+#  reject_reports :boolean          default(FALSE), not null


unrelated to this pull?

dced625#diff-2ca536250498005a761e8b1427fe2515R12

ClearlyClaire · 2018-10-06T08:10:11Z

Do conversations with blocked users disappear from the list? It doesn't seem so by having a quick look at the code.

ClearlyClaire · 2018-10-07T10:08:02Z

I'm still worried about possible race conditions, but otherwise, this looks fine.
I will probably try that next week on my instance.
What about existing conversations (prior to the migration), though?

rhaamo · 2018-10-07T10:23:00Z

That's really great !

I only have one concern about the fact that blocking or muting an user would remove the actual conversations : in a context of harrassment it could be useful to not delete thoses conversations but rather kept them, but still having the possibility to delete a whole conversation afterward. (you block the user or mute and then deal with reporting with the actual conversation accessible..)

Also if it stays like that (deleting conv on mute/block) this should be indicated in various block/mute(/mute instance maybe too ?) popups !

ClearlyClaire · 2018-10-07T10:33:50Z

Hm, yeah, having an API to delete conversations instead of deleting them on block/mute would be nice (sorry about my conflicting earlier proposal). Shouldn't be hard to do either, although I'm not sure how the UI would look like.

EDIT: It would also solve the issue about deleting conversations where some participant has been blocked and some haven't been, since it's not clear what to do in that situation.

nightpool · 2018-10-07T14:52:36Z

we could add the rails lock_version column to avoid race conditions

…

On Sun, Oct 7, 2018 at 6:33 AM ThibG ***@***.***> wrote: Hm, yeah, having an API to delete conversations instead of deleting them on block/mute would be nice (sorry about my conflicting earlier proposal). Shouldn't be hard to do either, although I'm not sure how the UI would look like. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8832 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAORV6g01Kwuj21Yz0QWAeGcN335ciF2ks5uidiQgaJpZM4XAut0> .

Gargron · 2018-10-07T16:22:29Z

I only have one concern about the fact that blocking or muting an user would remove the actual conversations : in a context of harrassment it could be useful to not delete thoses conversations but rather kept them, but still having the possibility to delete a whole conversation afterward. (you block the user or mute and then deal with reporting with the actual conversation accessible..)

Block & mute-with-notifications has always deleted notifications from the user. I think it would be strange if the conversations were not cleaned up the same way. The latest behaviour in the PR is consistent with everything else, I believe.

we could add the rails lock_version column to avoid race conditions

There might be no need, the docs say lock_version only works within a single Ruby process anyway so if anything we would need SELECT ... FOR UPDATE

Gargron · 2018-10-07T16:23:18Z

EDIT: It would also solve the issue about deleting conversations where some participant has been blocked and some haven't been, since it's not clear what to do in that situation.

The block behaviour hides posts where a blocked person is even just mentioned, so that seems already decided.

nightpool · 2018-10-07T16:33:46Z

what? lock_version definitely works accross ruby processes..... https://api.rubyonrails.org/classes/ActiveRecord/Locking/Optimistic.html

…

On Sun, Oct 7, 2018, 12:23 PM Eugen Rochko ***@***.***> wrote: EDIT: It would also solve the issue about deleting conversations where some participant has been blocked and some haven't been, since it's not clear what to do in that situation. The block behaviour hides posts where a blocked person is even just mentioned, so that seems already decided. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8832 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAORV_zJ3L9Wb9xBq6vmLW5s9lpYIoADks5uiip4gaJpZM4XAut0> .

Gargron · 2018-10-07T16:44:36Z

This locking mechanism will function inside a single Ruby process. To make it work across all web requests, the recommended approach is to add lock_version as a hidden field to your form.

Gargron · 2018-10-07T16:48:16Z

Anyway, what I was saying, DM conversations usually don't have so many participants that an update happens within the same second to the point a race condition would actually occur. But if it does, I am ready to add a locking mechanism in the future.

I just want to merge this now

ClearlyClaire · 2018-10-07T18:32:24Z

As I said, a likely possibility is getting two successive DMs from the same person. Depending on the load, they could very well be processed concurrently.

nightpool · 2018-10-07T21:38:54Z

streaming/index.js

@@ -525,9 +526,11 @@ const startWorker = (workerId) => {
      ws.isAlive = true;
    });

+    let channel;


Not sure I understand this change. Why make it mutable? We never fall through any of the switch cases, right?

A few of the switch cases needed to reuse the same assembled string (channel) in two places at once, but a switch clause is not a separate scope, so you can't redefine a const there. Therefore, it's easiest to define the channel up there.

nightpool · 2018-10-07T21:40:02Z

LGTM!

Gargron added api REST API, Streaming API, Web Push API work in progress Not to be merged, currently being worked on labels Sep 29, 2018

Gargron force-pushed the feature-conversations-api branch 3 times, most recently from 51fe688 to 946008c Compare October 2, 2018 23:47

ClearlyClaire self-requested a review October 3, 2018 11:06

ClearlyClaire reviewed Oct 3, 2018

View reviewed changes

Gargron force-pushed the feature-conversations-api branch from 946008c to 869c7bd Compare October 3, 2018 17:24

ClearlyClaire reviewed Oct 3, 2018

View reviewed changes

Gargron force-pushed the feature-conversations-api branch from 6c118e7 to a1b16cc Compare October 3, 2018 22:34

Gargron removed the work in progress Not to be merged, currently being worked on label Oct 3, 2018

nightpool reviewed Oct 5, 2018

View reviewed changes

Gargron added 5 commits October 5, 2018 18:49

Add conversations API

b091fa6

Add web UI for conversations

b6d5a8a

Add test for conversations API

31b9f21

Add tests for ConversationAccount

3af3da4

Improve web UI

74b20c8

Gargron force-pushed the feature-conversations-api branch from 2b045e3 to 74b20c8 Compare October 5, 2018 16:50

Rename ConversationAccount to AccountConversation

bbfcf66

Gargron added 2 commits October 7, 2018 03:57

Remove conversations on block and mute

edba9f0

Change last_status_id to be a denormalization of status_ids

8cb3f84

Gargron force-pushed the feature-conversations-api branch from 5556e3a to 8cb3f84 Compare October 7, 2018 02:31

Add optimistic locking

1d25ed0

nightpool reviewed Oct 7, 2018

View reviewed changes

nightpool approved these changes Oct 7, 2018

View reviewed changes

Gargron merged commit 774ac47 into master Oct 7, 2018

Gargron deleted the feature-conversations-api branch October 7, 2018 21:45

mayaeh mentioned this pull request Oct 8, 2018

Fix error when pinning the DM column #8922

Merged

neet mentioned this pull request Oct 23, 2018

Mastodon 2.6.0: Conversation API and min_id support neet/masto.js#18

Merged

This was referenced Oct 25, 2018

DMs column does not paginate #9097

Closed

optimize direct message 2: add mentions.direct column and some partial index #7649

Closed

amyspark mentioned this pull request Dec 6, 2018

Add Mastodon 'conversation' event h3poteto/megalodon#21

Closed

hiromi-mi mentioned this pull request Feb 25, 2020

Add Support for Conversations API mattn/go-mastodon#126

Merged

Add conversations API #8832

Add conversations API #8832

Conversation

Gargron commented Sep 29, 2018 • edited Loading

ClearlyClaire left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nightpool left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ClearlyClaire commented Oct 6, 2018

ClearlyClaire commented Oct 7, 2018

rhaamo commented Oct 7, 2018

ClearlyClaire commented Oct 7, 2018 • edited Loading

nightpool commented Oct 7, 2018 via email

Gargron commented Oct 7, 2018

Gargron commented Oct 7, 2018

nightpool commented Oct 7, 2018 via email

Gargron commented Oct 7, 2018

Gargron commented Oct 7, 2018

ClearlyClaire commented Oct 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nightpool commented Oct 7, 2018

Gargron commented Sep 29, 2018 •

edited

Loading

ClearlyClaire commented Oct 7, 2018 •

edited

Loading