Missing messages and chats after quitting and re-opening

[chadyj]

User Story

As a user, I want messages to not vanish so that I can read them :).

Description

Type: Bug

Every day I've been chatting a lot with Status Desktop Mac in 1-1 and public chats, but after quitting and re-opening the app I've seen many messages vanish.

Sometimes I still see the chat channel (eg #status-desktop) but messages are removed. Sometimes all but 1 message is removed, other times there are seemingly random messages removed.

Other times the chat channel is gone too.

So far I haven't been able to notice a pattern.

It has happened both when I quit for the night, then install Status desktop in the morning.
Also happens when I quit for 20 minutes then re-open.

5 other people have reported this on Slack.

UPDATE: This is also happening on mobile

Expected behavior

Visible message remain in the app.

Actual behavior

Messages vanish.

Reproduction

I can't reliably repro. But try talking lots. Quitting. Then re-open the app.

Additional Information

• Status version: Nightlies for past few weeks
  comment: # (Android or iOS.)
• Operating System: Mac + iOS + Android

[chadyj]

From Slack:

Vitaliy Vlasov
wondering if this line https://github.com/status-im/status-react/blob/22119ccf27d65fa4e26449fec732c20ad124ca68/src/status_im/ui/screens/desktop/main/chat/views.cljs#L197

Vitaliy Vlasov [11 minutes ago]
might have something to do with it

Vitaliy Vlasov [11 minutes ago]
we had some issues when React diff algorithm would fail to compare distinct records

Vitaliy Vlasov [11 minutes ago]
and some entries would be missing from the chat view

Vitaliy Vlasov [10 minutes ago]
this might affect seemingly distinct entries, as far as i recall

[churik]

Eventually got the same problem: after reopening application (around 1 hour) some messages are lost.

Log: https://drive.google.com/file/d/1scrMBbrMdRVfIUabVaDXhNveFkUbAEmr/view?usp=sharing

But after reopening app chat histrory is fetched fully.
@chadyj does it working for you sometimes or vanished messages never go back?

[chadyj]

@chadyj does it working for you sometimes or vanished messages never go back?

On desktop messages are never re-fetched for me.

[chadyj]

Not sure if this is related, but I logged out, created a new account, logged out of that, and logged into my original account. Status seemed to be download messages again and reset all the read counters. Eg many have 9+ unread messages even though no are new and I saw them before I logged out.

[oskarth]

There's a serious problem with the database on both mobile and desktop, it doesn't remember contacts or history beyond some weird state, and when it's like this it gives you a strange 'topic' error when adding a contact -- Jarrad

Have same issue, seems to apply at least partially to mobile nightlies too.

[oskarth]

Same if you delete a contact and app quits, it appears to come back. We should make sure we preserve basic durability etc properties for our database, it seems like whatever is done now is rather flaky.

[chadyj]

From Slack:

Andrea Maria Piana [1 minute ago]
Yesterday I had something that might explain the problem, I received an error storing a chat in the database (login/logout) was not there
the specific error was :chat-id is missing , opening a public chat, it might be that those error are not displayed in prod builds (but not sure)

[churik]

@oskarth
#6114 (comment) this scenario is not working for me.
Steps:

1. delete 1-1 chat with user from contact list
2. quit app
3. start app
4. check chat list.
   Chat is still deleted and doesn't appear.
   Contact is still in contact list, and you can start chat again.
   AFAIK there is no way to delete contact from contact list.

Something is wrong with this behavior?

[oskarth]

Sorry, I meant 'delete chat'. Chat reappears.

[chadyj]

@churik This is happening to me pretty reliably now on both desktop and mobile.
Can you try reproducing by joining several busy public chats and creating several 1-1 chats and then send lots of messages.

On desktop I have 60 chats open on desktop and about 50 open on mobile.

It seems that saving becomes a problem when there are many many public channels and 1-1 messages. Perhaps the app is always doing something (fetching, etc) that it never gets around to saving. @rasom or @yenda mentioned something like this happens during the last desktop call.

[cammellos]

@oskarth chat re-appearing upon receiving a new message is the expected behavior (it's a bit counter intuitive but we had discussions about this in the past, basically we need a block action), can you remember if there was a new message received in that chat (expected behavior)? or it re-appeared with some previous messages/no messages (bug)?

[oskarth]

@cammellos Fairly certain it was with old previous messages. Though in any case that's less of an issue than messages/contacts disappearing, which seems to indicate something is seriously wrong with DB persistence logic.

[janherich]

@chadyj So far I have been unable to replicate the issue, but I created PR where suspected area of code (db handling) is done in synchronous way, which should help is investigate the issue, because:

1. Normally, the db writing is done in async manner, so db writes don't slow down the rest of the applications (it remains responsive). With db writes done synchronously, instead of dropping them, the application would just slow down and become laggy, so that's something to test.
2. Even if the above is not happening, doing write in synchronous way will provide better debugging info about failed writes/transactions.

Here is the PR - #6195

[churik]

Ok, there are my observations according to #6114 (comment).

• Preconditions:
  I have the same account, used on nightly 08/10/2018 (used on desktop and mobile)
  Number of public chats on desktop: 24, 1-1 chats: 14, deleted 1-1 chats: 2, deleted public chats: 1; unread messages on: 4 chats;
  Number of public chats on android: 16, 1-1 chats: 14 deleted 1-1 chats: 1, deleted public chats: 1;
  unread messages: on 21 chats

• Steps for desktop:

1. quit on build nightly 08/10/2018
2. wait for ~8 hours
3. install new nightly 09/10/2018
4. check chats, messages, unread messages

• Steps for mobile:

1. app in the background ~8 hours
2. check chats, messages, unread messages

All unread messages are left unread, chats don't reappear after deleting, chat list is preserved on mobile and desktop.

One strange thing I noticed that messages disappear on replies, so some events created previously on nightly 08/10/2018 are vanished.

I used separate user for test and compress all data, link for debug.
User: User_Reliabilty
Password: qwerty
cc @chadyj

[Serhy]

We figured out that one of the major reason why chats and/or messages disappear (or re-appear like in #6114 (comment)) is that after transport.topic error appears, which described in #5204, - app “stops” to keep further history. And once you re-login to account the history may look vanished.

For example your account has only #status public channel with message history. Then account faced #5204 issue (which is the soft error pop-up) but you keep using the app: you join 5 more new chats and received 100 new messages #status public chat.
Once you re-login to the account (with app restart, or re-login with running app) - the only history contact will see is #status public channel without 100 new messages (received after transport.topic).

In order to reproduce from scratch:

1. Open Status on Desktop app and create new account (Contact1)
2. Join several public chats from Selected for you list bu the Contact1
3. Remember recovery phrase of the Contact1
4. Open Status on Device 2 and create new account (Contact2)
5. Log out of Contact1 on Desktop app
6. Recover Contact1 account of Device3
7. Contact1(from Device3) adds Contact2 in contacts (e.g. via QR code scan) and sends a text message
8. Contact2 navigates to 1-1 chat with Contact1 and taps 'Add to contacts'
9. Contact2 sends several text messages to Contact1
10. Contact1 logs in to the account from Desktop App (Note: Contact1 receives text messages from Contact2 from offline inbox after login)
11. Contact1 (on Desktop App) taps 'Add to Contacts' Note: at this step the transport.topic error appears (described in the Missing value for property transport.topic in 1-1 chat for recovered account #5204)
12. Contact1 joins one more public chat from Desktop app (e.g. #cryptokitties-co)
13. Contact1 replies to Contact2 in 1-1 chat with the text message Hello who are you?
14. Reopen the Desktop app Contact1 used and log in to account.

Actual result: There is no #cryptokitties-co chat and Hello who are you? message added/sent in 12th and 13th step.

Contact1 keep using the app normally and all is OK(re-logins keeps the history), but there is a Contact2 in the list with persistent 'Add to Contacts' option. If you tap it again, #5204 bug may not even fire up again but all further history won't be kept on Desktop app.

[cammellos]

I think I know what's going on, looking into this (basically one of the devices has not received the initial contactrequest but has received potentially a contactupdate), trying to replicate

[churik]

@cammellos in case it can help: messages are vanished after #6210 as well.
So when I faced this after #6210 and keep using application, after relogin new messages are gone.

[cammellos]

@churik thanks, I think there are 2 issues, one is the error we see one that is more problematic is that if a write to the database fails, all consequent writes seems to be failing, I'll investigate.

[oskarth]

Awesome investigation and repro @Serhy