Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Offload chat messages #9995

Merged
merged 1 commit into from
Feb 14, 2020
Merged

Offload chat messages #9995

merged 1 commit into from
Feb 14, 2020

Conversation

cammellos
Copy link
Contributor

@cammellos cammellos commented Feb 6, 2020

  1. Messages are offloaded from any chat once we go back from the home.
    This allows us to ignore any message that is coming in from a chat we
    are not currently focused.
  2. After 5 seconds of not-scrolling activity, any received message that
    is not currently visible will be offloaded to the database.
  3. Similarly received messages that are not visible will be offloaded to
    the database directly

This should improve performance in the following areas:

  1. Receiving messages when on the home should be much faster and less slow downs on the app
  2. Joining a chat with a lot of messages (i.e spam) should be much faster, as long as the user does not keep scrolling up as messages become visible

The drawback is that when you go from home -> to a chat, you won't see the messages immediately (it will always be the same as it is now when you first go to a chat after login, as messages will be loaded from the database), but this is likely necessary (or will be changed) anyway once we implement scroll to last not seen message.

status: ready

@cammellos cammellos requested a review from a team as a code owner February 6, 2020 15:17
@cammellos cammellos self-assigned this Feb 6, 2020
@ghost
Copy link

ghost commented Feb 6, 2020

Pull Request Checklist

  • Docs: Updated the documentation, if affected
  • Docs: Added or updated inline comments explaining intention of the code
  • Tests: Ensured that all new UI elements have been assigned accessibility IDs
  • Tests: Signaled need for E2E tests with label, if applicable
  • Tests: Briefly described what was tested and what platforms were used
  • UI: In case of UI changes, ensured that UI matches Figma
  • UI: In case of UI changes, requested review from a Core UI designer
  • UI: In case of UI changes, included screenshots of implementation

@status-im-auto
Copy link
Member

status-im-auto commented Feb 6, 2020

Jenkins Builds

Click to see older builds (15)
Commit #️⃣ Finished (UTC) Duration Platform Result
1b87819 #1 2020-02-06 15:28:17 ~10 min ios 📄log
1b87819 #1 2020-02-06 15:32:02 ~13 min android-e2e 📄log
1b87819 #1 2020-02-06 15:32:16 ~14 min android 📄log
✔️ 77663f3 #2 2020-02-06 15:46:26 ~12 min ios 📦ipa 📲
✔️ 77663f3 #2 2020-02-06 15:49:18 ~15 min android 📦apk 📲
✔️ 77663f3 #2 2020-02-06 15:49:18 ~15 min android-e2e 📦apk 📲
✔️ 753eca6 #3 2020-02-07 12:11:31 ~11 min ios 📦ipa 📲
✔️ 753eca6 #3 2020-02-07 12:20:54 ~21 min android 📦apk 📲
✔️ 753eca6 #3 2020-02-07 12:20:54 ~21 min android-e2e 📦apk 📲
✔️ 03b3b01 #4 2020-02-10 09:32:21 ~11 min ios 📦ipa
✔️ 03b3b01 #4 2020-02-10 09:35:51 ~15 min android 📦apk 📲
✔️ 03b3b01 #4 2020-02-10 09:35:51 ~15 min android-e2e 📦apk 📲
✔️ 388f898 #6 2020-02-10 11:53:51 ~10 min ios 📦ipa
✔️ 388f898 #6 2020-02-10 11:54:51 ~11 min android-e2e 📦apk 📲
✔️ 388f898 #6 2020-02-10 11:55:23 ~12 min android 📦apk 📲
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 62654db #7 2020-02-14 11:33:01 ~11 min ios 📦ipa 📲
✔️ 62654db #7 2020-02-14 11:36:57 ~15 min android-e2e 📦apk 📲
✔️ 62654db #7 2020-02-14 11:36:57 ~15 min android 📦apk 📲
a22f85e #8 2020-02-14 14:54:27 ~3 min ios 📄log
a22f85e #8 2020-02-14 14:55:34 ~4 min android 📄log
a22f85e #8 2020-02-14 14:55:41 ~4 min android-e2e 📄log

@cammellos cammellos force-pushed the performance/offload-chat-messages branch from 77663f3 to 753eca6 Compare February 7, 2020 11:59
@cammellos cammellos force-pushed the performance/offload-chat-messages branch 3 times, most recently from ebb97ed to 388f898 Compare February 10, 2020 11:43
@statustestbot
Copy link

99% of end-end tests have passed

Total executed tests: 97
Failed tests: 1
Passed tests: 96

Failed tests (1)

Click to expand
1. test_remove_member_from_group_chat

Device 2: Looking for a message by text: 'This message should not be visible for device 2'
Device 2: Wait for ChatElementByText

Message with text '@Longterm Glossy Xantus left the group' was not received

Device sessions

Passed tests (96)

Click to expand
1. test_can_see_all_transactions_in_history
Device sessions

2. test_install_pack_and_send_sticker
Device sessions

3. test_send_message_in_group_chat
Device sessions

4. test_login_with_new_account
Device sessions

5. test_clear_history_of_group_chat_via_group_view
Device sessions

6. test_timestamp_in_chats
Device sessions

7. test_password_in_logcat_sign_in
Device sessions

8. test_logcat_recovering_account
Device sessions

9. test_modify_transaction_fee_values
Device sessions

10. test_public_chat_clear_history
Device sessions

11. test_need_help_section
Device sessions

12. test_send_and_open_links
Device sessions

13. test_mobile_data_usage_popup_stop_syncing
Device sessions

14. test_mobile_data_usage_settings
Device sessions

15. test_create_new_group_chat
Device sessions

16. test_sign_typed_message
Device sessions

17. test_user_can_switch_network
Device sessions

18. test_mobile_data_usage_popup_continue_syncing
Device sessions

19. test_home_view
Device sessions

20. test_open_google_com_via_open_dapp
Device sessions

21. test_send_message_to_newly_added_contact
Device sessions

22. test_share_contact_code_and_wallet_address
Device sessions

23. test_make_admin_member_of_group_chat
Device sessions

24. test_connection_is_secure
Device sessions

25. test_add_to_contacts
Device sessions

26. test_add_account_to_multiaccount_instance
Device sessions

27. test_can_open_dapp_from_dapp_store
Device sessions

28. test_refresh_button_browsing_app_webview
Device sessions

29. test_long_press_to_delete_1_1_chat
Device sessions

30. test_can_add_existing_ens
Device sessions

31. test_transaction_wrong_password_wallet
Device sessions

32. test_public_chat_messaging
Device sessions

33. test_unread_messages_counter_public_chat
Device sessions

34. test_delete_one_to_one_chat_via_delete_button
Device sessions

35. test_long_press_to_delete_public_chat
Device sessions

36. test_text_message_1_1_chat
Device sessions

37. test_delete_public_chat_via_delete_button
Device sessions

38. test_can_use_purchased_stickers_on_recovered_account
Device sessions

39. test_open_transaction_on_etherscan
Device sessions

40. test_onboarding_screen_when_requesting_tokens_for_recovered_account
Device sessions

41. test_contact_profile_view
Device sessions

42. test_logcat_backup_recovery_phrase
Device sessions

43. test_delete_group_chat_via_delete_button
Device sessions

44. test_start_chat_with_ens
Device sessions

45. test_add_and_remove_contact_from_public_chat
Device sessions

46. test_open_blocked_site
Device sessions

47. test_open_chat_by_pasting_public_key
Device sessions

48. test_ens_username_recipient
Device sessions

49. test_token_with_more_than_allowed_decimals
Device sessions

50. test_fetch_more_history_in_empty_chat
Device sessions

51. test_open_public_chat_using_deep_link
Device sessions

52. test_switch_users_and_add_new_account
Device sessions

53. test_copy_contact_code_and_wallet_address
Device sessions

54. test_send_transaction_from_daap
Device sessions

55. test_send_two_transactions_one_after_another_in_dapp
Device sessions

56. test_message_marked_as_sent_in_1_1_chat
Device sessions

57. test_long_press_delete_clear_all_dapps
Device sessions

58. test_copy_and_paste_messages
Device sessions

59. test_deploy_contract_from_daap
Device sessions

60. test_back_forward_buttons_browsing_website
Device sessions

61. test_password_in_logcat_creating_account
Device sessions

62. test_add_custom_token
Device sessions

63. test_sign_message_from_daap
Device sessions

64. test_group_chat_system_messages
Device sessions

65. test_dapps_permissions
Device sessions

66. test_request_public_key_status_test_daap
Device sessions

67. test_logcat_send_transaction_from_wallet
Device sessions

68. test_log_level_and_fleet
Device sessions

69. test_add_and_delete_watch_only_account_to_multiaccount_instance
Device sessions

70. test_recover_account_from_new_user_seedphrase
Device sessions

71. test_user_can_complete_tx_to_dapp_when_onboarding_via_dapp_completed
Device sessions

72. test_send_two_transactions_in_batch_in_dapp
Device sessions

73. test_ens_in_public_and_1_1_chats
Device sessions

74. test_block_user_from_public_chat
Device sessions

75. test_fetching_balance_after_offline
Device sessions

76. test_logcat_sign_message_from_daap
Device sessions

77. test_offline_status
Device sessions

78. test_wallet_set_up
Device sessions

79. test_manage_assets
Device sessions

80. test_add_new_group_chat_member
Device sessions

81. test_offline_messaging_1_1_chat
Device sessions

82. test_decline_invitation_to_group_chat
Device sessions

83. test_account_recovery_with_uppercase_recovery_phrase
Device sessions

84. test_send_token_with_7_decimals
Device sessions

85. test_user_can_see_all_own_assets_after_account_recovering
Device sessions

86. test_pass_phrase_validation
Device sessions

87. test_send_emoji
Device sessions

88. test_filters_from_daap
Device sessions

89. test_redirect_to_public_chat_tapping_tag_message
Device sessions

90. test_pair_devices_sync_one_to_one_contacts
Device sessions

91. test_logcat_send_transaction_from_daap
Device sessions

92. test_send_eth_from_wallet_to_address
Device sessions

93. test_send_stt_from_wallet
Device sessions

94. test_messaging_in_different_networks
Device sessions

95. test_collectible_from_wallet_opens_in_browser_view
Device sessions

96. test_insufficient_funds_wallet_positive_balance
Device sessions

@statustestbot
Copy link

99% of end-end tests have passed

Total executed tests: 97
Failed tests: 1
Passed tests: 96

Failed tests (1)

Click to expand
1. test_remove_member_from_group_chat

Device 2: Looking for a message by text: 'This message should not be visible for device 2'
Device 2: Wait for ChatElementByText

Message with text '@Fine Warlike Feline left the group' was not received

Device sessions

Passed tests (96)

Click to expand
1. test_delete_public_chat_via_delete_button
Device sessions

2. test_request_public_key_status_test_daap
Device sessions

3. test_open_public_chat_using_deep_link
Device sessions

4. test_decline_invitation_to_group_chat
Device sessions

5. test_ens_username_recipient
Device sessions

6. test_delete_one_to_one_chat_via_delete_button
Device sessions

7. test_offline_status
Device sessions

8. test_open_transaction_on_etherscan
Device sessions

9. test_open_chat_by_pasting_public_key
Device sessions

10. test_back_forward_buttons_browsing_website
Device sessions

11. test_password_in_logcat_creating_account
Device sessions

12. test_can_use_purchased_stickers_on_recovered_account
Device sessions

13. test_modify_transaction_fee_values
Device sessions

14. test_insufficient_funds_wallet_positive_balance
Device sessions

15. test_mobile_data_usage_settings
Device sessions

16. test_delete_group_chat_via_delete_button
Device sessions

17. test_open_google_com_via_open_dapp
Device sessions

18. test_logcat_backup_recovery_phrase
Device sessions

19. test_unread_messages_counter_public_chat
Device sessions

20. test_send_two_transactions_one_after_another_in_dapp
Device sessions

21. test_message_marked_as_sent_in_1_1_chat
Device sessions

22. test_can_open_dapp_from_dapp_store
Device sessions

23. test_user_can_switch_network
Device sessions

24. test_public_chat_clear_history
Device sessions

25. test_wallet_set_up
Device sessions

26. test_timestamp_in_chats
Device sessions

27. test_group_chat_system_messages
Device sessions

28. test_fetch_more_history_in_empty_chat
Device sessions

29. test_mobile_data_usage_popup_continue_syncing
Device sessions

30. test_add_to_contacts
Device sessions

31. test_dapps_permissions
Device sessions

32. test_long_press_delete_clear_all_dapps
Device sessions

33. test_need_help_section
Device sessions

34. test_transaction_wrong_password_wallet
Device sessions

35. test_offline_messaging_1_1_chat
Device sessions

36. test_token_with_more_than_allowed_decimals
Device sessions

37. test_text_message_1_1_chat
Device sessions

38. test_install_pack_and_send_sticker
Device sessions

39. test_make_admin_member_of_group_chat
Device sessions

40. test_add_account_to_multiaccount_instance
Device sessions

41. test_send_emoji
Device sessions

42. test_copy_and_paste_messages
Device sessions

43. test_clear_history_of_group_chat_via_group_view
Device sessions

44. test_send_eth_from_wallet_to_address
Device sessions

45. test_messaging_in_different_networks
Device sessions

46. test_start_chat_with_ens
Device sessions

47. test_logcat_recovering_account
Device sessions

48. test_user_can_complete_tx_to_dapp_when_onboarding_via_dapp_completed
Device sessions

49. test_connection_is_secure
Device sessions

50. test_user_can_see_all_own_assets_after_account_recovering
Device sessions

51. test_add_new_group_chat_member
Device sessions

52. test_pair_devices_sync_one_to_one_contacts
Device sessions

53. test_add_and_remove_contact_from_public_chat
Device sessions

54. test_send_transaction_from_daap
Device sessions

55. test_onboarding_screen_when_requesting_tokens_for_recovered_account
Device sessions

56. test_long_press_to_delete_1_1_chat
Device sessions

57. test_add_and_delete_watch_only_account_to_multiaccount_instance
Device sessions

58. test_open_blocked_site
Device sessions

59. test_refresh_button_browsing_app_webview
Device sessions

60. test_public_chat_messaging
Device sessions

61. test_pass_phrase_validation
Device sessions

62. test_send_token_with_7_decimals
Device sessions

63. test_sign_message_from_daap
Device sessions

64. test_send_message_in_group_chat
Device sessions

65. test_deploy_contract_from_daap
Device sessions

66. test_recover_account_from_new_user_seedphrase
Device sessions

67. test_add_custom_token
Device sessions

68. test_send_and_open_links
Device sessions

69. test_manage_assets
Device sessions

70. test_share_contact_code_and_wallet_address
Device sessions

71. test_redirect_to_public_chat_tapping_tag_message
Device sessions

72. test_block_user_from_public_chat
Device sessions

73. test_ens_in_public_and_1_1_chats
Device sessions

74. test_sign_typed_message
Device sessions

75. test_create_new_group_chat
Device sessions

76. test_password_in_logcat_sign_in
Device sessions

77. test_account_recovery_with_uppercase_recovery_phrase
Device sessions

78. test_send_message_to_newly_added_contact
Device sessions

79. test_logcat_sign_message_from_daap
Device sessions

80. test_mobile_data_usage_popup_stop_syncing
Device sessions

81. test_collectible_from_wallet_opens_in_browser_view
Device sessions

82. test_logcat_send_transaction_from_daap
Device sessions

83. test_contact_profile_view
Device sessions

84. test_switch_users_and_add_new_account
Device sessions

85. test_logcat_send_transaction_from_wallet
Device sessions

86. test_send_two_transactions_in_batch_in_dapp
Device sessions

87. test_filters_from_daap
Device sessions

88. test_send_stt_from_wallet
Device sessions

89. test_login_with_new_account
Device sessions

90. test_home_view
Device sessions

91. test_log_level_and_fleet
Device sessions

92. test_can_add_existing_ens
Device sessions

93. test_copy_contact_code_and_wallet_address
Device sessions

94. test_long_press_to_delete_public_chat
Device sessions

95. test_fetching_balance_after_offline
Device sessions

96. test_can_see_all_transactions_in_history
Device sessions

@Serhy
Copy link
Contributor

Serhy commented Feb 12, 2020

@cammellos
I tested on Galaxy Note 4 (Android 6.0.1)
Yes, as you @cammellos stated receiving messages when on the home is faster (just don't see whether it slowing less downs on the app because nightly is responsive as well when navigating between tabs while messages are receiving):
Receiving 100 of stickers in public chat from time the unread counter is 1 till it's 99+ takes:
Nightly build: ~14s (video: https://drive.google.com/file/d/125S5alyBtIzNcvL4o5S4DMQ1rXmyxr24/view?usp=sharing)
PR build: ~11.5s (video: https://drive.google.com/file/d/15pnqb3gIK5J5hbm_Sc1oN7qhxZOHggQN/view?usp=sharing)

The same is on navigating to chat - redirect happens faster, though there is a delay on message appearing (not noticeable actually so fine)

This item is not really clear to me:

  • After 5 seconds of not-scrolling activity, any received message that
    is not currently visible will be offloaded to the database.
    So I opened a chat with 100 messages. Scrolled up to ~60th message. Waited ~15-20 sec. Then from another device I send text messages to this public chat. Staying in public chat it's visible messages are shifted up (the "dancing messages" bug when receiving message while view is in the middle of chat).
    Is that what 5-seconds not scrolling activity suppose to do?

@cammellos
Copy link
Contributor Author

@Serhy thanks for checking

After 5 seconds of not-scrolling activity, any received message that
Is a bit different, basically you open a chat with 100 messages for example, scroll up, then scroll down to the latest message, wait 5 seconds, scroll up again. You should notice that those messages are re-fetched from the database. This helps with memory consumption and rendering performance when receiving messages while you are on the chat.

Hope is clear, thanks for taking the time!

@Serhy
Copy link
Contributor

Serhy commented Feb 12, 2020

Hope is clear, thanks for taking the time!

Aha, I've got it, @cammellos !
Yes, actually this is the best part for me of this improvement :)
It's really noticeable when having 100+ messages (like join chat with 100+ new messages - scroll up like you read all of them and then when messaging in this same chat - there is a lag upon sending a message (due to we re-render all the message list)
But with 5-second non activity we almost always have to re-render last 20 messages and that's much better UX

@cammellos cammellos force-pushed the performance/offload-chat-messages branch from 388f898 to 62654db Compare February 14, 2020 11:21
@cammellos cammellos changed the title [WIP] offload chat messages Offload chat messages Feb 14, 2020
@errorists
Copy link
Contributor

@cammellos tested on Galaxy S8+, loaded both #status and #chitchat started scrolling as they were downloading messages. Opening chats works as expected, scroll is buttery smooth, didn't notice any lag or hiccup or other visual artefacts. Receiving messages on home looks ok, hard for me to compare if it's better or worse. anything else I should test?

@@ -334,6 +344,7 @@
:idx idx
:list-ref messages-list-ref}])
:inverted true
:onViewableItemsChanged on-viewable-items-changed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are the default good enough?
there is a bunch of visibility config for this for instance

:viewability-config (clj->js {:waitForInteraction false
                                                           :minimumViewTime 100
                                                           :viewAreaCoveragePercentThreshold 60})

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They should be ok

@@ -0,0 +1,3 @@
(ns status-im.ui.screens.chat.state)

(defonce viewable-item (atom nil))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be cleaned after logout?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can probably clean it up as we land on a chat, it's probably safer, but should get reset as soon as we load messages

@cammellos
Copy link
Contributor Author

@errorists that should be enough, thanks!

acc))
{}
(get-in db [:chats chat-id :messages]))]
(fx/merge cofx
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fx/merge and -> db looks redundant here

@flexsurfer flexsurfer requested a review from yenda February 14, 2020 14:26
@cammellos cammellos force-pushed the performance/offload-chat-messages branch 2 times, most recently from a22f85e to 4734a4e Compare February 14, 2020 14:56
@cammellos cammellos merged commit 4734a4e into develop Feb 14, 2020
This commit does a few things:

1) Messages are offloaded from any chat once we go back from the home.
This allows us to ignore any message that is coming in from a chat we
are not currently focused.
2) After 5 seconds of not-scrolling activity, any received message that
is not currently visible will be offloaded to the database.
3) Similarly received messages that are not visible will be offloaded to
the database directly

Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

7 participants