Implement Parallel CI Test Runs

[Samyoul]

One avenue for reducing test run time #5066 can be to introduce Parallel CI Test Runs.

This could be achieved by distributing the tests across different CI machines, or running tests in parallel on the same machine, or a combination.

[igor-sirotin]

testify doesn't support in-suite parallelism: stretchr/testify#187
and testify/v2 is never happening: stretchr/testify#1089 (comment)

[igor-sirotin]

Just a breakdown of execution time by packages.

Note that go test runs packages in parallel by default, so 1331.567s is kinda best we can achieve now just by making tests parallel.

Also, I think my machine is a bit faster than CI. But that's fixable (hopefully not by downgrading my side 😄)

gotestsum result

[7/08/24 1:33:35] ➜  status-go git:(develop) gotestsum --format pkgname --format-hide-empty-pkg --hide-summary "skipped,failed,errors,output" --rerun-fails-max-failures=20 -- ./... -timeout=60m
✓  abi-spec (311ms)
✓  account/generator (446ms)
✓  api/multiformat (230ms)
✓  centralizedmetrics/providers (1.696s)
✓  appmetrics (2.644s)
✓  circuitbreaker (1.92s)
✓  appdatabase (3.283s)
✓  centralizedmetrics (2.543s)
✓  cmd/statusd (886ms)
✓  cmd/statusd/topics (1.099s)
✓  account (6.401s)
✓  connection (1.221s)
✓  db (1.101s)
✓  discovery (934ms)
✓  eth-node/crypto (455ms)
✓  exportlogs (629ms)
✓  logutils (872ms)
✓  metrics/node (280ms)
✓  multiaccounts (405ms)
✓  mobile (1.241s)
✓  mailserver (3.055s)
✓  multiaccounts/settings (1.988s)
✓  images (4.942s)
✓  params (316ms)
✓  notifier (550ms)
✓  multiaccounts/accounts (3.694s)
✓  multiaccounts/settings_wallet (3.461s)
✓  peers/verifier (2.036s)
✓  peers (2.733s)
✓  profiling (2.854s)
✓  protocol/anonmetrics (319ms)
✓  protocol/encryption/publisher (888ms)
✓  protocol/identity (507ms)
✓  protocol/encryption/sharedsecret (1.638s)
✓  protocol/ens (1.552s)
✓  protocol/identity/alias (281ms)
✓  protocol/identity/colorhash (384ms)
✓  protocol/encryption/multidevice (3.455s)
✓  protocol/identity/emojihash (588ms)
✓  protocol/identity/identicon (597ms)
✓  protocol/peersyncing (873ms)
✓  protocol/requests (315ms)
✓  protocol/sqlite (1.066s)
✓  protocol/common (9.165s)
✓  protocol/pushnotificationclient (3.238s)
✓  protocol/storenodes (1.022s)
✓  protocol/pushnotificationserver (3.295s)
✓  protocol/v1 (572ms)
✓  protocol/transport (1.262s)
✓  node (16.868s)
✓  protocol/zaputil (775ms)
✓  protocol/verification (1.274s)
✓  rpc/chain (749ms)
✓  protocol/urls (2.981s)
✓  rpc (1.012s)
✓  rpc/network (1.204s)
✓  server (2.059s)
✓  server/pairing/statecontrol (1.573s)
✓  server/pairing/peers (2.346s)
✓  services/appmetrics (2.154s)
✓  services/browsers (3.346s)
✓  services/communitytokens/communitytokensdatabase (858ms)
✓  server/pairing/preflight (4.42s)
✓  services/communitytokens (1.175s)
✓  services/connector (1.237s)
✓  services/connector/commands (959ms)
✓  services/connector/database (1.044s)
✓  services/ens (1.314s)
✓  services/ext (1.686s)
✓  services/gif (2.301s)
✓  services/peer (1.305s)
✓  services/ext/mailservers (3.953s)
✓  services/local-notifications (2.454s)
✓  services/mailservers (2.614s)
✓  services/permissions (736ms)
✓  services/rpcfilters (890ms)
✓  services/subscriptions (730ms)
✓  services/stickers (1.167s)
✓  services/typeddata (1.387s)
✓  protocol/encryption (30.87s)
✓  services/wallet/balance (237ms)
✓  services/wallet (1.182s)
✓  services/wallet/async (2.198s)
✓  protocol/communities (34.997s)
✓  services/wallet/bigint (2.899s)
✓  services/wallet/blockchainstate (2.589s)
✓  services/wallet/collectibles (1.439s)
✓  services/wallet/common (915ms)
✓  services/wakuext (4.464s)
✓  services/wallet/activity (4.507s)
✓  services/wallet/community (1.018s)
✓  services/wallet/currency (782ms)
✓  services/wallet/onramp (740ms)
✓  services/wallet/history (1.597s)
✓  services/wallet/market (1.733s)
✓  services/wallet/router (1.893s)
✓  services/wallet/thirdparty (1.668s)
✓  services/wallet/thirdparty/alchemy (2.127s)
✓  services/wallet/thirdparty/coingecko (2.386s)
✓  services/wallet/router/pathprocessor (3.333s)
✓  services/wallet/thirdparty/fourbyte (1.927s)
✓  services/wallet/thirdparty/fourbytegithub (2.098s)
✓  services/wallet/thirdparty/opensea (1.25s)
✓  services/wallet/thirdparty/paraswap (1.123s)
✓  services/wallet/thirdparty/rarible (588ms)
✓  services/wallet/token/balancefetcher (391ms)
✓  services/wallet/token (1.538s)
✓  services/wallet/walletconnect (494ms)
✓  services/web3provider (1.102s)
✓  signal (1.381s)
✓  sqlite (1.573s)
✓  timesource (353ms)
✓  transactions (1.615s)
✓  telemetry (2.238s)
✓  services/wallet/transfer (4.699s)
✓  waku/v0 (1.518s)
✓  waku/v1 (1.601s)
✓  wakuv2/common (988ms)
✓  waku (9.371s)
✓  waku/common (9.607s)
✓  wakuv2 (15.343s)
✓  server/pairing (1m8.473s)
✓  api (2m8.919s)
✓  protocol (19m14.506s)

DONE 2748 tests, 35 skipped, 3 failures in 1331.567s
[7/08/24 2:09:33] ➜  status-go git:(develop) 

[igor-sirotin]

Current packages test execution time

< 1 second - 22.823s

  230ms - api/multiformat
  237ms - services/wallet/balance
  280ms - metrics/node
  281ms - protocol/identity/alias
  311ms - abi-spec
  315ms - protocol/requests
  316ms - params
  319ms - protocol/anonmetrics
  353ms - timesource
  384ms - protocol/identity/colorhash
  391ms - services/wallet/token/balancefetcher
  405ms - multiaccounts
  446ms - account/generator
  455ms - eth-node/crypto
  494ms - services/wallet/walletconnect
  507ms - protocol/identity
  550ms - notifier
  572ms - protocol/v1
  588ms - protocol/identity/emojihash
  588ms - services/wallet/thirdparty/rarible
  597ms - protocol/identity/identicon
  629ms - exportlogs
  730ms - services/subscriptions
  736ms - services/permissions
  740ms - services/wallet/onramp
  749ms - rpc/chain
  775ms - protocol/zaputil
  782ms - services/wallet/currency
  858ms - services/communitytokens/communitytokensdatabase
  872ms - logutils
  873ms - protocol/peersyncing
  886ms - cmd/statusd
  888ms - protocol/encryption/publisher
  890ms - services/rpcfilters
  915ms - services/wallet/common
  934ms - discovery
  959ms - services/connector/commands
  988ms - wakuv2/common

< 2 seconds - 57.342s

  1.012s - rpc
  1.018s - services/wallet/community
  1.022s - protocol/storenodes
  1.044s - services/connector/database
  1.066s - protocol/sqlite
  1.099s - cmd/statusd/topics
  1.101s - db
  1.102s - services/web3provider
  1.123s - services/wallet/thirdparty/paraswap
  1.167s - services/stickers
  1.175s - services/communitytokens
  1.182s - services/wallet
  1.204s - rpc/network
  1.221s - connection
  1.237s - services/connector
  1.241s - mobile
  1.25s - services/wallet/thirdparty/opensea
  1.262s - protocol/transport
  1.274s - protocol/verification
  1.305s - services/peer
  1.314s - services/ens
  1.381s - signal
  1.387s - services/typeddata
  1.439s - services/wallet/collectibles
  1.518s - waku/v0
  1.538s - services/wallet/token
  1.552s - protocol/ens
  1.573s - sqlite
  1.573s - server/pairing/statecontrol
  1.597s - services/wallet/history
  1.601s - waku/v1
  1.615s - transactions
  1.638s - protocol/encryption/sharedsecret
  1.668s - services/wallet/thirdparty
  1.686s - services/ext
  1.696s - centralizedmetrics/providers
  1.733s - services/wallet/market
  1.893s - services/wallet/router
  1.92s - circuitbreaker
  1.927s - services/wallet/thirdparty/fourbyte
  1.988s - multiaccounts/settings

< 5 seconds - 1m43.399s

  2.036s - peers/verifier
  2.059s - server
  2.098s - services/wallet/thirdparty/fourbytegithub
  2.127s - services/wallet/thirdparty/alchemy
  2.154s - services/appmetrics
  2.198s - services/wallet/async
  2.238s - telemetry
  2.301s - services/gif
  2.346s - server/pairing/peers
  2.386s - services/wallet/thirdparty/coingecko
  2.454s - services/local-notifications
  2.543s - centralizedmetrics
  2.589s - services/wallet/blockchainstate
  2.614s - services/mailservers
  2.644s - appmetrics
  2.733s - peers
  2.854s - profiling
  2.899s - services/wallet/bigint
  2.981s - protocol/urls
  3.055s - mailserver
  3.238s - protocol/pushnotificationclient
  3.283s - appdatabase
  3.295s - protocol/pushnotificationserver
  3.333s - services/wallet/router/pathprocessor
  3.346s - services/browsers
  3.455s - protocol/encryption/multidevice
  3.461s - multiaccounts/settings_wallet
  3.694s - multiaccounts/accounts
  3.953s - services/ext/mailservers
  4.42s - server/pairing/preflight
  4.464s - services/wakuext
  4.507s - services/wallet/activity
  4.699s - services/wallet/transfer
  4.942s - images

> 5 seconds - 24m44.52s

  6.401s - account
  9.165s - protocol/common
  9.371s - waku
  9.607s - waku/common
  15.343s - wakuv2
  16.868s - node
  30.87s - protocol/encryption
  34.997s - protocol/communities
  1m8.473s - server/pairing
  2m8.919s - api
  19m14.506s - protocol

[igor-sirotin]

Just parallelising the tests will give us 20 minutes at best (protocol ~20min, all other packages are ~9min).
This is very good and should be done first thing.

Yet, I believe that test should run <5 minutes.
To achieve this, we should also spot slow things in protocol package. And probably split the tests into multiple packages.

[igor-sirotin]

We run those tests not in parallel for reasons unknown. It gives us 3,5 minutes runtim on top.
Obviously, api package is the most problematic here.

✓  api (2m28.963s) (coverage: 13.9% of statements in ./...)
✓  mailserver (2.67s) (coverage: 8.3% of statements in ./...)
✓  multiaccounts/settings (1.336s) (coverage: 10.2% of statements in ./...)
✓  node (20.105s) (coverage: 3.1% of statements in ./...)
✓  services/wakuext (3.973s) (coverage: 4.1% of statements in ./...)

DONE 115 tests in 211.096s

[igor-sirotin]

Running same tests in parallel doesn't seem to raise any problems 🤔

✓  mailserver (2.649s) (coverage: 8.3% of statements in ./...)
✓  multiaccounts/settings (3.961s) (coverage: 10.2% of statements in ./...)
✓  services/wakuext (4.353s) (coverage: 4.1% of statements in ./...)
✓  node (22.347s) (coverage: 3.1% of statements in ./...)
✓  api (2m25.173s) (coverage: 13.9% of statements in ./...)

DONE 115 tests in 154.805s

But maybe they are conflicting with the other tests.

UPD: They were conflicting with themselves when running with -test.count >= 1. It's now fixed.

[igor-sirotin]

Stats for protocol package tests run time: protocol-tests.log

[igor-sirotin]

The leaders of the protocol package:

PASS protocol.TestMessengerCommunitiesSuite (280.31s)
PASS protocol.TestMessengerProfilePictureHandlerSuite (141.64s)
PASS protocol.TestTokenMasterCommunityEventsSuite (104.18s)
PASS protocol.TestAdminCommunityEventsSuite (103.60s)
PASS protocol.TestMessengerContactRequestSuite (92.67s)
PASS protocol.TestOwnerWithoutCommunityKeyCommunityEventsSuite (84.83s)
PASS protocol.TestMessengerPushNotificationSuite (39.29s)
PASS protocol.TestMessengerCommunitiesSharedMemberAddressSuite (36.66s)
PASS protocol.TestMessengerSuite (36.43s)
PASS protocol.TestMessengerCommunitiesSignersSuite (20.51s)
PASS protocol.TestCommunityEventsEventualConsistencySuite (19.18s)
PASS protocol.TestMessengerVerificationRequests (18.64s)
PASS protocol.TestGroupChatSuite (17.25s)
PASS protocol.TestMessengerProfileShowcaseSuite (14.73s)
PASS protocol.TestMessengerSyncActivityCenter (14.44s)
PASS protocol.TestMessengerDeleteMessageForMeSuite (13.72s)
PASS protocol.TestMessengerBackupSuite (12.09s)
PASS protocol.TestMessengerEditMessageSuite (12.21s)
PASS protocol.TestMessengerRemoveMessageSuite (11.09s)
PASS protocol.TestMessengerSyncKeycardChangeSuite (11.57s)

[igor-sirotin]

The most surprising one here is TestMessengerProfilePictureHandlerSuite 👀 :

PASS protocol.TestMessengerProfilePictureHandlerSuite/TestChatIdentity (0.64s)
PASS protocol.TestMessengerProfilePictureHandlerSuite/TestE2eSendingReceivingProfilePicture (137.62s)
PASS protocol.TestMessengerProfilePictureHandlerSuite/TestEncryptDecryptIdentityImagesWithContactPubKeys (0.66s)
PASS protocol.TestMessengerProfilePictureHandlerSuite/TestPictureInPrivateChatOneSided (2.72s)

lol, good point 😂

status-go/protocol/messenger_identity_image_test.go

Line 265 in 0e37ec2

// TODO see if possible to push each test scenario into a go routine

[igor-sirotin]

The most surprising one here is TestMessengerProfilePictureHandlerSuite:

The longest parts in this test are:

• 40s:
  

  status-go/protocol/messenger_identity_image_test.go

  Line 272 in 0e37ec2

  s.SetupTest()

• 1m26s:
  

  status-go/protocol/messenger_identity_image_test.go

  Line 406 in 0e37ec2

  err = tt.RetryWithBackOff(func() error {

And I'm pretty sure it's a common pain point for all tests in protocol.

[igor-sirotin]

42 of 72 test cases in TestMessengerProfilePictureHandlerSuite rely on not receiving an image:

status-go/protocol/messenger_identity_image_test.go

Line 434 in 0e37ec2

s.Require().EqualError(err, "no new contacts with images received")

... with backoff of 2 seconds:

status-go/protocol/messenger_identity_image_test.go

Line 404 in 0e37ec2

b.MaxElapsedTime = 2 * time.Second

[osmaczko]

PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunity_ImportCommunity (64.95s)
PASS protocol.TestMessengerCommunitiesSuite/TestImportCommunity (64.15s)
PASS protocol.TestMessengerCommunitiesSuite/TestRolesAfterImportCommunity (63.91s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityMaxNumberOfMembers (12.98s)
PASS protocol.TestMessengerCommunitiesSuite/TestRetrieveBigCommunity (4.48s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunitySettings_EditCommunity (4.21s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunity_Leave (3.88s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunity_RequestToJoin (3.79s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityContactCodeAdvertisement (3.39s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunity_OutdatedDescription (3.31s)
PASS protocol.TestMessengerCommunitiesSuite/TestLeaveAndRejoinCommunity (3.27s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityLastOpenedAt (3.16s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunity_EncryptionKeys (3.06s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunity (3.04s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunityLastOpenedAt (2.94s)
PASS protocol.TestMessengerCommunitiesSuite/TestSyncCommunitySettings (2.92s)
PASS protocol.TestMessengerCommunitiesSuite/TestCancelRequestAccess (2.66s)
PASS protocol.TestMessengerCommunitiesSuite/TestStartCommunityRekeyLoop (2.63s)
PASS protocol.TestMessengerCommunitiesSuite/TestDeclineAccess (2.56s)
PASS protocol.TestMessengerCommunitiesSuite/TestShareCommunity (2.56s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityRekeyAfterBan (2.54s)
PASS protocol.TestMessengerCommunitiesSuite/TestJoinCommunity (2.48s)
PASS protocol.TestMessengerCommunitiesSuite/TestBanUserAndDeleteAllUserMessages (2.45s)
PASS protocol.TestMessengerCommunitiesSuite/TestDeletePendingRequestAccess (2.44s)
PASS protocol.TestMessengerCommunitiesSuite/TestAliceDoesNotReceiveMentionWhenSpectating (2.40s)
PASS protocol.TestMessengerCommunitiesSuite/TestDeletePendingRequestAccessWithDeclinedState (2.36s)
PASS protocol.TestMessengerCommunitiesSuite/TestRequestAccess (2.36s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityRekeyAfterBanDisableCompatibility (2.35s)
PASS protocol.TestMessengerCommunitiesSuite/TestBanUser (2.31s)
PASS protocol.TestMessengerCommunitiesSuite/TestShareCommunityWithPreviousMember (2.31s)
PASS protocol.TestMessengerCommunitiesSuite/TestIsDisplayNameDupeOfCommunityMember (2.29s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityBanUserRequestToJoin (2.27s)
PASS protocol.TestMessengerCommunitiesSuite/TestJoiningOpenCommunityReturnsChatsResponse (2.24s)
PASS protocol.TestMessengerCommunitiesSuite/TestRetrieveCommunity (2.21s)
PASS protocol.TestMessengerCommunitiesSuite/TestAliceDidNotProcessOutdatedCommunityRequestToJoinResponse (2.16s)
PASS protocol.TestMessengerCommunitiesSuite/TestOpenAndNotJoinedCommunityNewChannelIsNotEmpty (2.12s)
PASS protocol.TestMessengerCommunitiesSuite/TestPostToCommunityChat (2.01s)
PASS protocol.TestMessengerCommunitiesSuite/TestMemberMessagesHasImageLink (2.00s)
PASS protocol.TestMessengerCommunitiesSuite/TestHandleImport (1.99s)
PASS protocol.TestMessengerCommunitiesSuite/TestPinMessageInCommunityChat (1.96s)
PASS protocol.TestMessengerCommunitiesSuite/TestRequestAccessAgain (1.94s)
PASS protocol.TestMessengerCommunitiesSuite/TestCreateCommunity (1.91s)
PASS protocol.TestMessengerCommunitiesSuite/TestCheckCommunitiesToUnmute (1.89s)
PASS protocol.TestMessengerCommunitiesSuite/TestCreateCommunity_WithoutDefaultChannel (1.89s)
PASS protocol.TestMessengerCommunitiesSuite/TestSetMutePropertyOnChatsByCategory (1.83s)
PASS protocol.TestMessengerCommunitiesSuite/TestCommunityNotInDB (1.81s)
PASS protocol.TestMessengerCommunitiesSuite/TestExtractDiscordChannelsAndCategories (1.81s)
PASS protocol.TestMessengerCommunitiesSuite/TestExtractDiscordChannelsAndCategories_WithErrors (1.81s)
PASS protocol.TestMessengerCommunitiesSuite/TestMuteAllCommunityChats (1.77s)
PASS protocol.TestMessengerCommunitiesSuite/TestIgnoreOutdatedCommunityDescription (1.76s)
PASS protocol.TestMessengerCommunitiesSuite/TestSerializedCommunities (1.74s)
PASS protocol.TestMessengerCommunitiesSuite/TestRemovePrivateKey (1.71s)
PASS protocol.TestMessengerCommunitiesSuite/TestRequestAndCancelCommunityAdminOffline (1.69s)
PASS protocol.TestMessengerCommunitiesSuite/TestGetCommunityIdFromKey (1.66s)

I'll look into import community tests. This test tries to fetch community from store nodes 👀