Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Flaky Tests in TestSync_Basic #2165

Closed
Tracked by #1975
danforbes opened this issue Jan 10, 2022 · 1 comment · Fixed by #2470
Closed
Tracked by #1975

Fix Flaky Tests in TestSync_Basic #2165

danforbes opened this issue Jan 10, 2022 · 1 comment · Fixed by #2470
Assignees
@qdm12

Comments

@danforbes
Copy link
Contributor

danforbes commented Jan 10, 2022 •
edited
Loading

Logs
@danforbes danforbes mentioned this issue Jan 10, 2022
Open
13 tasks
@danforbes danforbes changed the title TestSync_Basic (logs) Fix Flaky Tests in TestSync_Basic Jan 10, 2022
@qdm12
Copy link
Contributor

qdm12 commented Apr 4, 2022

This was done indirectly (branch qdm12/tests-refactor-17 not in PR review yet, waiting for previous branches to be merged)

@qdm12 qdm12 self-assigned this Apr 4, 2022
@qdm12 qdm12 mentioned this issue Apr 11, 2022
Merged
qdm12 added a commit that referenced this issue Jun 10, 2022
@qdm12 @timwu20
chore(end-to-end tests): refactor and fix flaky tests (#2470)
6220402 
- make RPC helper functions not test aware
- `waitForNode` context based function
- Use `t.TempDir()` for nodes base paths
- Add `GenerateGenesisAuths` helper function
- Remove global variables
  - `GenesisDefault`, `GenesisDev`, `GenesisTwoAuthsSecondaryVRF0_9_10`
  - framework variable
  - configuration variables
  - `HOSTNAME` variable
  - `maxTries` in stress tests
  - `testTimeout` in stress tests
- Write node configurations to `t.TempDir()` per test
- Remove `DecodeRPC_NT` function
- Split `tests/utils` package
  - `tests/utils/rpc` package
  - `tests/utils/websocket` package
  - `tests/utils/config` package
- Rework start and stop of nodes
  - Init nodes in parallel
  - Remove unneeded logs
  - Log when node is ready
  - Prefix writer with node string for multiple nodes
  - Wait for nodes after starting all nodes
- Add test writer using `t.Logf`
- Refactor `getResponse`
  - not test aware
  - get passed target interface
  - push test assertion/skip to actual test
- Add helper `retry.UntilNoError` and `retry.UntilOK` functions
  - Use in `compareBlocksByNumber`
  - Use in `waitForNode`
  - Change `GetBlockHash` to not retry RPC
  - Remove `PostWithRetry`
  - Replace `compareFinalizedHeadsWithRetry` with `retry.UntilOK`
- Refactor and fix `TestSystemRPC`
  - Add empty skipped tests for missing cases
  - Split each subtest individually
  - Keep on retrying until main context is canceled
  - Fix `networkState` test case
  - Assert more fields
  - Fix #2161 and #807
- Refactor and fix `TestSync_SingleBlockProducer`
- Refactor and make faster `TestAuthorSubmitExtrinsic`
- Configure nodes using TOML Go struct only
- Pass `--no-telemetry` as flag since it's not in TOML config
- `dot`: remove `exportConfig` and `TestExportConfig`
- `dot`: remove `ExportTomlConfig` and its test
- Install NPM dependencies from Go code for polkdaot JS test
- Remove `websocket` package
  - Use `Decode` from rpc package instead
  - Add fields to RPC server response struct
- Do not use `reflect`
  - Add `fetchWithTimeout` helper function
  - Change test cases slices to individual subtests
  - Add `t.SkipNow()` to parent tests where all subtests are skipped
  - Make `TestChainRPC` event driven and increase testing depth
  - Add a few `TODO`s to increase testing depth
  - Run each query call in parallel
- Refactor `TestChainSubscriptionRPC`
  - Refactor websocket code
  - Assert things and increase test depth
  - Graceful shutdown of websocket
- Merge stable workflow in rpc workflow
  - Use Go to implement `integration-test-all.sh` directly in single 'stable' test
  - The test is fast and only does RPC calls, it should be part of the RPC workflow
  - Remove stable workflow from Makefile, github workflows directory and document
- Log out command, logs and configuration file content on runtime error
- Fixes directly related issues #2389, #2390, #2391
- Fixes flaky tests issues #807, #2161, #2165, #2167

Co-authored-by: Timothy Wu <timwu20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qdm12 qdm12

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

end-to-end tests: refactor and fix flaky tests ChainSafe/gossamer
2 participants
@danforbes @qdm12