Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rptest: create max connections test using ProducerSwarm #13138

Merged
merged 5 commits into from
Sep 3, 2023
Merged

rptest: create max connections test using ProducerSwarm #13138

merged 5 commits into from
Sep 3, 2023

Conversation

savex
Copy link
Contributor

@savex savex commented Aug 30, 2023

Check and implement functions that needed to run ProducerSwarm class
with RedpandaServiceCloud.

After that, create main test that will ramp-up connection load node-by-node

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.2.x
  • v23.1.x
  • v22.3.x

Release Notes

  • none

@savex savex self-assigned this Aug 30, 2023
@savex savex marked this pull request as draft August 30, 2023 23:24
@savex savex force-pushed the cloudv2-ht005-maxconn branch 2 times, most recently from 23bb7ff to fa5d7e6 Compare September 1, 2023 00:05
@savex
Copy link
Contributor Author

savex commented Sep 1, 2023

Simple workflow log:

ubuntu@ip-172-31-14-193:~/tests$  cd /home/ubuntu/tests ; /usr/bin/env /bin/python3 /home/ubuntu/.vscode-server/extensions/ms-python.python-2023.15.12301911/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 44133 -- -m ducktape --cluster=ducktape.cluster.json.JsonCluster --cluster-file=cluster.json --globals=globals.json --max-parallel=2 --repeat=1 --test-runner-timeout=86400000 rptest/scale_tests/high_throughput_test.py::HighThroughputTest2.test_max_connections 
[INFO:2023-08-31 23:54:24,824]: starting test run with session id 2023-08-31--014...
[INFO:2023-08-31 23:54:24,824]: running 1 tests...
[INFO:2023-08-31 23:54:24,825]: Triggering test 1 of 1...
[INFO:2023-08-31 23:54:25,872]: RunnerClient: Loading test {'directory': '/home/ubuntu/redpanda/tests/rptest/scale_tests', 'file_name': 'high_throughput_test.py', 'cls_name': 'HighThroughputTest2', 'method_name': 'test_max_connections', 'injected_args': None}
[INFO:2023-08-31 23:54:25,975]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Setting up...
[WARNING - 2023-08-31 23:54:25,976 - redpanda_cloud - create - lineno:435]: will not create cluster; already have cluster_id cjofem67v6b4tcp89mdg
...
rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Running...
[WARNING - 2023-08-31 23:54:34,413 - high_throughput_test - test_max_connections - lineno:304]: Starting swarm (500 connections / 3060 msg each)
[WARNING - 2023-08-31 23:54:36,563 - high_throughput_test - test_max_connections - lineno:317]: '0' connections at 0.000s
[WARNING - 2023-08-31 23:54:46,643 - high_throughput_test - test_max_connections - lineno:317]: '452' connections at 10.071s
[WARNING - 2023-08-31 23:54:57,003 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 20.151s
[WARNING - 2023-08-31 23:55:07,131 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 30.515s
[WARNING - 2023-08-31 23:55:17,249 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 40.639s
[WARNING - 2023-08-31 23:55:27,361 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 50.762s
[WARNING - 2023-08-31 23:55:37,466 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 60.873s
[WARNING - 2023-08-31 23:55:47,571 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 70.978s
[WARNING - 2023-08-31 23:55:57,666 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 81.083s
[WARNING - 2023-08-31 23:56:07,769 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 91.178s
[WARNING - 2023-08-31 23:56:17,879 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 101.275s
[WARNING - 2023-08-31 23:56:27,995 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 111.391s
[WARNING - 2023-08-31 23:56:38,104 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 121.507s
[WARNING - 2023-08-31 23:56:48,222 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 131.615s
[WARNING - 2023-08-31 23:56:58,341 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 141.734s
[WARNING - 2023-08-31 23:57:08,461 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 151.851s
[WARNING - 2023-08-31 23:57:18,577 - high_throughput_test - test_max_connections - lineno:317]: '500' connections at 161.971s
[WARNING - 2023-08-31 23:58:43,097 - high_throughput_test - test_max_connections - lineno:323]: Done swarming after 246.59887027740479s
[WARNING - 2023-08-31 23:58:43,604 - high_throughput_test - test_max_connections - lineno:333]: Expected messages 1530000, actual 1530000
[WARNING - 2023-08-31 23:58:54,598 - high_throughput_test - test_max_connections - lineno:337]: Deleting topic
[INFO:2023-08-31 23:58:54,934]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: PASS
[INFO:2023-08-31 23:58:54,935]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Tearing down...
[WARNING - 2023-08-31 23:58:56,335 - runner_client - log - lineno:278]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Test requested 6 nodes, used only 1
[WARNING:2023-08-31 23:58:56,335]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Test requested 6 nodes, used only 1
[INFO:2023-08-31 23:58:56,336]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Summary: 
[INFO:2023-08-31 23:58:56,336]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Data: None
test_id:    rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections
status:     PASS
run time:   4 minutes 30.359 seconds
---------------------------------------------------------------------------------------------------------------------------
============================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.8
session_id:       2023-08-31--014
run time:         4 minutes 32.279 seconds
tests run:        1
passed:           1
failed:           0
ignored:          0
opassed:          0
ofailed:          0
============================================================================================

@savex savex changed the title rptest: create simple ProducerSwarm workflow rptest: create max connections test using ProducerSwarm Sep 1, 2023
@savex savex marked this pull request as ready for review September 1, 2023 22:37
@savex savex force-pushed the cloudv2-ht005-maxconn branch from d81aab0 to 4fcddac Compare September 1, 2023 22:39
savex added 3 commits September 1, 2023 17:42
@savex
rptest: create simple ProducerSwarm workflow
935f478 
  Implemented functions that needed to run ProducerSwarm class
  with RedpandaServiceCloud. Also implemented active isAlive cluster
  method to properly support wait_until method.

  This simple POC tracks connection count on single node and asserts
  results. Actual duration of the test is longer that expected due to
  higher throughput set.
@savex
rptest: Update redpanda_cloud with proper user/acl creation workflow
3e382f7 
   In case of BYOC there can be users and ACLs already present in the
   cloud. Proper user creation/listing workflow added.
@savex
rptest: Update swarm workflow for multinodes
fd34cf0 
    Make test utilize the number of nodes supplied by the cluster and
    use them in the test. Check reasonable message count and expected
    connection count
@savex savex force-pushed the cloudv2-ht005-maxconn branch from 4fcddac to fd34cf0 Compare September 1, 2023 22:45
@savex savex requested a review from andrewhsu September 1, 2023 22:45
@savex
rptest: Simplify workflow and add exclusion
e67aa94 
    In order to speedup the test and simplify workflow lower sizes used
    and all nodes started sumultaneously. Also, due to the fact that
    test uses cluster.nodes in calculations, it can't be run against
    locally deployed redpanda (non-Cloudv2)
@savex savex force-pushed the cloudv2-ht005-maxconn branch from 328c8d2 to e67aa94 Compare September 2, 2023 00:17
@savex
Copy link
Contributor Author

savex commented Sep 2, 2023

Final workflow example:

ubuntu@ip-172-31-2-74:~/tests$  cd /home/ubuntu/tests ; /usr/bin/env /bin/python3 /home/ubuntu/.vscode-server/extensions/ms-python.python-2023.15.12301911/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 50605 -- -m ducktape --cluster=ducktape.cluster.json.JsonCluster --cluster-file=cluster.json --globals=globals.json --max-parallel=2 --repeat=1 --test-runner-timeout=86400000 rptest/scale_tests/high_throughput_test.py::HighThroughputTest2.test_max_connections 
[INFO:2023-09-02 00:04:15,500]: starting test run with session id 2023-09-02--002...
[INFO:2023-09-02 00:04:15,500]: running 1 tests...
[INFO:2023-09-02 00:04:15,501]: Triggering test 1 of 1...
[INFO:2023-09-02 00:04:16,512]: RunnerClient: Loading test {'directory': '/home/ubuntu/redpanda/tests/rptest/scale_tests', 'file_name': 'high_throughput_test.py', 'cls_name': 'HighThroughputTest2', 'method_name': 'test_max_connections', 'injected_args': None}
[INFO:2023-09-02 00:04:16,623]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Setting up...
[WARNING - 2023-09-02 00:04:16,623 - redpanda_cloud - create - lineno:490]: will not create cluster; already have cluster_id cjp3vvm7v6b4tcp89u5g
...
[INFO:2023-09-02 00:04:23,286]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Running...
[WARNING - 2023-09-02 00:05:37,374 - high_throughput_test - test_max_connections - lineno:335]: Start swarming from 3 nodes: 550 connections per node, 60 msg each producer
[WARNING - 2023-09-02 00:05:37,374 - high_throughput_test - test_max_connections - lineno:348]: Starting swarm node 1
[WARNING - 2023-09-02 00:05:39,554 - high_throughput_test - test_max_connections - lineno:348]: Starting swarm node 2
[WARNING - 2023-09-02 00:05:40,631 - high_throughput_test - test_max_connections - lineno:348]: Starting swarm node 3
[WARNING - 2023-09-02 00:05:41,833 - high_throughput_test - test_max_connections - lineno:361]: 164 connections (103/61/0) at 4.392s
[WARNING - 2023-09-02 00:05:52,128 - high_throughput_test - test_max_connections - lineno:361]: 1482 connections (511/502/469) at 14.469s
[WARNING - 2023-09-02 00:06:02,307 - high_throughput_test - test_max_connections - lineno:361]: 1642 connections (548/549/545) at 24.761s
[WARNING - 2023-09-02 00:06:02,308 - high_throughput_test - test_max_connections - lineno:371]: Reached target of 1500 connections (1642)
[WARNING - 2023-09-02 00:06:02,308 - high_throughput_test - test_max_connections - lineno:386]: Waiting for swarm to finish
[WARNING - 2023-09-02 00:09:03,344 - high_throughput_test - test_max_connections - lineno:390]: Done swarming after 205.96988940238953s
[WARNING - 2023-09-02 00:09:03,700 - high_throughput_test - test_max_connections - lineno:406]: Expected more than 178200 messages out of 198000, actual 198000
[WARNING - 2023-09-02 00:09:03,700 - high_throughput_test - test_max_connections - lineno:411]: Deleting topic
[WARNING - 2023-09-02 00:09:04,031 - high_throughput_test - test_max_connections - lineno:415]: Reached 1642 of 1500 needed
[INFO:2023-09-02 00:09:04,034]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: PASS
[INFO:2023-09-02 00:09:04,034]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Tearing down...
[INFO:2023-09-02 00:09:05,026]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Summary: 
[INFO:2023-09-02 00:09:05,027]: RunnerClient: rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections: Data: None
test_id:    rptest.scale_tests.high_throughput_test.HighThroughputTest2.test_max_connections
status:     PASS
run time:   4 minutes 48.403 seconds

@savex
Copy link
Contributor Author

savex commented Sep 2, 2023

/cdt
tests/rptest/tests/services_self_test.py::SimpleSelfTest

@savex
Copy link
Contributor Author

savex commented Sep 2, 2023

/cdt
tests/rptest/tests/services_self_test.py::SimpleSelfTest
rp_version=build

@savex
Copy link
Contributor Author

savex commented Sep 2, 2023

Build is green: https://buildkite.com/redpanda/redpanda/builds/36302#018a536f-308f-43a6-86c7-1eb8a6b78720

@savex
Copy link
Contributor Author

savex commented Sep 2, 2023

/cdt
rptest/scale_tests/high_throughput_test.py::HighThroughputTest2.test_max_connections
rp_version=build

@savex savex requested a review from andrewhsu September 2, 2023 16:54
@savex
Copy link
Contributor Author

savex commented Sep 2, 2023

Fresh build with test implemented is here: https://buildkite.com/redpanda/redpanda/builds/36314

andrewhsu
andrewhsu approved these changes Sep 2, 2023
View reviewed changes
Copy link
Member

@andrewhsu andrewhsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

known failures:

@savex savex merged commit 418e4cf into dev Sep 3, 2023
22 of 28 checks passed
@savex savex deleted the cloudv2-ht005-maxconn branch September 3, 2023 14:49
@vbotbuildovich
Copy link
Collaborator

/backport v23.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to run cherry-pick command. I executed the commands below:

git checkout -b backport-pr-13138-v23.2.x-867 remotes/upstream/v23.2.x
git cherry-pick -x 935f478d96e5a93d281e08ddb3f5048ef1557528 3e382f71cd8fae95468f3b32f21086647a7520b1 fd34cf090ea60a51c9d5b15c1f47b2b832c6986b e67aa94179c4ba3866fb774ccc6e2f57b859cba5 f048e4eec8c661456836ce695e56e10cedc608cf

Workflow run logs.

@vbotbuildovich vbotbuildovich mentioned this pull request Sep 3, 2023
Closed
@savex savex mentioned this pull request Sep 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewhsu andrewhsu andrewhsu approved these changes

Assignees

@savex savex

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@savex @vbotbuildovich @andrewhsu