Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: cloud failure injector #11234

Merged
merged 11 commits into from
Jun 16, 2023

Conversation

andrewhsu
Copy link
Member

@andrewhsu andrewhsu commented Jun 6, 2023

Fixes https://github.com/redpanda-data/devprod/issues/680

No changes to existing test behaviour that does not use redpanda cloud.

This PR adds a factory method make_failure_injector() to instantiate object from either the existing behaviour FailureInjector class or the newly-added class FailureInjectorCloud based on config settings in ducktape_globals.json (similar to factory pattern in PR #10521)

A simple test class FailureInjectorSelfTest is created in tests/rptest/tests/services_self_test.py to test the _isolate() method on cloud if cloud is enabled in the globals config file. I verified test passes with a cluster that was created before ducktape was run:

ducktape \
  --debug \
  --globals=/path/to/ducktape_globals.json \
  --cluster=ducktape.cluster.json.JsonCluster \
  --cluster-file=/path/to/ducktape_cluster.json \
  --test-runner-timeout=3600000 \
  tests/rptest/tests/services_self_test.py::FailureInjectorSelfTest

results:

[INFO:2023-06-16 12:15:40,408]: starting test run with session id 2023-06-16--002...
[INFO:2023-06-16 12:15:40,408]: running 1 tests...
[INFO:2023-06-16 12:15:40,408]: Triggering test 1 of 1...
[INFO:2023-06-16 12:15:40,413]: RunnerClient: Loading test {'directory': '/home/a/src/redpanda/tests/rptest/tests', 'file_name': 'services_self_test.py', 'cls_name': 'FailureInjectorSelfTest', 'method_name': 'test_finjector', 'injected_args': None}
[INFO:2023-06-16 12:15:40,417]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Setting up...
[WARNING - 2023-06-16 12:15:40,417 - redpanda - create - lineno:1317]: will not create cluster; already have cluster_id ci0motok30vsi89l501g
[INFO:2023-06-16 12:15:40,417]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Running...
[INFO:2023-06-16 12:16:13,228]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: PASS
[INFO:2023-06-16 12:16:13,229]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Tearing down...
[WARNING - 2023-06-16 12:16:13,229 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Test requested 3 nodes, used only 0
[WARNING:2023-06-16 12:16:13,230]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Test requested 3 nodes, used only 0
[INFO:2023-06-16 12:16:13,230]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Summary: 
[INFO:2023-06-16 12:16:13,230]: RunnerClient: rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector: Data: None
test_id:    rptest.tests.services_self_test.FailureInjectorSelfTest.test_finjector
status:     PASS
run time:   32.813 seconds
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
===========================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.8
session_id:       2023-06-16--002
run time:         32.826 seconds
tests run:        1
passed:           1
failed:           0
ignored:          0
opassed:          0
ofailed:          0
==========================================================================================================================

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.1.x
  • v22.3.x
  • v22.2.x

Release Notes

  • none

@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch 2 times, most recently from 9a6385e to 5c6c95b Compare June 8, 2023 14:09
@andrewhsu andrewhsu changed the title tests: cloud failure injector [skip-ci] tests: cloud failure injector Jun 9, 2023
@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch 2 times, most recently from a591a36 to 0570f25 Compare June 9, 2023 14:10
@andrewhsu
Copy link
Member Author

checkpoint: 0570f25 passed all ducktape-build-debug-clang-amd64 tests

@andrewhsu andrewhsu changed the title tests: cloud failure injector tests: cloud failure injector [skip-ci] Jun 12, 2023
@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch 5 times, most recently from 6b26de5 to b180615 Compare June 14, 2023 15:59
@andrewhsu andrewhsu changed the title tests: cloud failure injector [skip-ci] tests: cloud failure injector Jun 14, 2023
@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch from b180615 to d57cb1a Compare June 14, 2023 16:00
@andrewhsu andrewhsu requested a review from dlex June 14, 2023 16:01
@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch from d57cb1a to 6e968e6 Compare June 15, 2023 14:42
@andrewhsu andrewhsu marked this pull request as ready for review June 15, 2023 14:52
@andrewhsu andrewhsu requested a review from joejulian June 15, 2023 14:52
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joejulian i could use your review on just this file everything-allowed-exec-pod.yml

@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch from 6e968e6 to 8da1169 Compare June 15, 2023 14:59
@andrewhsu andrewhsu force-pushed the cloud-failure-injector branch from 8da1169 to a53d741 Compare June 15, 2023 18:03
@andrewhsu
Copy link
Member Author

andrewhsu commented Jun 16, 2023

@andrewhsu andrewhsu merged commit bcdff19 into redpanda-data:dev Jun 16, 2023
@andrewhsu andrewhsu deleted the cloud-failure-injector branch June 16, 2023 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants