[DPE-4427] Address main instability sources on backups integration tests #496
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lucasgameiroborges
changed the title
| async with ops_test.fast_forward(fast_interval="60s"): | ||
| await ops_test.model.wait_for_idle( | ||
| apps=[database_app_name], status="active", timeout=1000 | ||
| ) |
There was a problem hiding this comment.
This was added in order to avoid the charm getting stuck due to https://warthogs.atlassian.net/browse/DPE-4596. Once underlying issue is resolved, this change should get reverted, but we shouldn't let CI suffer in the meantime IMO
| # Run the "create backup" action. | ||
| # With a stable cluster, Run the "create backup" action | ||
| async with ops_test.fast_forward(): | ||
| await ops_test.model.wait_for_idle(status="active", timeout=1000, idle_period=30) |
There was a problem hiding this comment.
This was added because, in rare cases, the charm was not idle/ready for a backup to be created.
dragomirp
reviewed
Jun 10, 2024
dragomirp
reviewed
Jun 10, 2024
|
For the record: this full CI retry was the first time I've seen our entire CI pass in one go! |
taurus-forever
approved these changes
Jun 11, 2024
marceloneppel
approved these changes
Jun 11, 2024
marceloneppel
approved these changes
Jun 11, 2024
dragomirp
approved these changes
Jun 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
The
test_backups.pyintegration test has been one of the main sources of CI failure in recent past. This PR aims to help stabilize the test.Solution
This PR addresses a number of root causes to the problem, namely:
DPE-4427
hook failed: "certificates-relation-changed": The function callpush_tls_files_to_workload()was failing with a transient connection error after multiple retries, but tenacity'sRetryErrorwas not one of the catched exceptions, thus making the hook fail instead of deferring.DPE-4425 charm stuck on scale-up: Caused by an infinite defer loop on
on_peer_relation_changedevent because cluster was not initialized. Added an extra check to initialize cluster if on leader unit, which breaks the loop.DPE-2107 Switchover on scale-down: If the primary unit is the one being removed in a scale-down, the charm doesn't recover ==> implement a switchover on
storage_detachinghook.[No ticket?] Add connection check after restart operation: implements a
_can_connect_to_postgresqlcheck that is verified on restart hook.[No ticket?]
update_read_only_endpointcall may fail due to permission denied when updating databag/secret in the case of read-only cluster in the async_replication test ==> catch and log the error and move forward with the event.Related Follow-ups
test_backups.pycharm may get stuck when TLS and S3-config events happen at the same time (?): https://warthogs.atlassian.net/browse/DPE-4596test_charm.pyinstability over theshared_bufferssetting: https://warthogs.atlassian.net/browse/DPE-4594test_tls.pyinstability due topg_rewindsometimes not using TLS: https://warthogs.atlassian.net/browse/DPE-4590If you see an immediate solution for the issues mentioned, please let me know so I can test it and include in the PR. Otherwise, we have tickets for further investigation.