KVStore Tools #177

arcsector · 2023-03-21T23:14:11Z

Summary

This PR provides additional KVStore tools available to the user to be configured, including:

Disable
Backup
Oplog and storage engine set at install time
KVStore Storage Engine & Server Version Upgrade

- backup - upgrade - disable - include in vars and post-install steps

arcsector · 2023-03-21T23:15:00Z

Possible other features to include:

KVStore Restore from backup
KVStore Clean
KVStore Resync
Oplog change in SHC

roles/splunk/tasks/configure_kvstore.yml

roles/splunk/tasks/kvstore_disable.yml

roles/splunk/tasks/adhoc_backup_kvstore.yml

roles/splunk/tasks/kvstore_upgrade.yml

dtwersky · 2023-03-22T13:59:19Z

@arcsector This is a great PR, especially the migration part. As I mentioned in the other comments, the commands need authentication, so we either need to add them, or start the whole task with splunk login so we don't need to set no_log on all the commands. I have not tested the splunk login method, but I assume it would work.

- clean - destructive resync - get kvstore captain - get shcluster captain

arcsector · 2023-03-23T22:33:38Z

Fixed all the auth issues (sorry it slipped my mind) in ce2c80a

roles/splunk/tasks/adhoc_clean_kvstore.yml

jewnix · 2023-03-29T17:19:48Z

roles/splunk/tasks/kvstore_upgrade.yml

+    - name: Backup KVStore
+      include_tasks: adhoc_backup_kvstore.yml
+      vars:
+        - archive_name: "-archiveName preAnsibleVersionUpgradeBackup"


Should this maybe be customizable?

Eh, not in my opinion, but it's trivial to do so.

roles/splunk/tasks/kvstore_upgrade.yml

- removed unused var - check that we can backup before we do - checks are changed_when false

jewnix · 2023-03-31T17:46:07Z

roles/splunk/defaults/main.yml

@@ -69,6 +70,10 @@ splunk_shc_target_group: shc
 splunk_shc_deployer: "{{ groups['shdeployer'] | first }}" # If you manage multiple SHCs, configure the var value in group_vars
 splunk_shc_uri_list: "{% for h in groups[splunk_shc_target_group] %}https://{{ hostvars[h].ansible_fqdn }}:{{ splunkd_port }}{% if not loop.last %},{% endif %}{% endfor %}" # If you manage multiple SHCs, configure the var value in group_vars
 start_splunk_handler_fired: false # Do not change; used to prevent unnecessary splunk restarts
+splunk_enable_kvstore: true
+splunk_kvstore_storage: undefined # Can be defined here or at the group_vars level - accepted values: "wiredTiger" or "undefined", which leaves as default
+splunk_kvstore_version: undefined # Can be defined here or at the group_vars level - accepted values: 4.2 or "undefined", which leaves as default1


I don't see this variable used either

It's used here: 9f689aa#diff-0977dd8c4cb1bc6903196d2c0c0efd06d11183fbdb1dccf3620ffff234438804R28

Oh I see, the splunk_kvstore_version is unused - I'll add it to the conditionals for the bottom of the upgrade procedure

…-for-splunk into feat-kv-migration

arcsector · 2023-04-03T07:17:22Z

Guess I accidentally created a merge commit - my bad. Feel free to remove - I'm not brave enough to force push to a fork.

jewnix · 2023-04-03T17:11:47Z

After some preliminary testing, there are some issues that need to be addressed here.

I think the adhoc_destructive_resync_kvstore.yml task should be removed from the PR. Some things there do not completely make sense to me. Maybe I'm understanding it wrong, but I don't think you need to remove a member from the SHC to do that.
The kvstore_upgrade.yml does not work on version 8.2.x because the splunk_kvstore_version does not return anything, so it fails on the conditional for the block.
Leaving the splunk_kvstore_storage as undefined, also will not update the engine because of the conditionals on the block level. At least on the 8.2 that I tested.

arcsector · 2023-04-04T01:58:59Z

@jewnix My thoughts:

Destructive resync is something that has been recommended to me through Splunk Support multiple times in order to fix KV issues, so I included it here - it seems that the advantage to it over just a normal clean is that it pulls both a fresh copy of the SH Bundle as well as the KV bundle, so that's the benefit there.
Good call, I'll put a default value in there for versions <= 8.2
That's what I want, I want people to have to specify they're upgrading in order to be able to do so one cluster component at a time.

dtwersky · 2023-04-25T14:12:12Z

@arcsector

Destructive resync is something that has been recommended to me through Splunk Support multiple times in order to fix KV issues, so I included it here - it seems that the advantage to it over just a normal clean is that it pulls both a fresh copy of the SH Bundle as well as the KV bundle, so that's the benefit there.

So here is what I think. The destructive resync should be removed from this PR. 1. Because this is a snowflake issue, and destructive KVStore sync is not something that is documented. 2. Because this also destroys the SHC completely.
Even though support sometimes tells you sometimes to do it, does not mean it's something that should be done.

deleting destructive resync task

arcsector · 2023-04-28T22:55:59Z

@dtwersky @jewnix Sorry I've been inactive on this, I removed the destructive resync, and I added a default value, though it's not for splunk_kvstore_version, rather for the splunk_current_server_version_out.stdout check, where the former is powered by default vars, and the latter is performed by a CLI call

Get SHCluster and KVstore status as JSON blobs

arcsector · 2024-03-15T20:56:18Z

Updating this with oplog size increase, as well as some helpful tasks to get KVStore-status and SHCluster-status as JSON blobs for ansible consumption. I will note this isn't using the docs' oplog increase method, but rather a method that support had been passing around for ages a while ago, so if it is requested that it reflects this document, I can do that instead. Let me know!

dtwersky · 2024-07-08T18:14:26Z

Hi @arcsector ,

Sorry this was left dormant for so long after so much work has gone in to this. I have been working internally to figure out all of this for a while on a different project, that was more for ephemeral docker instances, but I discovered a lot of things related to this PR that made me look at KVStore upgrades a little differently. There are so many differences between Splunk versions, MongoDB versions and MongoDB engines regarding to upgrade paths. I'm not sure if we should assume that people are still running version 8, and because later versions already automatically migrate and update, there may only be a need to run some of these commands in specific scenarios only.

There are so many amazing things in this PR, and I don't want to close this out and start from fresh, but maybe it needs to be revisited, and think if we want to make this compatible with older versions, or major version jumps.

What are your thoughts?

arcsector · 2024-07-09T03:55:01Z

Thanks so much for the positive comments, glad you like the materials here - I'm definitely open to revisiting this as a PR of optional tasks and then making a playbook that calls all of them to do an all-in-one upgrade. Does that sound like a good plan - I could even put them in a sub-folder roles/splunk/tasks/kvstore/... if that would help to centralize this.

Do you happen to have a good map of those version transitions and what they entail as far as mongod version and engine? I'm having to go through the docs and switch back and forth between versions, as it's not clear what the approach should even be... Thanks!

arcsector added 2 commits March 17, 2023 23:01

init kv features

9f689aa

- backup - upgrade - disable - include in vars and post-install steps

README and post_install changes

2e10ee3