| kvstorebackup fails with defaults on Windows due to invalid path

[jmartens]

While using the default values from defaults/kvstore_config.conf

C:\Program Files\Splunk\bin>where /r ..\etc\apps\kvstore_tools kvstore_tools.conf
C:\Program Files\Splunk\etc\apps\kvstore_tools\default\kvstore_tools.conf

C:\Program Files\Splunk\bin>more "C:\Program Files\Splunk\etc\apps\kvstore_tools\default\kvstore_tools.conf"
[settings]
log_level = INFO
default_path = $SPLUNK_HOME/etc/apps/kvstore_tools/backups
compression = 1
retention_days = 0
retention_size = 0
backup_batch_size = 50000



C:\Program Files\Splunk\bin>echo %SPLUNK_HOME%
C:\Program Files\Splunk

C:\Program Files\Splunk\bin>

a default backup w/o parameters fails with the following error:

C:\Program Files\Splunk\bin>splunk search "| kvstorebackup"
ERROR: kvstorebackup: Path does not exist: C:\C:\Program Files\Splunk\etc\apps\kvstore_tools\backups (kvstorebackup)

But it does work when manually specifying it as path parameter:

C:\Program Files\Splunk\bin>splunk search "| kvstorebackup path=$SPLUNK_HOME/etc/apps/kvstore_tools/backups"
                     _time                                  app                       collection           result  records        message                                                                     file
----------------------------------------------- ---------------------------- ----------------------------- ------- ------- --------------------- ------------------------------------------------------------------------------------------------------------------------------
2023-08-02 11:14:42.903 W. Europe Daylight Time system                       JsonWebTokensV1               skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\system#JsonWebTokensV1#20230802_111441.json.gz
2023-08-02 11:14:42.910 W. Europe Daylight Time system                       SamlIdpCerts                  skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\system#SamlIdpCerts#20230802_111442.json.gz
2023-08-02 11:14:42.920 W. Europe Daylight Time system                       SavedSearchHistory            success      18 Downloaded collection C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\system#SavedSearchHistory#20230802_111442.json.gz
2023-08-02 11:14:42.927 W. Europe Daylight Time system                       SearchHeadClusterHealthStates skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\system#SearchHeadClusterHealthStates#20230802_111442.json.gz
2023-08-02 11:14:42.935 W. Europe Daylight Time system                       SearchHeadClusterMemberInfo   skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\system#SearchHeadClusterMemberInfo#20230802_111442.json.gz
2023-08-02 11:14:42.941 W. Europe Daylight Time system                       SSLCertificates               skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\system#SSLCertificates#20230802_111442.json.gz
2023-08-02 11:14:42.948 W. Europe Daylight Time python_upgrade_readiness_app pra_dismiss_app               skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\python_upgrade_readiness_app#pra_dismiss_app#20230802_111442.json.gz
2023-08-02 11:14:42.956 W. Europe Daylight Time python_upgrade_readiness_app pra_dismiss_file              skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\python_upgrade_readiness_app#pra_dismiss_file#20230802_111442.json.gz
2023-08-02 11:14:42.963 W. Europe Daylight Time python_upgrade_readiness_app pra_get_progress              success       1 Downloaded collection C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\python_upgrade_readiness_app#pra_get_progress#20230802_111442.json.gz
2023-08-02 11:14:42.970 W. Europe Daylight Time python_upgrade_readiness_app pra_schedule_scan             skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\python_upgrade_readiness_app#pra_schedule_scan#20230802_111442.json.gz
2023-08-02 11:14:42.982 W. Europe Daylight Time splunk-dashboard-studio      splunk-dashboard-icons        success      16 Downloaded collection C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk-dashboard-studio#splunk-dashboard-icons#20230802_111442.json.gz
2023-08-02 11:14:42.989 W. Europe Daylight Time splunk-dashboard-studio      splunk-dashboard-images       skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk-dashboard-studio#splunk-dashboard-images#20230802_111442.json.gz
2023-08-02 11:14:42.995 W. Europe Daylight Time splunk_instrumentation       instrumentation               skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_instrumentation#instrumentation#20230802_111442.json.gz
2023-08-02 11:14:43.003 W. Europe Daylight Time splunk_secure_gateway        alert_recipient_devices       skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#alert_recipient_devices#20230802_111442.json.gz
2023-08-02 11:14:43.010 W. Europe Daylight Time splunk_secure_gateway        alerts_config                 skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#alerts_config#20230802_111443.json.gz
2023-08-02 11:14:43.017 W. Europe Daylight Time splunk_secure_gateway        application_types             skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#application_types#20230802_111443.json.gz
2023-08-02 11:14:43.025 W. Europe Daylight Time splunk_secure_gateway        companion_apps                skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#companion_apps#20230802_111443.json.gz
2023-08-02 11:14:43.033 W. Europe Daylight Time splunk_secure_gateway        dashboard_meta                skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#dashboard_meta#20230802_111443.json.gz
2023-08-02 11:14:43.042 W. Europe Daylight Time splunk_secure_gateway        device_public_keys            skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#device_public_keys#20230802_111443.json.gz
2023-08-02 11:14:43.051 W. Europe Daylight Time splunk_secure_gateway        device_role_mapping           skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#device_role_mapping#20230802_111443.json.gz
2023-08-02 11:14:43.061 W. Europe Daylight Time splunk_secure_gateway        features                      skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#features#20230802_111443.json.gz
2023-08-02 11:14:43.070 W. Europe Daylight Time splunk_secure_gateway        meta                          skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#meta#20230802_111443.json.gz
2023-08-02 11:14:43.079 W. Europe Daylight Time splunk_secure_gateway        mobile_alerts                 skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#mobile_alerts#20230802_111443.json.gz
2023-08-02 11:14:43.087 W. Europe Daylight Time splunk_secure_gateway        registered_devices            skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#registered_devices#20230802_111443.json.gz
2023-08-02 11:14:43.095 W. Europe Daylight Time splunk_secure_gateway        registered_users              skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#registered_users#20230802_111443.json.gz
2023-08-02 11:14:43.103 W. Europe Daylight Time splunk_secure_gateway        search_updates                skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#search_updates#20230802_111443.json.gz
2023-08-02 11:14:43.111 W. Europe Daylight Time splunk_secure_gateway        searches                      skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#searches#20230802_111443.json.gz
2023-08-02 11:14:43.119 W. Europe Daylight Time splunk_secure_gateway        snoozed_scopes                skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#snoozed_scopes#20230802_111443.json.gz
2023-08-02 11:14:43.128 W. Europe Daylight Time splunk_secure_gateway        subscription_credentials      skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#subscription_credentials#20230802_111443.json.gz
2023-08-02 11:14:43.136 W. Europe Daylight Time splunk_secure_gateway        subscriptions                 skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#subscriptions#20230802_111443.json.gz
2023-08-02 11:14:43.144 W. Europe Daylight Time splunk_secure_gateway        unconfirmed_devices           skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#unconfirmed_devices#20230802_111443.json.gz
2023-08-02 11:14:43.153 W. Europe Daylight Time splunk_secure_gateway        user_meta                     skipped       0 Collection is empty   C:\Program Files\Splunk/etc/apps/kvstore_tools/backups\splunk_secure_gateway#user_meta#20230802_111443.json.gz

C:\Program Files\Splunk\bin>

Contents of defaults/kvstore_config.

It seems some internal setting on windows fails to evaluate properly IMHO

[jmartens]

The problem occurs because of the fact that when no default is set there is some special construction done to generate the default path as can be observed at https://github.com/deductiv/kvstore_tools/blob/master/bin/kvstore_backup.py#L119-L125

On Windows this results in the following due to not evaluating the value of $SPLUNK_BASE:

>>> import os
>>> default_path = "$SPLUNK_HOME/etc/apps/kvstore_tools/backups".split("/")
>>> default_path
['$SPLUNK_HOME', 'etc', 'apps', 'kvstore_tools', 'backups']
>>> os.path.abspath(os.path.join(os.sep, *default_path))
'C:\\$SPLUNK_HOME\\etc\\apps\\kvstore_tools\\backups'

Suggested change is to wrap the default_path with os.path.expandvars before splitting it :

>>> default_path = os.path.expandvars("$SPLUNK_HOME/etc/apps/kvstore_tools/backups").split("/")
>>> default_path
['C:\\Program Files\\Splunk', 'etc', 'apps', 'kvstore_tools', 'backups']
>>> os.path.abspath(os.path.join(os.sep, *default_path))
'C:\\Program Files\\Splunk\\etc\\apps\\kvstore_tools\\backups'
>>>

[jmartens]

Perhaps it is even better not to do any massaging and just take the path verbatim from the configuration file.

I fail to see why the splitting and merging is done and I noticed the os.path.expandvars is done later on a more general code path at https://github.com/deductiv/kvstore_tools/blob/master/bin/kvstore_backup.py#L128