deletekeys leaves lots of TIME_WAIT sockets

[larrys]

I have a huge (millions of records) kvstore that I am trying to prune based off age. This usually works fine with a few hundred at a time, but when I run with 100000 or higher rows at a time, it will cause the a spike in the number of time wait sockets on the Splunk box, and in turn causes occasional UI issues elsewhere in Splunk since it can't open a new socket.

Looking at search.log for the search running deletekeys, it has a bunch of errors that look to be similar problems where it can create a new socket.

11-30-2021 13:02:04.779 ERROR ScriptRunner [40268 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/kvstore_tools/bin/kvstore_deletekeys.py __EXECUTE__ collection=event_history':  OSError: [Errno 99] Cannot assign requested address
11-30-2021 13:02:04.779 ERROR ScriptRunner [40268 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/kvstore_tools/bin/kvstore_deletekeys.py __EXECUTE__ collection=event_history':  During handling of the above exception, another exception occurred:
11-30-2021 13:02:04.779 ERROR ScriptRunner [40268 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/kvstore_tools/bin/kvstore_deletekeys.py __EXECUTE__ collection=event_history':  Traceback (most recent call last):

Glancing at the code, it is using urllib, but Splunk's python does come with Requests, why not use Requests and urllib3, where we can reuse existing ones already used? https://docs.python-requests.org/en/latest/user/advanced/#keep-alive

Also, too bad Splunk's KVStore REST api does not let you pass in multiple keys to delete at a time...

Granted, this is only for cases where you are deleting a bunch of items from the kvstore, but it might make sense in other cases.

[jrzmurray]

@larrys can you take a look at the issue3 branch to see if this resolves the issue?