Seeing "sqlite3.OperationalError: database is locked" Error

[nikunj-badjatya]

Hello,

Running c7n (0.9.27) for AWS in default poll mode.
Seeing below error sometimes. We are running c7n in a subprocess. Multiple such subprocesses run on a single node.

1. Are there any recommended upper limits for number of subprocesses ?
2. Is it possible to use a separate sqlite db for cache per subprocess to avoid the locking ?
3. Could disabling the cache be one option as a workaround ?

2023-07-06 08:10:02,087: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/c7n/resources/rds.py", line 135, in resources
    return super(RDS, self).resources(query=query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/c7n/query.py", line 545, in resources
    self._cache.save(cache_key, resources)
  File "/usr/local/lib/python3.11/site-packages/c7n/cache.py", line 160, in save
    cursor.execute(
sqlite3.OperationalError: database is locked
2023-07-06 08:10:02,091: custodian.commands:ERROR Error while executing policy rds-resize-down, continuing
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/c7n/commands.py", line 307, in run
    policy()
  File "/usr/local/lib/python3.11/site-packages/c7n/policy.py", line 1352, in __call__
    resources = PullMode(self).run()
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/c7n/resources/rds.py", line 135, in resources
    return super(RDS, self).resources(query=query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/c7n/query.py", line 545, in resources
    self._cache.save(cache_key, resources)
  File "/usr/local/lib/python3.11/site-packages/c7n/cache.py", line 160, in save
    cursor.execute(
sqlite3.OperationalError: database is locked
2023-07-06 08:10:02,092: custodian.commands:ERROR The following policies had errors while executing
 - rds-resize-down

[ajkerrigan]

Thanks, good question. This error seems expected if you try to share a cache across processes, as you suspected 👍 .

1. Are there any recommended upper limits for number of subprocesses ?

Depends a bit on where you're running. c7n-org is the most common tool for fanning out custodian executions, and it defaults to cpu count * 4. Which seems as reasonable a starting point as any.

2. Is it possible to use a separate sqlite db for cache per subprocess to avoid the locking?

Yes. You can use the -f / --cache argument to the custodian run command. Again using c7n-org as a reference, it fans out into separate workers for account/region pairs. And it uses ~/.cache/c7n-org/<account>-<region>.cache as a template cache file name.

3. Could disabling the cache be one option as a workaround?

Sure that's an option too, though that may cause you to hit provider APIs harder than necessary. custodian run --cache-period 0 effectively disables the cache.

[nikunj-badjatya]

Thank you for quick response !
Any idea if Redis or equivalent support is coming for distributed cache amongst multiple different c7n processes ?