feat(driver,jobs): local persistent driver based on the boltdb, fix(worker_watcher): bug with failed worker while TTL-ed

[rustatian]

Reason for This PR

closes #770
closes #772

Description of Changes

THIS DRIVER IS FOR THE LOCAL DEVELOPMENT OR FOR THE LOW RPC TASKS (1-5 per second)

• Driver based on the boltdb persistent storage.
• Driver can't share the same file with the KV storage due to the boltdb limitation. File can't be opened from the different threads/processes.
• Driver can be declared in the runtime as well as other jobs drivers.
• On close, the driver closes a connection to the file. Cleaning up the old DB files is the user's responsibility (might be changed if needed).
• Handle allocate errors case for the worker_watcher. If RR can't RE-allocate workers (after initial success allocation) for the allocate_timeout time, it will show the error. If there are no workers at all, RR panics.
• prefetch, for the boltdb means, that no more than prefetch amount of jobs will be read from the DB at the same time.
• Guarantees:
  • Delay at least for the specified time with a 10ms window. After the delay expired, the job will be in the priority queue in position according to its priority.
  • RR inherits boltdb transaction guarantees for the whole job lifecycle.

Config sample:

rpc:
  listen: tcp://127.0.0.1:6001

server:
  command: "php ../../client.php echo pipes"
  relay: "pipes"
  relay_timeout: "20s"

boltdb:
    permissions: 0777

logs:
  level: info
  encoding: console
  mode: development

jobs:
  num_pollers: 10
  pipeline_size: 100000
  pool:
    num_workers: 10
    max_jobs: 0
    allocate_timeout: 60s
    destroy_timeout: 60s

  pipelines:
    test-1:
      driver: boltdb
      prefetch: 100
      file: "rr1.db"
      priority: 1

    test-2:
      driver: boltdb
      prefetch: 100
      file: "rr2.db"
      priority: 2

Arch:

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the MIT license.

PR Checklist

[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

• All commits in this PR are signed (git commit -s).
• The reason for this PR is clearly provided (issue no. or explanation).
• The description of changes is clear and encompassing.
• Any required documentation changes (code and docs) are included in this PR.
• Any user-facing changes are mentioned in CHANGELOG.md.
• All added/changed functionality is tested.

[codecov]

Codecov Report

Merging #771 (31cf040) into master (0a66fae) will decrease coverage by 0.11%.
The diff coverage is 69.29%.

@@            Coverage Diff             @@
##           master     #771      +/-   ##
==========================================
- Coverage   67.91%   67.80%   -0.12%     
==========================================
  Files         128      132       +4     
  Lines       10548    11047     +499     
==========================================
+ Hits         7164     7490     +326     
- Misses       2765     2900     +135     
- Partials      619      657      +38     

Impacted Files	Coverage Δ
pkg/worker/worker.go	91.15% <ø> (ø)
plugins/amqp/amqpjobs/config.go	14.28% <ø> (ø)
plugins/beanstalk/config.go	10.00% <ø> (ø)
plugins/beanstalk/connection.go	57.82% <ø> (ø)
plugins/beanstalk/plugin.go	93.33% <ø> (ø)
plugins/boltdb/boltkv/config.go	75.00% <ø> (ø)
plugins/boltdb/boltkv/driver.go	51.39% <0.00%> (ø)
plugins/ephemeral/item.go	81.03% <ø> (ø)
plugins/ephemeral/plugin.go	90.90% <ø> (ø)
plugins/memcached/config.go	25.00% <ø> (ø)
... and 43 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a66fae...31cf040. Read the comment docs.

[steffendietz]

Hi, could you elaborate a bit on these limitations of the boltdb driver:

THIS DRIVER IS FOR THE LOCAL DEVELOPMENT OR FOR THE LOW RPC TASKS (1-5 per second)

[rustatian]

Hey @steffendietz . Local driver based on the key-value DB called boltdb. Boltdb wasn't developed to be a queue 😃 , but, we provided this driver to make a local development easier. You don't need to set up and configure rabbitmq or another driver. You can just declare the pipeline and immediately use it (no external dependencies). However, this driver can be used for production, but it has very-very-very low performance compared to the other drivers (it can handle no more than 20-30 RPS). It can be used for some fire-and-forget tasks which are not sensitive to the performance (this is my opinion, not a limitation).
We also provided the same guarantees as for the amqp, sqs, or other drivers using boltdb transactions (and that because queue based on the boltdb is so slow).
For example, memory driver can handle more than 300k RPS (depends on the worker or course), 100k RPS can provide rabbitmq. Feel free to ping me on our discord server (https://discord.gg/TFeEmCs) if you need a more detailed answer.