Code

Multiprocess RQ workers#4233

Code

Closed

rauchy wants to merge 43 commits intomastergetredash/redash:masterfrom multi-process-rq-workersgetredash/redash:multi-process-rq-workersCopy head branch name to clipboard

+53-7

Commits

Commits on Oct 29, 2019

• add rq and an rq_worker service

  

  Omer Lachish

  committed
  824977b

• add rq_scheduler and an rq_scheduler service

  

  Omer Lachish

  committed
  7a59a1a

• move beat schedule to periodic_jobs queue

  

  Omer Lachish

  committed
  4a7c7b7

• move version checks to RQ

  

  Omer Lachish

  committed
  965ad8a

• move query result cleanup to RQ

  

  Omer Lachish

  committed
  13c2891

• move custom tasks to RQ

  

  Omer Lachish

  committed
  2b070f3

• do actual schema refreshes in rq

  

  Omer Lachish

  committed
  6479c1f

• move send_email to rq

  

  Omer Lachish

  committed
  c368833

• DRY up enqueues

  

  Omer Lachish

  committed
  af94ea9

• ditch and use a partially applied decorator

  

  Omer Lachish

  committed
  d9f6d31

• 👋 beat

  

  Omer Lachish

  committed
  8257983

• rename rq_scheduler to plain scheduler, now that there's no Celery scheduler entrypoint

  

  Omer Lachish

  committed
  c906b36

• add logging context to rq jobs (while keeping execute_query context via get_task_logger for now)

  

  Omer Lachish

  committed
  6b72ff6

• move schedule to its own module

  

  Omer Lachish

  committed
  6073a41

• cancel previously scheduled periodic jobs. not sure this is a good idea.

  

  Omer Lachish

  committed
  a627080

• rename redash.scheduler to redash.schedule

  

  Omer Lachish

  committed
  1581920

• allow custom dynamic jobs to be added decleratively

  

  Omer Lachish

  committed
  67ff9e1

• pleasing the CodeClimate overlords

  

  Omer Lachish

  committed
  4116fe5

• adjust cypress docker-compose.yml to include rq changes

  

  Omer Lachish

  committed
  a5adbf3

• DRY up Cypress docker-compose

  

  Omer Lachish

  committed
  97a3dac

• an odd attempt at watching docker-compose logs when running with Cypress

  

  Omer Lachish

  committed
  9d50738

• Revert "an odd attempt at watching docker-compose logs when running with Cypress"

  Show description for 3c4d25c
  

  Omer Lachish

  committed
  3c4d25c

• show docker-compose logs at Cypress shutdown

  

  Omer Lachish

  committed
  f714dd6

• Revert "DRY up Cypress docker-compose"

  Show description for 6c40c4e
  

  Omer Lachish

  committed
  6c40c4e

• minimal version for binding is 3.2

  

  Omer Lachish

  committed
  77df227

• remove unneccesary code reloads on cypress

  

  Omer Lachish

  committed
  0540472

• SCHEMAS_REFRESH_QUEUE is no longer a required setting

  

  Omer Lachish

  committed
  ec5afb4

• split tasks/queries.py to execution.py and maintenance.py

  

  Omer Lachish

  committed
  814110f

• rename worker to celery_worker and rq_worker to worker

  

  Omer Lachish

  committed
  f7fa7dc

• delete all existing periodic jobs before scheduling them

  

  Omer Lachish

  committed
  47c5287

• remove some unrequired requires

  

  Omer Lachish

  committed
  c3d43c7

• move schedule example to redash.schedule

  

  Omer Lachish

  committed
  a16e392

• pleasing the CodeClimate overlords

  

  Omer Lachish

  committed
  2aae335

• revert to calling a function in dynamic settings to allow periodic jobs to be scheduled after app has been loaded

  

  Omer Lachish

  committed
  0be56a5

• set the timeout_ttl to double the interval to avoid job results from expiring and having periodic jobs not reschedule

  

  Omer Lachish

  committed
  5c499c8

• a naive way to launch multiple workers in one container

  

  Omer Lachish

  committed
  9f5a68e

• updated and less brittle healthcheck

  

  Omer Lachish

  committed
  1e44681

• describe custom jobs and don't actually schedule them

  

  Omer Lachish

  committed
  7479493

• launch multiple workers with Honcho

  

  Omer Lachish

  committed
  91a2eff

• fix my faulty rebase

  

  Omer Lachish

  committed
  d0e3721

• restart all workers when a worker dies

  

  Omer Lachish

  committed
  770eaa2

• Revert "restart all workers when a worker dies"

  Show description for 306e258
  

  Omer Lachish

  committed
  306e258

• optimize work horse initialization by configuration mappers on the worker process

  

  Omer Lachish

  committed
  d5c5411