Code

Feature/celery multi#2470

Code

Merged

rkuo-danswer merged 128 commits intomainonyx-dot-app/onyx:mainfrom feature/celery_multionyx-dot-app/onyx:feature/celery_multiCopy head branch name to clipboardSep 27, 2024

+1,488-1,049

Commits

Commits on Aug 23, 2024

• first cut at redis

  rkuo-danswer

  committed
  10330fb

  Copy full SHA for 10330fb

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  7442957

  Copy full SHA for 7442957

• some new helper functions for the db

  rkuo-danswer

  committed
  09e95fe

  Copy full SHA for 09e95fe

• Merge from feature/redis

  rkuo-danswer

  committed
  7244bae

  Copy full SHA for 7244bae

Commits on Aug 29, 2024

• ignore kombu tables in alembic migrations (used by celery)

  rkuo-danswer

  committed
  25f4b08

  Copy full SHA for 25f4b08

Commits on Aug 30, 2024

• multiline commands for readability, add vespa_metadata_sync queue to worker

  rkuo-danswer

  committed
  d4c06b3

  Copy full SHA for d4c06b3

• typo fix

  rkuo-danswer

  committed
  5100065

  Copy full SHA for 5100065

• fix returning tuple fields

  rkuo-danswer

  committed
  34a8bde

  Copy full SHA for 34a8bde

• add constants

  rkuo-danswer

  committed
  d856297

  Copy full SHA for d856297

• fix _get_access_for_document

  rkuo-danswer

  committed
  93fb8ef

  Copy full SHA for 93fb8ef

• docstrings!

  rkuo-danswer

  committed
  8d09dcf

  Copy full SHA for 8d09dcf

• fix double function declaration and typing

  rkuo-danswer

  committed
  6d05e29

  Copy full SHA for 6d05e29

• fix type hinting

  rkuo-danswer

  committed
  024d4dd

  Copy full SHA for 024d4dd

• add a global redis pool

  rkuo-danswer

  committed
  186d4f1

  Copy full SHA for 186d4f1

• Add get_document function

  rkuo-danswer

  committed
  5f7ca15

  Copy full SHA for 5f7ca15

• use task_logger in various celery tasks

  rkuo-danswer

  committed
  66fb6b1

  Copy full SHA for 66fb6b1

• add celeryconfig.py to simplify configuration. Will be used in a subsequent commit

  rkuo-danswer

  committed
  f06efb9

  Copy full SHA for f06efb9

• Add celery redis helper. used in a subsequent PR

  rkuo-danswer

  committed
  e36fd3c

  Copy full SHA for e36fd3c

• kombu warning getting spammy since celery is not self managing its queue in Postgres any more

  rkuo-danswer

  committed
  ab9cf9a

  Copy full SHA for ab9cf9a

• add last_modified and last_synced to documents

  rkuo-danswer

  committed
  48963ed

  Copy full SHA for 48963ed

• fix task naming convention

  rkuo-danswer

  committed
  b4b7801

  Copy full SHA for b4b7801

• use celeryconfig.py

  rkuo-danswer

  committed
  1cfbfe1

  Copy full SHA for 1cfbfe1

• the big one. adds queues and tasks, updates functions to use the queues with priorities, etc

  rkuo-danswer

  committed
  2144617

  Copy full SHA for 2144617

• change vespa index log line to debug

  rkuo-danswer

  committed
  eea178f

  Copy full SHA for eea178f

• mypy fixes

  rkuo-danswer

  committed
  27eae3d

  Copy full SHA for 27eae3d

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for db252a6
  rkuo-danswer

  committed
  db252a6

  Copy full SHA for db252a6

• update alembic migration

  rkuo-danswer

  committed
  c174d00

  Copy full SHA for c174d00

Commits on Aug 31, 2024

• fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call

  rkuo-danswer

  committed
  c0a7c7c

  Copy full SHA for c0a7c7c

Commits on Sep 1, 2024

• mypy

  rkuo-danswer

  committed
  d17f49f

  Copy full SHA for d17f49f

Commits on Sep 2, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  838a7fb

  Copy full SHA for 838a7fb

• switch to monotonic time

  rkuo-danswer

  committed
  0652698

  Copy full SHA for 0652698

• fix startup dependencies on redis

  LostVector

  committed
  9dbd29d

  Copy full SHA for 9dbd29d

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer

  LostVector

  committed
  575db62

  Copy full SHA for 575db62

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  LostVector

  committed
  015fab9

  Copy full SHA for 015fab9

• rebase alembic migration

  LostVector

  committed
  bdcd553

  Copy full SHA for bdcd553

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  LostVector

  committed
  e731a33

  Copy full SHA for e731a33

• kombu cleanup - fail silently

  LostVector

  committed
  c388a81

  Copy full SHA for c388a81

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for a9aca68
  LostVector

  committed
  a9aca68

  Copy full SHA for a9aca68

• mypy

  LostVector

  committed
  f0bdb39

  Copy full SHA for f0bdb39

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  LostVector

  committed
  e53ff25

  Copy full SHA for e53ff25

Commits on Sep 3, 2024

• add redis_host environment override

  LostVector

  committed
  076ed18

  Copy full SHA for 076ed18

• update REDIS_HOST env var in docker-compose.dev.yml

  LostVector

  committed
  aa1eec8

  Copy full SHA for aa1eec8

• update the rest of the docker files

  LostVector

  committed
  f358acf

  Copy full SHA for f358acf

• in flight

  LostVector

  committed
  1d4b8cf

  Copy full SHA for 1d4b8cf

Commits on Sep 4, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  6724a13

  Copy full SHA for 6724a13

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  69d762e

  Copy full SHA for 69d762e

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for a56e3d7
  rkuo-danswer

  committed
  a56e3d7

  Copy full SHA for a56e3d7

• harden indexing-status endpoint against db changes happening in the background. Needs further improvement but OK for now.

  rkuo-danswer

  committed
  d6d3d24

  Copy full SHA for d6d3d24

• allow no task syncs to run because we create certain objects with no entries but initially marked as out of date

  rkuo-danswer

  committed
  6335f5b

  Copy full SHA for 6335f5b

• add back writing to vespa on indexing

  rkuo-danswer

  committed
  4c5c4a2

  Copy full SHA for 4c5c4a2

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  957bd55

  Copy full SHA for 957bd55

Commits on Sep 5, 2024

• actually working connector deletion

  rkuo-danswer

  committed
  7133450

  Copy full SHA for 7133450

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  590bd0f

  Copy full SHA for 590bd0f

• update contributing guide

  rkuo-danswer

  committed
  98ec0b7

  Copy full SHA for 98ec0b7

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  ab097ce

  Copy full SHA for ab097ce

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  f363c6b

  Copy full SHA for f363c6b

• backporting fixes from background_deletion

  rkuo-danswer

  committed
  daf46b3

  Copy full SHA for daf46b3

• renaming cache to cache_volume

  rkuo-danswer

  committed
  dafbe13

  Copy full SHA for dafbe13

• add redis password to various deployments

  rkuo-danswer

  committed
  bd07908

  Copy full SHA for bd07908

• try setting up pr testing for helm

  rkuo-danswer

  committed
  0bb2e57

  Copy full SHA for 0bb2e57

• fix indent

  rkuo-danswer

  committed
  0b293dd

  Copy full SHA for 0b293dd

• hopefully this release version actually exists

  rkuo-danswer

  committed
  82bd7f4

  Copy full SHA for 82bd7f4

• fix command line option to --chart-dirs

  rkuo-danswer

  committed
  0988947

  Copy full SHA for 0988947

• fetch-depth 0

  rkuo-danswer

  committed
  c20d921

  Copy full SHA for c20d921

• edit values.yaml

  rkuo-danswer

  committed
  d6c3bd3

  Copy full SHA for d6c3bd3

• try setting ct working directory

  rkuo-danswer

  committed
  b8d9205

  Copy full SHA for b8d9205

• bypass testing only on change for now

  rkuo-danswer

  committed
  f90cfa7

  Copy full SHA for f90cfa7

Commits on Sep 6, 2024

• move files and lint them

  rkuo-danswer

  committed
  af34d19

  Copy full SHA for af34d19

• update helm testing

  rkuo-danswer

  committed
  5d6fe92

  Copy full SHA for 5d6fe92

• some issues suggest using --config works

  rkuo-danswer

  committed
  8cf4098

  Copy full SHA for 8cf4098

• add vespa repo

  rkuo-danswer

  committed
  bc5cb2d

  Copy full SHA for bc5cb2d

• add postgresql repo

  rkuo-danswer

  committed
  ea35879

  Copy full SHA for ea35879

• increase timeout

  rkuo-danswer

  committed
  b427439

  Copy full SHA for b427439

• try amd64 runner

  rkuo-danswer

  committed
  97d3671

  Copy full SHA for 97d3671

• fix redis password reference

  rkuo-danswer

  committed
  3574c23

  Copy full SHA for 3574c23

• add comment to helm chart testing workflow

  rkuo-danswer

  committed
  f02a231

  Copy full SHA for f02a231

• rename helm testing workflow to disable it

  rkuo-danswer

  committed
  3323d4c

  Copy full SHA for 3323d4c

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  3afa07a

  Copy full SHA for 3afa07a

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  2af4926

  Copy full SHA for 2af4926

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for 7512d68
  rkuo-danswer

  committed
  7512d68

  Copy full SHA for 7512d68

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for 29860f8
  rkuo-danswer

  committed
  29860f8

  Copy full SHA for 29860f8

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  f5cbb6d

  Copy full SHA for f5cbb6d

• adding clarifying comments

  rkuo-danswer

  committed
  89619ce

  Copy full SHA for 89619ce

Commits on Sep 7, 2024

• address code review

  rkuo-danswer

  committed
  69983ab

  Copy full SHA for 69983ab

• missed a file

  rkuo-danswer

  committed
  138866a

  Copy full SHA for 138866a

• remove commented warning ... just not needed

  LostVector

  committed
  84ec5f4

  Copy full SHA for 84ec5f4

Commits on Sep 9, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  ab77692

  Copy full SHA for ab77692

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for ca6858b
  rkuo-danswer

  committed
  ca6858b

  Copy full SHA for ca6858b

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for fe9bb06
  rkuo-danswer

  committed
  fe9bb06

  Copy full SHA for fe9bb06

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  ba58e7d

  Copy full SHA for ba58e7d

• fix imports

  rkuo-danswer

  committed
  8aef77a

  Copy full SHA for 8aef77a

• refactor to use update_single

  rkuo-danswer

  committed
  30f6463

  Copy full SHA for 30f6463

• mypy fixes

  rkuo-danswer

  committed
  092e190

  Copy full SHA for 092e190

Commits on Sep 10, 2024

• add vespa test

  rkuo-danswer

  committed
  be61371

  Copy full SHA for be61371

• merge

  rkuo-danswer

  committed
  026bca4

  Copy full SHA for 026bca4

Commits on Sep 13, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for 5f85632
  rkuo-danswer

  committed
  5f85632

  Copy full SHA for 5f85632

• multiple celery workers

  rkuo-danswer

  committed
  20ba19b

  Copy full SHA for 20ba19b

• update logs as well and set prefetch multipliers appropriate to the worker intent

  rkuo-danswer

  committed
  9b99c14

  Copy full SHA for 9b99c14

Commits on Sep 14, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for b5528e3
  LostVector

  committed
  b5528e3

  Copy full SHA for b5528e3

• add db refresh to connector deletion

  LostVector

  committed
  aca5204

  Copy full SHA for aca5204

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  LostVector

  committed
  fc3af78

  Copy full SHA for fc3af78

Commits on Sep 15, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  LostVector

  committed
  a4d36b9

  Copy full SHA for a4d36b9

Commits on Sep 16, 2024

• add some preliminary locking

  rkuo-danswer

  committed
  9ac8a10

  Copy full SHA for 9ac8a10

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  rkuo-danswer

  committed
  76b0996

  Copy full SHA for 76b0996

• organize tasks into separate files

  rkuo-danswer

  committed
  90cc753

  Copy full SHA for 90cc753

Commits on Sep 17, 2024

• celery auto associates tasks created inside another task, which bloats the result metadata considerably. trail=False prevents this.

  rkuo-danswer

  committed
  faaccc7

  Copy full SHA for faaccc7

• code review fixes

  LostVector

  committed
  c032861

  Copy full SHA for c032861

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  e2f3363

  Copy full SHA for e2f3363

• move monitor_usergroup_taskset to ee, improve logging

  rkuo-danswer

  committed
  905f104

  Copy full SHA for 905f104

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  6a8e4ef

  Copy full SHA for 6a8e4ef

Commits on Sep 18, 2024

• Merge branch 'feature/background_deletion' of https://github.com/danswer-ai/danswer into feature/celery_multi

  Show description for 166b405
  rkuo-danswer

  committed
  166b405

  Copy full SHA for 166b405

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  Show description for 3a76733
  rkuo-danswer

  committed
  3a76733

  Copy full SHA for 3a76733

• add multi workers to dev_run_background_jobs.py

  rkuo-danswer

  committed
  bffdf94

  Copy full SHA for bffdf94

• update supervisord with some recommended settings for celery

  rkuo-danswer

  committed
  2f6b5f4

  Copy full SHA for 2f6b5f4

Commits on Sep 19, 2024

• name celery workers and shorten dev script prefixing

  rkuo-danswer

  committed
  bb4ee6b

  Copy full SHA for bb4ee6b

• add configurable sql alchemy engine settings on startup (needed for various intents like API server, different celery workers and tasks, etc)

  rkuo-danswer

  committed
  4f7e5cb

  Copy full SHA for 4f7e5cb

• fix comments

  rkuo-danswer

  committed
  6ebc623

  Copy full SHA for 6ebc623

Commits on Sep 20, 2024

• merge main

  rkuo-danswer

  committed
  6433d47

  Copy full SHA for 6433d47

• autoscale sqlalchemy pool size to celery concurrency (allow override later?)

  rkuo-danswer

  committed
  12219b2

  Copy full SHA for 12219b2

• supervisord needs the percent symbols escaped

  rkuo-danswer

  committed
  e0829a9

  Copy full SHA for e0829a9

• use name as primary check, some minor refactoring and type hinting too.

  rkuo-danswer

  committed
  84e3fbd

  Copy full SHA for 84e3fbd

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  rkuo-danswer

  committed
  236b1e0

  Copy full SHA for 236b1e0

Commits on Sep 24, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  rkuo-danswer

  committed
  ece7d64

  Copy full SHA for ece7d64

Commits on Sep 26, 2024

• addressing code review

  rkuo-danswer

  committed
  47f9a85

  Copy full SHA for 47f9a85

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  Show description for d5ebb67
  rkuo-danswer

  committed
  d5ebb67

  Copy full SHA for d5ebb67

• fix import

  rkuo-danswer

  committed
  e4127c1

  Copy full SHA for e4127c1

• fix prune_documents_task references

  rkuo-danswer

  committed
  a8ddd3d

  Copy full SHA for a8ddd3d

Commits on Sep 27, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  Show description for 82b49b6
  rkuo-danswer

  committed
  82b49b6

  Copy full SHA for 82b49b6