Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/celery multi #2470

Merged
merged 128 commits into from
Sep 27, 2024
Merged

Feature/celery multi #2470

merged 128 commits into from
Sep 27, 2024

Conversation

rkuo-danswer
Copy link
Contributor

Description

Fixes DAN-652.

How Has This Been Tested?

[Describe the tests you ran to verify your changes]

Accepted Risk

[Any know risks or failure modes to point out to reviewers]

Related Issue(s)

[If applicable, link to the issue(s) this PR addresses]

Checklist:

  • All of the automated tests pass
  • All PR comments are addressed and marked resolved
  • If there are migrations, they have been rebased to latest main
  • If there are new dependencies, they are added to the requirements
  • If there are new environment variables, they are added to all of the deployment methods
  • If there are new APIs that don't require auth, they are added to PUBLIC_ENDPOINT_SPECS
  • Docker images build and basic functionalities work
  • Author has done a final read through of the PR right before merge

@rkuo-danswer
first cut at redis
10330fb
@rkuo-danswer
Merge branch 'main' of https://github.com/danswer-ai/danswer into fea…
7442957 
…ture/redis
@rkuo-danswer
some new helper functions for the db
09e95fe
@rkuo-danswer
Merge from feature/redis
7244bae
@rkuo-danswer
ignore kombu tables in alembic migrations (used by celery)
25f4b08
@rkuo-danswer
multiline commands for readability, add vespa_metadata_sync queue to …
d4c06b3 
…worker
@rkuo-danswer
typo fix
5100065
@rkuo-danswer
fix returning tuple fields
34a8bde
@rkuo-danswer
add constants
d856297
@rkuo-danswer
fix _get_access_for_document
93fb8ef
@rkuo-danswer
docstrings!
8d09dcf
@rkuo-danswer
fix double function declaration and typing
6d05e29
@rkuo-danswer
fix type hinting
024d4dd
@rkuo-danswer
add a global redis pool
186d4f1
@rkuo-danswer
Add get_document function
5f7ca15
@rkuo-danswer
use task_logger in various celery tasks
66fb6b1
@rkuo-danswer
add celeryconfig.py to simplify configuration. Will be used in a subs…
f06efb9 
…equent commit
@rkuo-danswer
Add celery redis helper. used in a subsequent PR
e36fd3c
@rkuo-danswer
kombu warning getting spammy since celery is not self managing its qu…
ab9cf9a 
…eue in Postgres any more
@rkuo-danswer
add last_modified and last_synced to documents
48963ed
@rkuo-danswer
fix task naming convention
b4b7801
@rkuo-danswer
use celeryconfig.py
1cfbfe1
@rkuo-danswer
the big one. adds queues and tasks, updates functions to use the queu…
2144617 
…es with priorities, etc
@rkuo-danswer
change vespa index log line to debug
eea178f
@rkuo-danswer
mypy fixes
27eae3d
@rkuo-danswer
Merge branch 'main' of https://github.com/danswer-ai/danswer into fea…
db252a6 
…ture/background_processing

# Conflicts:
#	backend/danswer/db/document_set.py
#	backend/ee/danswer/user_groups/sync.py
@rkuo-danswer
update alembic migration
c174d00
@rkuo-danswer
fix fence ordering, rename to "monitor", fix fetch_versioned_implemen…
c0a7c7c 
…tation call
@rkuo-danswer
mypy
d17f49f
@rkuo-danswer
Merge branch 'main' of https://github.com/danswer-ai/danswer into fea…
838a7fb 
…ture/background_processing
Weves
Weves reviewed Sep 26, 2024
View reviewed changes
backend/danswer/background/celery/celery_app.py Show resolved Hide resolved
backend/danswer/background/celery/celery_app.py Outdated Show resolved Hide resolved
backend/danswer/background/celery/celery_app.py Outdated Show resolved Hide resolved
backend/danswer/background/celery/celery_app.py Outdated Show resolved Hide resolved
backend/supervisord.conf Outdated Show resolved Hide resolved
backend/danswer/background/celery/celery_utils.py Show resolved Hide resolved
@rkuo-danswer
Merge branch 'main' of https://github.com/danswer-ai/danswer into fea…
d5ebb67 
…ture/celery_multi

# Conflicts:
#	backend/danswer/background/celery/celery_app.py
@rkuo-danswer
Merge branch 'main' of https://github.com/danswer-ai/danswer into fea…
82b49b6 
…ture/celery_multi

# Conflicts:
#	backend/ee/danswer/background/celery_utils.py
@rkuo-danswer rkuo-danswer added this pull request to the merge queue Sep 27, 2024
Merged via the queue into main with commit fbf51b7 Sep 27, 2024
7 checks passed
@rkuo-danswer rkuo-danswer deleted the feature/celery_multi branch October 1, 2024 18:41
This was referenced Nov 27, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Weves Weves Weves approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rkuo-danswer @Weves @LostVector