backend: upload + chat with compass

[scott-cohere]

This PR introduces using Compass as a storage solution for files. Here are some of the key changes:

• In the file service, adding a ton of logic and functions to query Compass for our needs instead of the DB.
• Splitting uploads into two different categories (conversation vs. agent).
• Files are indexed into Compass under a conversation ID or agent ID. This makes things a bit complicated when want to retrieve files as we also need the index. Made changes to accomodate this, left comments for clarity in PR.
• For files uploaded for an agent, they are indexed into temporary indices then when an agent is created they are consolidated into one index (agent ID).

AI Description

This PR introduces changes related to file handling and storage in the backend of an application. Here's a summary of the modifications:

• src/backend/chat/custom/custom.py:
  • Added conversation_id to the call_tools function.
• src/backend/config/settings.py:
  • Introduced a new field, use_compass_file_storage, in the FeatureFlags class.
• src/backend/config/tools.py:
  • Modified the ToolName class:
    • Changed the filenames field to files, now a list of tuples containing the filename and file ID.
    • Updated the ReadFileTool parameter definition to use the new file field instead of filename.
• src/backend/routers/agent.py:
  • Added imports for RequestFile, UploadFile, and Settings from fastapi.
  • Included imports for ToolName from backend.config.tools and DeleteFileResponse, UploadFileResponse from backend.schemas.file.
  • Introduced functions for batch file upload and file deletion:
    • batch_upload_file: Allows uploading multiple files and returns a list of UploadFileResponse.
    • delete_file: Deletes a file by its ID.
  • Added logic to consolidate agent files into a single index in Compass.
• src/backend/routers/chat.py:
  • Removed file_paths from the chat_stream and chat functions.
• src/backend/routers/conversation.py:
  • Added imports for Settings from backend.config.settings and Compass from backend.services.compass.
  • Modified the delete_conversation function to use the validate_conversation function and updated file deletion logic.
  • Updated the upload_file function to check if Compass file storage is enabled before validating file size.
  • Adjusted the batch_upload_file function to use the ctx parameter and updated the file creation process.
  • Removed the update_file function.
• src/backend/schemas/file.py:
  • Made conversation_id optional in the File class.
  • Introduced a new optional field, file_content, in the File class.
• src/backend/services/chat.py:
  • Removed the get_file_service import.
  • Removed file_paths from the process_chat and create_message functions.
  • Deleted the handle_file_retrieval function.
• src/backend/services/compass.py:
  • Modified the _raw_parsing function to include a new parameter, custom_context.
• src/backend/services/file.py:
  • Added imports for uuid, datetime, Settings, Context, Compass, and get_context.
  • Introduced the get_file_service and get_compass functions to initialize instances of FileService and Compass.
  • Modified the FileService class:
    • Added a property, is_compass_enabled, to check if Compass is enabled for file storage.
    • Updated the create_conversation_files and introduced the create_agent_files function to support Compass file storage.
    • Adjusted the get_files_by_agent_id function to import ToolName from backend.config.tools.
    • Modified the get_files_by_conversation_id function to support Compass file storage.
    • Introduced the delete_conversation_file_by_id and delete_agent_file_by_id functions to handle file deletion.
    • Removed the get_file_by_id, get_files_by_ids, update_file, and bulk_delete_files functions.
    • Added new functions for Compass operations: delete_file_in_compass, get_files_in_compass, consolidate_agent_files_in_compass, and insert_files_in_compass.
  • Adjusted the validate_file function to support Compass file storage.
  • Introduced the insert_files_in_db function to insert files into the database.
  • Added the attach_conversation_id_to_files function to attach a conversation ID to files.
• src/backend/tests/conftest.py:
  • Added a new fixture, mock_compass_settings, to mock Compass-related settings.
• src/backend/tests/routers/test_conversation.py:
  • Modified the test_list_files function to use the mock_compass_settings fixture and updated file handling.
  • Adjusted the test_upload_file_existing_conversation function to use mock_compass_settings and updated file retrieval logic.
  • Updated the test_batch_upload_file_existing_conversation function to use mock_compass_settings and modified file handling.
  • Modified the test_batch_upload_file_nonexistent_conversation_creates_new_conversation function to use mock_compass_settings and updated file handling.
  • Adjusted the test_delete_file function to use mock_compass_settings and modified file retrieval.
• src/backend/tools/files.py:
  • Added imports for SearchFilter, Settings, Compass, get_compass, and get_file_service.
  • Introduced the compass_file_search function to search for files in Compass.
  • Modified the ReadFileTool and SearchFileTool classes to support Compass file storage and introduced a SEARCH_LIMIT attribute.

[GangGreenTemperTatum]

such a well detailed PR! this was a pleasure and excellent written code 🙏 thank you!

[tianjing-li]

Some suggestions

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 47.05882% with 153 lines in your changes missing coverage. Please review.

Project coverage is 76.25%. Comparing base (b05ce43) to head (a124673).

Files	Patch %	Lines
src/backend/services/file.py	36.87%	89 Missing ⚠️
src/backend/tools/files.py	26.08%	34 Missing ⚠️
src/backend/routers/agent.py	40.62%	19 Missing ⚠️
src/backend/tests/unit/routers/test_chat.py	22.22%	7 Missing ⚠️
src/backend/services/compass.py	25.00%	3 Missing ⚠️
src/backend/chat/custom/custom.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #569      +/-   ##
==========================================
- Coverage   77.10%   76.25%   -0.85%     
==========================================
  Files         238      238              
  Lines       10455    10591     +136     
==========================================
+ Hits         8061     8076      +15     
- Misses       2394     2515     +121     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tianjing-li]

Looks good to me - good to merge after/if you've tested this logic