Refactor File Indexing for Background Asynchronous Processing

[ArnoChenFx]

This PR refactors the file indexing process in LightRAG server to leverage background asynchronous processing, improving the responsiveness of the API and overall user experience.

Key Changes:

• Asynchronous File Indexing: Introduces asynchronous tasks for file processing, allowing the API to return immediately while indexing occurs in the background.
• Temporary File Handling: Implements a temporary file mechanism for uploaded files, ensuring proper handling and cleanup during asynchronous indexing.
• Improved Error Handling: Enhances error handling and logging throughout the file indexing process, providing better insights into potential issues.
• Response Handling: Updates API responses to provide immediate feedback to the user, indicating that file processing is in progress.
• Consolidated File Parsing Logic: Identifies and merges duplicated file parsing implementations across different API endpoints into a single, reusable function.

Benefits:

• Improved Responsiveness: The API remains responsive during file indexing, preventing delays for other requests.
• Enhanced User Experience: Users receive immediate feedback upon uploading files, providing a smoother and more intuitive experience.
• Robust Error Handling: Comprehensive error handling and logging provide better insights into potential issues, facilitating troubleshooting and maintenance.
• Reduced Code Duplication: Consolidation of file parsing logic reduces code duplication, improving code maintainability and reducing the risk of inconsistencies.

Detailed Changes:

• Replaced synchronous file indexing with asynchronous tasks using BackgroundTasks in FastAPI.
• Introduced temporary file handling for uploaded files, saving them to a temporary location before indexing.
• Implemented error handling and logging for file processing, capturing exceptions and providing detailed error messages.
• Updated API responses to provide immediate feedback to the user, indicating that file processing is in progress.
• Implemented cleanup of temporary files after successful or failed indexing.
• Extracted common file parsing code into a dedicated function (e.g., parse_file_content) and reused it across the /documents/upload, /documents/file, and /documents/batch endpoints.
• Removed duplicated parsing logic from individual endpoint handlers.

Additional Notes:

• Consider testing the changes thoroughly to ensure they function as expected and do not introduce any regressions.

[YanSte]

@ArnoChenFx by the way, thank you very much for this PR.

If you can apply, my feedback could be really awesome.

[ArnoChenFx]

@ArnoChenFx by the way, thank you very much for this PR.

If you can apply, my feedback could be really awesome.

Applied already, thanks for your suggestion.

[YanSte]

Super thanks a lot !