Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: refactor loading and attaching data sources, and upgrade to llama-index==0.10.6 #1016

Merged
merged 20 commits into from
Feb 19, 2024

Conversation

sarahwooders
Copy link
Collaborator

@sarahwooders sarahwooders commented Feb 16, 2024

Please describe the purpose of this pull request.
Currently, you can only load data sources from the CLI, and only attached data sources from the CLI and via /attach. This PR makes it so that passages can be programmatically inserted into a data source, and also loaded into agents.

This PR makes changes to make it easier to build the REST servers on top of the attach/loading code:

  • Define an extensible DataConnector object for connecting to external data sources
  • Upgrade to llama_index==0.10.6 and resolve breaking changes (note: migration will no longer work, since the previously saved vector indices are no longer compatible with the most recent version of llama index)

TODOS

  • Create common base function that are called from both CLI and server:
    • Creating a data source
    • Parsing files into passages (other data sources types are a TODO)
    • Inserting passages into the data source
    • Attaching data to an agent

TODOs for next PR:

  • Support data loading/attaching from REST API
  • Implement File API for OpenAI-compatible assistants API

Related issues or PRs
#892

@sarahwooders sarahwooders marked this pull request as draft February 16, 2024 03:08
@sarahwooders sarahwooders marked this pull request as ready for review February 18, 2024 23:33
@sarahwooders sarahwooders changed the title feat: add data loading and attaching data sources to server and REST APIs + File API support (OpenAI Assistants) feat: refactor loading and attaching data sources, and upgrade to llama-index==0.10.6 Feb 19, 2024
@sarahwooders sarahwooders requested a review from cpacker February 19, 2024 00:30
Copy link
Collaborator

@cpacker cpacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest ez

@sarahwooders sarahwooders merged commit 38c184c into main Feb 19, 2024
5 checks passed
mattzh72 pushed a commit that referenced this pull request Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants