Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guidance API extensions added #2637

Closed
wants to merge 0 commits into from
Closed

Conversation

danikhan632
Copy link

this is an extension to microsoft/guidance to allow text-generation-webui models to be used which allow for updated models, easy setup, mutiple GPUs etc to be used with gudiance

@danikhan632
Copy link
Author

corrensponds with this pr

guidance-ai/guidance#221

@Chrusciki
Copy link

im sorry, but i don't see any code in the folder, just a new sub module.

@danikhan632
Copy link
Author

just fixed the submodule issue and it should show a new extension

@romansop
Copy link

several issues found after trying the patch:

  1. port 9000 conflicts with ports 9000-9004 used by vscode python extensions, better to use more random port e.g. 9555
  2. extension can be loaded only after LLM model is already loaded, otherwise error in GuidanceGenerator.init on line self.bos_token= shared.tokenizer.bos_token, where self.tokenizer is None
  3. print_type is not defined but used in several places, triggering runtime error

@danikhan632 danikhan632 force-pushed the main branch 2 times, most recently from 740664b to 909d8c6 Compare June 16, 2023 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants