Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow selection of demonstrations in ICL based on similarity to current instance #1028

Open
yoavkatz opened this issue Jul 18, 2024 · 0 comments
Assignees

Comments

@yoavkatz
Copy link
Member

yoavkatz commented Jul 18, 2024

Currently demo examples are selected from the demo pool (which is typically extracted from the train set).

The selection of demos per instance is random by default, or more complex (e.g. selection of demos with diverse labels).

This is done via Sampler object (for example: sampler=DiverseLabelsSampler(choices="class", labels="label"))

We have multiple requests to allow selection of demos based on similarity to the current instance.

This means the Sampler.sample method should also receive the current instance:

@AbstractMethod
def sample(
self, instances_pool: List[Dict[str, object]], current_instance: List[Dict[str, object]]
) -> List[Dict[str, object]]:
pass

Then we can create different samplers like

EditDistanceSimilaritySampler
SentenceBertSimilaritySampler

that would return the most similar instances to the current instance.

@yoavkatz yoavkatz self-assigned this Jul 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant