↖️ Vector-ICL ↗️

Vector-ICL: In-context Learning with Continuous Vector Representations (Zhuang et al., arxiv).

Vector-ICL is a way of conducting in-context learning with data of any forms.
1. Encode data into embedding vector
2. Project the vector into LLMs' representation space
3. Build the context with the projected embeddings

And we only need to train a light-weight projector (a linear matrix works most of the time!).
(a) Pretraining with next token prediction (such as language modelling objective) enables Vector-ICL,
(b) task finetunes further improves LLMs' ability to conduct Vector-ICL.

We show that Vector-ICL works for a wide range of modalities and tasks, surpassing few-shot ICL and domain-specific models and tunings.

Example Usage

We show examples of using Vector-ICL at text-based tasks over here (Updating in Progress):

Example	Script
Pretraining Projector	example script
Fituning Projector	example script
Evaluating text classification	example script
Evaluating text generation	example script
Evaluating ICL Baselines	example script

Questions?

If you have any questions related to the code or the paper, feel free to reach out to us at y5zhuang@ucsd.edu.

Citation

If you find our paper and code useful, please cite us:

@misc{zhuang2024vectoriclincontextlearningcontinuous,
      title={Vector-ICL: In-context Learning with Continuous Vector Representations}, 
      author={Yufan Zhuang and Chandan Singh and Liyuan Liu and Jingbo Shang and Jianfeng Gao},
      year={2024},
      eprint={2410.05629},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.05629}, 
}