How long to prepare the instruction data?

[Xnhyacinth]

Hi, @Hannibal046 How long did it take to perform this retrieval on all the training data?

[Hannibal046]

Hi @Xnhyacinth ,

Thanks for your interest in our work! For retrieval, we use a multi-vector retrieval model ColBERT-v2 to build index and search over wikipedia dump. The overall duration for indexing and searching takes about ten-plus hours.

[Xnhyacinth]

Sure, I used pyserini for retrieval and found that it requires 200+ hours. Can the processed data be provided directly via git lfs or some cloud service?

[Xnhyacinth]

Done. I use the server to speed up the process