Skip to content

Commit

Permalink
embedding : adjust n_ubatch value (#6296)
Browse files Browse the repository at this point in the history
* embedding: assign `n_ubatch` value, print error on `n_batch` overflow

* Update examples/embedding/embedding.cpp

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* use %ld instead of %lld

* Revert "use %ld instead of %lld"

This reverts commit ea753ed.

---------

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
  • Loading branch information
mscheong01 and ngxson authored Mar 26, 2024
1 parent 3d032ec commit deb7240
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion examples/embedding/embedding.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ int main(int argc, char ** argv) {
}

params.embedding = true;
// For non-causal models, batch size must be equal to ubatch size
params.n_ubatch = params.n_batch;

print_build_info();

Expand Down Expand Up @@ -114,7 +116,9 @@ int main(int argc, char ** argv) {
for (const auto & prompt : prompts) {
auto inp = ::llama_tokenize(ctx, prompt, true, false);
if (inp.size() > n_batch) {
inp.resize(n_batch);
fprintf(stderr, "%s: error: number of tokens in input line (%lld) exceeds batch size (%lld), increase batch size and re-run\n",
__func__, (long long int) inp.size(), (long long int) n_batch);
return 1;
}
inputs.push_back(inp);
}
Expand Down

0 comments on commit deb7240

Please sign in to comment.