Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have n_batch default to 512 when BLAS is enabled #1091

Merged
merged 5 commits into from Apr 22, 2023
Merged

Have n_batch default to 512 when BLAS is enabled #1091

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4b781c2
set default n_batch to 512 when using BLAS
Apr 20, 2023
c6dfc44
spacing
Apr 20, 2023
94cb00a
alternate implementation of setting different n_batch for BLAS
Apr 21, 2023
131159f
Merge branch 'ggerganov:master' into master
Apr 22, 2023
b2e8a32
set n_batch to 512 for all cases
Apr 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ struct gpt_params {
int32_t repeat_last_n = 64; // last n tokens to penalize
int32_t n_parts = -1; // amount of model parts (-1 = determine from model dimensions)
int32_t n_ctx = 512; // context size
int32_t n_batch = 8; // batch size for prompt processing
int32_t n_batch = 512; // batch size for prompt processing (must be >=32 to use BLAS)
int32_t n_keep = 0; // number of tokens to keep from initial prompt

// sampling parameters
Expand Down