[Model] Add gemma model support. #259
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marvin-Yu
force-pushed
the
gemma_model
branch
5 times, most recently
from
130cfbe to
7fcea4b
Compare
marvin-Yu
force-pushed
the
gemma_model
branch
2 times, most recently
from
4aca9cf to
8adcda6
Compare
marvin-Yu
changed the title
marvin-Yu
force-pushed
the
gemma_model
branch
2 times, most recently
from
b8ed43b to
ad79da3
Compare
marvin-Yu
force-pushed
the
gemma_model
branch
3 times, most recently
from
b175049 to
6ce1221
Compare
marvin-Yu
requested review from
pujiang2018,
changqi1 and
abenmao
and removed request for
pujiang2018
Duyi-Wang
changed the title
marvin-Yu
force-pushed
the
gemma_model
branch
2 times, most recently
from
55cbf57 to
c12408c
Compare
|
Add support in example.cpp please. |
the Gemma support for example.cpp will be updated in a new PR, not adding to the new content of this PR. |
pujiang2018
reviewed
Apr 10, 2024
src/layers/attention.h
Outdated
| @@ -218,6 +218,7 @@ class Attention { | |||
| bool useSelfAttn, bool doLnBefore, int *positionIds = nullptr) { | |||
|
|
|||
| auto hiddenSize = ctx->hiddenSize; | |||
| auto attSize = ctx->attHeadNum * ctx->attHeadSize; | |||
There was a problem hiding this comment.
if 'attSize' not used, let's remove it.
| ctx->mmHelper->compute_silu(false, M, N, K, 1.0f, A, lda, B, scaleB, zeroB, sumB, 0.0f, C, ldc); | ||
| if (ctx->actType == DecoderContext::SILU) { | ||
| ctx->mmHelper->compute_silu(false, M, N, K, 1.0f, A, lda, B, scaleB, zeroB, sumB, 0.0f, C, ldc); | ||
| } else if (ctx->actType == DecoderContext::SWIGLU) { // chatglm2/3 |
There was a problem hiding this comment.
let's use the original path.
pujiang2018
approved these changes
Apr 10, 2024
changqi1
reviewed
Apr 10, 2024
src/utils/decoder_util.h
Outdated
| // compute gelu on the left half and then add it with the right half | ||
| template <typename T1, typename T2> | ||
| static void geluSum(hpj::Matrix<T1> &src, hpj::Matrix<T2> &dst) { | ||
| __m512 c1 = _mm512_set1_ps(0.044715f); |
|
Wait for checking Silu API. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODO list:
geluACT support.geluACT support.