Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[API] Add LLaMA decoder API. #386

Merged
merged 5 commits into from
May 13, 2024
Merged

[API] Add LLaMA decoder API. #386

merged 5 commits into from
May 13, 2024

Conversation

changqi1
Copy link
Contributor

No description provided.

@Duyi-Wang Duyi-Wang added the interface related to interface label May 11, 2024
@changqi1
Copy link
Contributor Author

@pujiang2018 Do you have some idea to export layernorm or rms_norm param in the decoder API?
void invokeLayerLLaMA(DataType dt, ActivationType at, NormType nt, ....)

pujiang2018
pujiang2018 reviewed May 13, 2024
View reviewed changes
include/dtype.h Outdated

enum NormType {
RMS = 0,
Layer,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "LN" better than "Layer"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

src/layers/decoder_layer.cpp
int accSeqLen = pastSeqLen + currentSeqLen;
float *mask = nullptr;

auto getAttnMask = [](int sizeRequired) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are deprecating attention mask, so need to change this in future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add TODO.

@pujiang2018
Copy link
Contributor

@pujiang2018 Do you have some idea to export layernorm or rms_norm param in the decoder API? void invokeLayerLLaMA(DataType dt, ActivationType at, NormType nt, ....)

I think like this is OK.

@changqi1
Copy link
Contributor Author

@pujiang2018 @Duyi-Wang Done

@changqi1 changqi1 merged commit bff98bf into intel:main May 13, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pujiang2018 pujiang2018 pujiang2018 approved these changes

Assignees
No one assigned
Labels
interface related to interface
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@changqi1 @pujiang2018 @Duyi-Wang