Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

question about the position embedding of query and key-value #7

Open
wzn0828 opened this issue Dec 6, 2023 · 2 comments
Open

question about the position embedding of query and key-value #7

wzn0828 opened this issue Dec 6, 2023 · 2 comments

Comments

@wzn0828
Copy link

wzn0828 commented Dec 6, 2023

0790ffe046b3fb52d7dbe6a58248504

why the position embedding of Query is one step different from the ones of Key and Value ?
Here, the Query、Key、Value means the QKV of MultiheadAttention.
@chen-wl20
Copy link
Collaborator

chen-wl20 commented Dec 7, 2023 •
edited
Loading

Thank you for your interest!
The position embedding serves the purpose of distinguishing various frames. For further details, please consult our paper.
Thank you for your continued attention. Feel free to star our project and follow subsequent updates!

@wzn0828
Copy link
Author

wzn0828 commented Dec 8, 2023

Thank you for your interest! The position embedding serves the purpose of distinguishing various frames. For further details, please consult our paper. Thank you for your continued attention. Feel free to star our project and follow subsequent updates!

您好像回答了，又好像没回答。

why the position embedding of Query is one step different from the ones of Key and Value ？
Query 和 Key Value的contend embedding是一样的，但position embedding是错一位。

consult your paper, 你paper里面哪里有说这个点？截图出来看看？或者给个参考文献也好啊~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wzn0828 @chen-wl20