-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LLM]support QWenVL second part #7808
Conversation
Thanks for your contribution! |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #7808 +/- ##
===========================================
- Coverage 56.67% 56.66% -0.02%
===========================================
Files 588 588
Lines 89243 89276 +33
===========================================
+ Hits 50580 50586 +6
- Misses 38663 38690 +27 ☔ View full report in Codecov by Sentry. |
dbb9645
to
08a2d39
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
代码非常好,留了几个 comment,麻烦有时间看看。
另外,麻烦针对于这里添加一下VL 模型的单测,模型的 forward 和 动转静的过程都是需要编写的。
batch_size = 1 | ||
seq_len = 1 | ||
if bos_token_id is None: | ||
raise ValueError("`bos_token_id` should be defined when no " "`input_ids` are provided.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里没有传入 input_ids,所以没必要描述与 input_ids 之间的关联吧,你这个 error message 会给开发者一头雾水。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里是在img2txt的模型中,在第一次进入QWenInferenceModel的forward的时候,generate中不会传入input_ids,所以需要根据inputs_embeds生成一个fake的input_ids,input_ids根据bos_token_id来生成
ab5388d
to
70d2671
Compare
7bfd5da
to
ac4ea3b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR types
PR changes
Description
qwen/modeling.py
中添加了QWenForQWenVLInferenceModel
generate_text_with_image_features
逻辑to_static
函数qwen-vl/qwen-vl-7b
权重的keyqwen-vl/qwen-vl-7b
中权重的key
与paddlenlp
模型加载需要的key
存在差异,需要转换qwen-vl/qwen-vl-7b
模型,拷贝一份qwen-vl/qwen-vl-7b-inference