Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LLM]support QWenVL second part #7808

Merged
merged 5 commits into from
Jan 23, 2024

Conversation

DanGuge
Copy link
Contributor

@DanGuge DanGuge commented Jan 9, 2024

PR types

PR changes

Description

  • qwen/modeling.py中添加了QWenForQWenVLInferenceModel
    • 添加了qwen-vl对应的generate_text_with_image_features逻辑
    • 重写了to_static函数
  • 用户可以通过以下步骤导出模型静态图
    1. 转换qwen-vl/qwen-vl-7b权重的key
      • qwen-vl/qwen-vl-7b中权重的keypaddlenlp模型加载需要的key存在差异,需要转换
      • 下载qwen-vl/qwen-vl-7b模型,拷贝一份qwen-vl/qwen-vl-7b-inference
    import paddle
    
    state_dict = paddle.load("/path/to/.paddlenlp/models/qwen-vl/qwen-vl-7b-inference/model_state.pdparams")
    new_state_dict = {}
    for key in state_dict.keys():
        if key.startswith("transformer"):
            new_key = key.replace("transformer", "qwen")
            new_state_dict[new_key] = state_dict[key]
        else:
            new_state_dict[key] = state_dict[key]
    paddle.save(new_state_dict, "/path/to/.paddlenlp/models/qwen-vl/qwen-vl-7b-inference/model_state.pdparams")
    1. 导出静态图
    #!/bin/bash
    
    export CUDA_VISIBLE_DEVICES=0
    export PYTHONPATH=/path/to/PaddleNLP/
    
    python export_model.py \
        --model_name_or_path "/path/to/.paddlenlp/models/qwen-vl/qwen-vl-7b-inference/" \
        --output_path ./checkpoints/encode_text/ \
        --dtype float16 \
        --inference_model \
        --model_prefix qwen \
        --model_type qwen-img2txt

Copy link

paddle-bot bot commented Jan 9, 2024

Thanks for your contribution!

Copy link

codecov bot commented Jan 9, 2024

Codecov Report

Attention: 37 lines in your changes are missing coverage. Please review.

Comparison is base (c1ccafa) 56.67% compared to head (734de07) 56.66%.
Report is 2 commits behind head on develop.

Files Patch % Lines
...ddlenlp/experimental/transformers/qwen/modeling.py 0.00% 37 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #7808      +/-   ##
===========================================
- Coverage    56.67%   56.66%   -0.02%     
===========================================
  Files          588      588              
  Lines        89243    89276      +33     
===========================================
+ Hits         50580    50586       +6     
- Misses       38663    38690      +27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DanGuge DanGuge force-pushed the qwenvl/inference_model branch from dbb9645 to 08a2d39 Compare January 16, 2024 05:54
Copy link
Contributor

@wj-Mcat wj-Mcat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

代码非常好,留了几个 comment,麻烦有时间看看。

另外,麻烦针对于这里添加一下VL 模型的单测,模型的 forward 和 动转静的过程都是需要编写的。

paddlenlp/experimental/transformers/qwen/modeling.py Outdated Show resolved Hide resolved
batch_size = 1
seq_len = 1
if bos_token_id is None:
raise ValueError("`bos_token_id` should be defined when no " "`input_ids` are provided.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里没有传入 input_ids,所以没必要描述与 input_ids 之间的关联吧,你这个 error message 会给开发者一头雾水。

Copy link
Contributor Author

@DanGuge DanGuge Jan 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里是在img2txt的模型中,在第一次进入QWenInferenceModel的forward的时候,generate中不会传入input_ids,所以需要根据inputs_embeds生成一个fake的input_ids,input_ids根据bos_token_id来生成

paddlenlp/experimental/transformers/qwen/modeling.py Outdated Show resolved Hide resolved
paddlenlp/experimental/transformers/qwen/modeling.py Outdated Show resolved Hide resolved
paddlenlp/experimental/transformers/qwen/modeling.py Outdated Show resolved Hide resolved
paddlenlp/experimental/transformers/qwen/modeling.py Outdated Show resolved Hide resolved
@DanGuge DanGuge force-pushed the qwenvl/inference_model branch from ab5388d to 70d2671 Compare January 18, 2024 15:09
@DanGuge DanGuge force-pushed the qwenvl/inference_model branch from 7bfd5da to ac4ea3b Compare January 19, 2024 11:58
@DanGuge DanGuge requested a review from wj-Mcat January 22, 2024 05:33
Copy link
Collaborator

@wawltor wawltor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wawltor wawltor merged commit 2a8d138 into PaddlePaddle:develop Jan 23, 2024
7 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants