add Cogagent #445
Merged
add Cogagent #445
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
LokeZhou
reviewed
Mar 4, 2024
paddlemix/models/cogagent/README.md
Outdated
| ## 1. 模型简介 | ||
|
|
||
| 该模型是 [CogAgent](https://arxiv.org/abs/2312.08914) 的 paddle 实现。对齐的是 huggingface 上的 `THUDM/cogagent-chat-hf`, tokenizer 采用的是 huggingface 上的 `lmsys/vicuna-7b-v1.5` | ||
|
|
LokeZhou
reviewed
Mar 4, 2024
paddlemix/models/cogagent/README.md
Outdated
|
|
||
| ### 2.1 依赖安装 | ||
|
|
||
| 1） 安装PaddleNLP develop版本 |
There was a problem hiding this comment.
这里如果没有特殊的依赖包，可以引导到首页的环境安装那里，不用具体写
LokeZhou
reviewed
Mar 4, 2024
| from functools import partial | ||
| from typing import Optional, Tuple, Union | ||
|
|
||
| from paddlenlp.transformers.bit.modeling import drop_path |
LokeZhou
reviewed
Mar 4, 2024
| super().__init__() | ||
| # >>>>>> img_size = timm.layers.to_2tuple(img_size) | ||
| img_size = (img_size, img_size) | ||
| # >>>>>> patch_size = timm.layers.to_2tuple(patch_size) |
LokeZhou
reviewed
Mar 4, 2024
paddlemix/models/cogagent/visual.py
Outdated
| boi = self.boi.expand(shape=[x.shape[0], -1, -1]) | ||
| eoi = self.eoi.expand(shape=[x.shape[0], -1, -1]) | ||
| x = paddle.concat(x=(boi, x, eoi), axis=1) | ||
| return x |
There was a problem hiding this comment.
这个visual与cross_visual能否合成一个脚本
LokeZhou
reviewed
Mar 4, 2024
LokeZhou
reviewed
Mar 4, 2024
paddlemix/models/cogagent/README.md
Outdated
|
|
||
| ```bash | ||
| python paddlemix/examples/cogagent/chat_demo.py | ||
| ``` |
LokeZhou
reviewed
Mar 5, 2024
| ## 1. 模型介绍 | ||
|
|
||
| 该模型是 [CogAgent](https://arxiv.org/abs/2312.08914) 的 paddle 实现。对齐的是 huggingface 上的 `THUDM/cogagent-chat-hf`, tokenizer 采用的是 huggingface 上的 `lmsys/vicuna-7b-v1.5` | ||
|
|
LokeZhou
reviewed
Mar 5, 2024
| parser.add_argument("--from_pretrained", type=str, default="THUDM/cogagent-chat-hf", help="pretrained ckpt") | ||
| parser.add_argument("--local_tokenizer", type=str, default="lmsys/vicuna-7b-v1.5") | ||
| parser.add_argument("--local_tokenizer", type=str, default="lmsys/vicuna-7b-v1.5") | ||
| args = parser.parse_args() |
LokeZhou
reviewed
Mar 5, 2024
|
|
||
| torch_type = "float32" | ||
| print("========Use torch type as:{} with device:{}========\n\n".format(torch_type, DEVICE)) | ||
| paddle.set_device(DEVICE) |
There was a problem hiding this comment.
删除torch相关的命名和print,DEVICE可要可不要，paddle默认gpu
LokeZhou
reviewed
Mar 5, 2024
|
|
||
|
|
||
| class EVA2CLIPModel(paddle.nn.Layer): | ||
| def __init__(self, config): |
There was a problem hiding this comment.
这个eva2clip，对比一下paddlemix/model/eva02，看是否相同能复用，如果可以尽量复用
There was a problem hiding this comment.
MLP组网设计与paddlemix/model/eva02中的实现略有不同，hf上的实现，多了一个可学习的gate_proj当作门来控制mlp里面第一次linear后的hidden feature，paddle上就是直接两个linear串起来，应该不能复用
28 tasks
LokeZhou
approved these changes
Mar 11, 2024
westfish
pushed a commit
to westfish/PaddleMIX
that referenced
this pull request
Sep 25, 2024
Co-authored-by: LokeZhou <aishenghuoaiqq@163.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.