Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add module __repr__ methods #2191

Closed
wants to merge 1 commit into from
Closed

Conversation

1ytic
Copy link
Contributor

@1ytic 1ytic commented Sep 4, 2024

Add __repr__ method to Module and ModuleList. This is a simple copy-paste from the PyTorch implementation and it helps to debug mapping between HF and TensorRT-LLM. For example:

[09/04/2024-16:23:28] [TRT-LLM] [I] HuggingFace model: Qwen2MoeForCausalLM(
  (model): Qwen2MoeModel(
    (embed_tokens): Embedding(151936, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2MoeDecoderLayer(
        (self_attn): Qwen2MoeSdpaAttention(
          (rotary_emb): Qwen2MoeRotaryEmbedding()
          (k_proj): QuantLinear()
          (o_proj): QuantLinear()
          (q_proj): QuantLinear()
          (v_proj): QuantLinear()
        )
        (mlp): Qwen2MoeSparseMoeBlock(
          (gate): Linear(in_features=3584, out_features=64, bias=False)
          (experts): ModuleList(
            (0-63): 64 x Qwen2MoeMLP(
              (act_fn): SiLU()
              (down_proj): QuantLinear()
              (gate_proj): QuantLinear()
              (up_proj): QuantLinear()
            )
          )
          (shared_expert): Qwen2MoeMLP(
            (act_fn): SiLU()
            (down_proj): QuantLinear()
            (gate_proj): QuantLinear()
            (up_proj): QuantLinear()
          )
          (shared_expert_gate): Linear(in_features=3584, out_features=1, bias=False)
        )
        (input_layernorm): Qwen2MoeRMSNorm()
        (post_attention_layernorm): Qwen2MoeRMSNorm()
      )
    )
    (norm): Qwen2MoeRMSNorm()
  )
  (lm_head): Linear(in_features=3584, out_features=151936, bias=False)
)
[09/04/2024-16:23:28] [TRT-LLM] [I] TensorRT-LLM model: QWenForCausalLM(
  (transformer): QWenModel(
    (vocab_embedding): Embedding()
    (layers): DecoderLayerList(
      (0-27): 28 x QWenDecoderLayer(
        (input_layernorm): RmsNorm()
        (attention): Attention(
          (qkv): WeightOnlyGroupwiseQuantLinear()
          (dense): WeightOnlyGroupwiseQuantRowLinear()
        )
        (shared_expert): MLP(
          (fc): WeightOnlyGroupwiseQuantLinear()
          (proj): WeightOnlyGroupwiseQuantRowLinear()
        )
        (shared_expert_gate): RowLinear()
        (mlp): MixtureOfExperts(
          (router): RowLinear()
          (fc): MOEWeightWrapper()
          (proj): MOEWeightWrapper()
        )
        (post_layernorm): RmsNorm()
      )
    )
    (ln_f): RmsNorm()
  )
  (lm_head): Linear()
)

@lfr-0531
Copy link
Collaborator

lfr-0531 commented Sep 8, 2024

@jershi425 Could you please take a look at this PR? Thanks

@lfr-0531 lfr-0531 added the triaged Issue has been triaged by maintainers label Sep 8, 2024
@lfr-0531 lfr-0531 requested a review from jershi425 September 8, 2024 10:14
@lfr-0531
Copy link
Collaborator

lfr-0531 commented Sep 9, 2024

@1ytic Thanks for your contribution. Checked with @jershi425, looks good to him. We'll merge your changes into the internal code base.

@kaiyux kaiyux mentioned this pull request Sep 24, 2024
@hchings
Copy link
Collaborator

hchings commented Sep 27, 2024

Closing this out as it's been merged.

@hchings hchings closed this Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Merged triaged Issue has been triaged by maintainers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants