SmoothQuant mappings tutorial

[rahul-tuli]

Description:

This PR adds a tutorial that guides users on specifying the correct mappings for the SmoothQuant Modifier in llm-compressor. The tutorial explains how to smooth inputs to the q/k/v projections in self-attention and the fc1 block in feed-forward layers, based on the SmoothQuant paper.

Changes:

• New Tutorial:
  • Located at src/llmcompressor/modifiers/smoothquant/README.md the tutorial provides instructions for identifying layers, targeting leaf modules, and using regular expressions for mappings.
  • A sample mapping for LLaMA-like models is included.

Motivation:

This tutorial simplifies applying SmoothQuant, helping users correctly target layers for efficient model quantization.

Testing:

• Manually reviewed and tested on a sample LLaMA model to ensure accuracy of instructions.

Relevant Links:

• SmoothQuant Paper: https://arxiv.org/abs/2211.10438
• Code Reference: smoothquant/base.py

[dsikka]

Followed for the most part apart from one comment for clarity. Maybe throw this into a grammar editor to help with some of the wording. Otherwise, LGTM

[dsikka]

just a clarity question otherwise LGTM.

@robertgshaw2-neuralmagic would be good to get your feedback

[rahul-tuli]

just a clarity question otherwise LGTM.

@robertgshaw2-neuralmagic would be good to get your feedback

Dipika smoothquant shifts the difficulty of quantizing activations to weights by scaling up the the weights and performing the inverse transformation to activations; to accomplish this it needs to know which layer the smoothed out activations pass into i.e which layers weights need to be scaled up.

[dsikka]

just a clarity question otherwise LGTM.
@robertgshaw2-neuralmagic would be good to get your feedback

Dipika smoothquant shifts the difficulty of quantizing activations to weights by scaling up the the weights and performing the inverse transformation to activations; to accomplish this it needs to know which layer the smoothed out activations pass into i.e which layers weights need to be scaled up.

Yes I understand that. My comment was that I did not understand the wording behind layers smoothed inputs pass into - read awkwardly. Maybe something like layers that have smoothed inputs being passed into or something along those lines

[rahul-tuli]

just a clarity question otherwise LGTM.
@robertgshaw2-neuralmagic would be good to get your feedback

Dipika smoothquant shifts the difficulty of quantizing activations to weights by scaling up the the weights and performing the inverse transformation to activations; to accomplish this it needs to know which layer the smoothed out activations pass into i.e which layers weights need to be scaled up.

Yes I understand that. My comment was that I did not understand the wording behind layers smoothed inputs pass into - read awkwardly. Maybe something like layers that have smoothed inputs being passed into or something along those lines

I tried to address it in my last commit kindly let me know if that's better

[kylesayrs]

Why do inputs and outputs need to be specified in the mappings? Correct me if I'm wrong, but smoothquant is non-applicable for layers with multiple inputs, so specifying the input module should be enough to uniquely identify that activation, right?

[rahul-tuli]

Why do inputs and outputs need to be specified in the mappings? Correct me if I'm wrong, but smoothquant is non-applicable for layers with multiple inputs, so specifying the input module should be enough to uniquely identify that activation, right?

You are correct we could do something like that, but that's an algorithmic change which is out of the scope for this PR

[kylesayrs]

@rahul-tuli We can make a future change to support output inference, conditioned on the type of the mapping. Ie List[List[Union[List[str], str]] is the current mapping type, List[str] can be an inferrable mapping type.