Extract eval code from GPTQ for more general usage #275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/275
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 3410422 with merge base 7511b1d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
msaroufim
approved these changes
May 25, 2024
| @@ -186,9 +186,14 @@ def test_8da4w_quantizer(self): | |||
| assert isinstance(m.linear2, Int8DynActInt4WeightLinear) | |||
| m(*example_inputs) | |||
|
|
|||
| # TODO: save model weights as artifacts and re-enable in CI | |||
There was a problem hiding this comment.
Until this is running in CI mind if we have an eval.py we can run in a new scripts/ folder?
There was a problem hiding this comment.
Ok, will do in the next PR. Currently the models still live under test/ so we'll probably have to move those out as well
torchao/quantization/_eval.py
Outdated
| @@ -0,0 +1,228 @@ | |||
| # Copyright (c) Meta Platforms, Inc. and affiliates. | |||
There was a problem hiding this comment.
Can we put this in a higher level namespace? Something like torchao._eval? or torchao.model._eval or torchao.util
| "Int4WeightOnlyGPTQQuantizer", | ||
| "Int4WeightOnlyQuantizer", | ||
| ] + add_ons | ||
|
|
||
| if lm_eval_available: |
There was a problem hiding this comment.
awesome <3 thank you for making this change
Summary: This commit extracts all the eval code from GPTQ.py.
This is the first step towards having a general eval framework
in torchao. The eventual goal is to use lm_eval to produce
reproducible benchmarks for the quantization APIs in torchao
that we can showcase on the main README. This will have the
added benefit of allowing us to add (possibly nightly)
regression test suites for important models.
Test Plan:
```
2024-05-24:14:50:32,647 INFO [task.py:395] Building contexts for wikitext on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1132.98it/s]
2024-05-24:14:50:32,648 INFO [evaluator.py:362] Running loglikelihood_rolling requests
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:51<00:00, 51.39s/it]
wikitext: {'word_perplexity,none': 7.877762491958485, 'word_perplexity_stderr,none': 'N/A', 'byte_perplexity,none': 1.488984329919892, 'byte_perplexity_stderr,none': 'N/A', 'bits_per_byte,none': 0.5743285710685551, 'bits_per_byte_stderr,none': 'N/A', 'alias': 'wikitext'}
.
----------------------------------------------------------------------
Ran 1 test in 858.105s
OK
```
python test/quantization/test_quant_api.py -k test_8da4w_gptq_quantizer
jerryzh168
pushed a commit
to jerryzh168/executorch-1
that referenced
this pull request
May 29, 2024
Summary: Fix broken import after pytorch/ao#275 Reviewed By: jerryzh168 Differential Revision: D57888168
jerryzh168
pushed a commit
to jerryzh168/executorch-1
that referenced
this pull request
May 29, 2024
Summary: Fix broken import after pytorch/ao#275 Differential Revision: D57888168
|
|
||
| import torch | ||
|
|
||
| from .utils import _lm_eval_available, _MultiInput |
There was a problem hiding this comment.
These two are in a different directory. i.e. the package name should be .quantization.utils.
There was a problem hiding this comment.
thanks for catching that, will submit a fix
andrewor14
added a commit
that referenced
this pull request
May 29, 2024
jerryzh168
pushed a commit
to jerryzh168/executorch-1
that referenced
this pull request
May 29, 2024
Summary: Fix broken import after pytorch/ao#275 Reviewed By: jerryzh168 Differential Revision: D57888168
andrewor14
added a commit
that referenced
this pull request
May 29, 2024
jerryzh168
pushed a commit
to jerryzh168/executorch-1
that referenced
this pull request
May 29, 2024
Summary: Fix broken import after pytorch/ao#275 Reviewed By: jerryzh168 Differential Revision: D57888168
facebook-github-bot
pushed a commit
to pytorch/executorch
that referenced
this pull request
May 30, 2024
Summary: Pull Request resolved: #3760 Fix broken import after pytorch/ao#275 Reviewed By: jerryzh168 Differential Revision: D57888168 fbshipit-source-id: 51a63131ae14e362991ef962df325ec24f958e2d
dbyoung18
pushed a commit
to dbyoung18/ao
that referenced
this pull request
Jul 31, 2024
Summary: This commit extracts all the eval code from GPTQ.py.
This is the first step towards having a general eval framework
in torchao. The eventual goal is to use lm_eval to produce
reproducible benchmarks for the quantization APIs in torchao
that we can showcase on the main README. This will have the
added benefit of allowing us to add (possibly nightly)
regression test suites for important models.
Test Plan:
```
2024-05-24:14:50:32,647 INFO [task.py:395] Building contexts for wikitext on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1132.98it/s]
2024-05-24:14:50:32,648 INFO [evaluator.py:362] Running loglikelihood_rolling requests
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:51<00:00, 51.39s/it]
wikitext: {'word_perplexity,none': 7.877762491958485, 'word_perplexity_stderr,none': 'N/A', 'byte_perplexity,none': 1.488984329919892, 'byte_perplexity_stderr,none': 'N/A', 'bits_per_byte,none': 0.5743285710685551, 'bits_per_byte_stderr,none': 'N/A', 'alias': 'wikitext'}
.
----------------------------------------------------------------------
Ran 1 test in 858.105s
OK
```
python test/quantization/test_quant_api.py -k test_8da4w_gptq_quantizer
dbyoung18
pushed a commit
to dbyoung18/ao
that referenced
this pull request
Jul 31, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: This commit extracts all the eval code from GPTQ.py. This is the first step towards having a general eval framework in torchao. The eventual goal is to use lm_eval to produce reproducible benchmarks for the quantization APIs in torchao that we can showcase on the main README. This will have the added benefit of allowing us to add (possibly nightly) regression test suites for important models.
Test Plan:
python test/quantization/test_quant_api.py -k test_8da4w_gptq_quantizer