[PTQ][OV] FP8 implementation #2283

nikita-malininn · 2023-11-22T11:32:02Z

Changes

Added FP8 implementation
Added Mode parameter

Reason for changes

New FP8 implementation

Related tickets

119805

Tests

tests/openvino/native/quantization/test_graphs.py
tests/openvino/native/test_model_transformer.py

On top of openvinotoolkit/openvino#21034 - Merged

codecov · 2023-11-22T11:49:33Z

Codecov Report

Merging #2283 (d6dedc2) into develop (0c389c3) will decrease coverage by 0.15%.
The diff coverage is 67.04%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2283      +/-   ##
===========================================
- Coverage    90.74%   90.60%   -0.15%     
===========================================
  Files          489      489              
  Lines        43975    44196     +221     
===========================================
+ Hits         39906    40044     +138     
- Misses        4069     4152      +83

Files	Coverage Δ
nncf/__init__.py	`97.14% <100.00%> (+0.08%)`	⬆️
nncf/common/hardware/config.py	`94.40% <100.00%> (ø)`
nncf/common/quantization/initialization/range.py	`95.52% <100.00%> (ø)`
...common/quantization/quantizer_propagation/graph.py	`89.36% <100.00%> (ø)`
...ommon/quantization/quantizer_propagation/solver.py	`93.64% <100.00%> (ø)`
nncf/common/quantization/quantizer_setup.py	`91.42% <100.00%> (ø)`
nncf/common/quantization/structs.py	`96.12% <100.00%> (ø)`
nncf/openvino/graph/metatypes/groups.py	`100.00% <100.00%> (ø)`
...ncf/openvino/graph/metatypes/openvino_metatypes.py	`99.43% <100.00%> (+<0.01%)`	⬆️
nncf/openvino/quantization/backend_parameters.py	`100.00% <100.00%> (ø)`
... and 28 more

... and 2 files with indirect coverage changes

Flag	Coverage Δ
COMMON	`45.07% <53.90%> (+0.07%)`	⬆️
ONNX	`33.88% <31.06%> (-0.05%)`	⬇️
OPENVINO	`38.79% <46.59%> (-0.04%)`	⬇️
TENSORFLOW	`29.84% <24.62%> (-0.07%)`	⬇️
TORCH	`62.61% <29.92%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	`93.35% <100.00%> (ø)`
torch	`93.22% <84.61%> (-0.02%)`	⬇️
tensorflow	`94.01% <87.50%> (-0.02%)`	⬇️
onnx	`96.91% <84.61%> (-0.09%)`	⬇️
openvino	`90.84% <48.57%> (-1.19%)`	⬇️
ptq	`88.37% <76.66%> (-0.34%)`	⬇️

nncf/quantization/advanced_parameters.py

…plementation

l-bat · 2023-11-24T11:24:49Z

@KodiaqQ Are you planning to add a new mode parameter to the documentation?

nikita-malininn · 2023-11-24T13:38:12Z

@KodiaqQ Are you planning to add a new mode parameter to the documentation?

Good catch, thanks. I'll add it.
But first, we need to discuss this parameter first.

alexsu52

Some general comments:

FBC and BC support FakeConvert operations?
What test plan do you propose for FP8 quantization?

tests/openvino/tools/calibrate.py

nncf/quantization/fake_quantize.py

This reverts commit 65c25a6.

andreyanufr · 2023-11-28T09:38:06Z

nncf/quantization/fake_quantize.py

+    :return: Parameters of the FakeConvert layer.
+    """
+
+    destination_type_maximum = {"HF8": 448, "BF8": 57344}


I suggest to align names with https://github.com/openvinotoolkit/openvino/blob/74f6b6454d2b503b7a412b80281bd54bfd328c43/src/core/reference/include/openvino/reference/fake_convert.hpp#L253C29-L253C37: "f8e5m2", "f8e4m3"

alexsu52

It looks like this PR requires additional validation. I would suggest to run ptq conformance test and DLB calibrate test on small model scope.

nncf/quantization/algorithms/min_max/algorithm.py

alexsu52 · 2023-12-12T12:07:24Z

The code ov_quantized_model = nncf.quantize(ov_model, calibration_dataset, mode=nncf.QuantizationMode.FP8_E5M2) produces the following warnings:

What do you think about printing only warning about the fact that FP8 is experimental option and raise runtime error if user specified parameters is not compatible with FP8 mode?

nikita-malininn · 2023-12-12T12:12:24Z

The code ov_quantized_model = nncf.quantize(ov_model, calibration_dataset, mode=nncf.QuantizationMode.FP8_E5M2) produces the following warnings:

What do you think about printing only warning about the fact that FP8 is experimental option and raise runtime error if user specified parameters is not compatible with FP8 mode?

Ok, I'll remove these warnings and add RuntimeError instead of the redefining provided options.
upd., Removed warnings & added errors.
upd 2., As a result, we must define almost all MinMax options properly to run optimization with any mode.

nikita-malininn · 2023-12-12T18:04:22Z

@alexsu52, please, review.

alexsu52

LGTM, @KodiaqQ please merge it after passing additional validation.

alexsu52 · 2023-12-13T05:49:40Z

nncf/quantization/algorithms/min_max/backend.py

@@ -154,6 +169,7 @@ def get_start_nodes_for_activation_path_tracing(nncf_graph: NNCFGraph) -> List[N

        :param nncf_graph: NNCFGraph to get the start nodes.
        :return: List of NNCFNodes to use as start nodes for activation path tracing.
+


nikita-malininn · 2023-12-14T15:53:25Z

LGTM, @KodiaqQ please merge it after passing additional validation.

Conformance validation run - 229 (manual).
Observed errors were fixed with the latest commit (for per-channel activations aka depth-wise layers).
Is it enough, @alexsu52?

github-actions bot added NNCF PT Pull requests that updates NNCF PyTorch experimental NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF ONNX Pull requests that updates NNCF ONNX NNCF PTQ Pull requests that updates NNCF PTQ labels Nov 22, 2023

openvino-nncf-ci added the API Public API-impacting changes label Nov 22, 2023

nikita-malininn force-pushed the nm/fp8_implementation branch from c048a85 to 097aaee Compare November 22, 2023 19:42

nikita-malininn added 9 commits November 22, 2023 20:48

Initial commit

8522249

Add activation

d316ac8

Update data types

1a2a224

Unify Fake nodes creation

434705a

Remove extra

b243a92

Added description

f0154a4

Fix tests

097aaee

Add tests for model transformer

9db1e70

Calibrate fix

f473c57

nikita-malininn marked this pull request as ready for review November 23, 2023 14:45

nikita-malininn requested a review from a team as a code owner November 23, 2023 14:45

Added graph test

12cd199

nikita-malininn requested review from alexsu52, l-bat, andreyanufr and AlexKoff88 November 23, 2023 15:02

nikita-malininn commented Nov 23, 2023

View reviewed changes

nncf/quantization/advanced_parameters.py Outdated Show resolved Hide resolved

nikita-malininn added 2 commits November 23, 2023 21:40

Fix test

123bee3

Merge remote-tracking branch 'openvinotoolkit/develop' into nm/fp8_im…

50e807d

…plementation

alexsu52 reviewed Nov 29, 2023

View reviewed changes

tests/openvino/tools/calibrate.py Outdated Show resolved Hide resolved

nncf/quantization/fake_quantize.py Outdated Show resolved Hide resolved

nikita-malininn added 2 commits December 6, 2023 20:55

Fix tests

88e9b89

Revert "Global change mode to scheme"

4301113

This reverts commit 65c25a6.

github-actions bot removed the documentation Improvements or additions to documentation label Dec 7, 2023

nikita-malininn added 7 commits December 7, 2023 10:16

Remove estimator params redefining

6c0325b

Merge branch 'develop' into nm/fp8_implementation

0cac099

Fix tests after merge

0d02ccb

Rollback QuantizationScheme renaming, change QuantizationMode import

d46f604

Rollback not needed changes

288281a

Fix tests after rollback

540b84f

Fix tests again

2511f1a

nikita-malininn requested a review from alexsu52 December 7, 2023 20:20

Update QuantizationScheme name, import

e8d5944

andreyanufr reviewed Dec 11, 2023

View reviewed changes

andreyanufr approved these changes Dec 11, 2023

View reviewed changes

alexsu52 reviewed Dec 12, 2023

View reviewed changes

nncf/quantization/algorithms/min_max/algorithm.py Outdated Show resolved Hide resolved

Apply comments

98574fe

nikita-malininn requested a review from alexsu52 December 12, 2023 15:50

nikita-malininn added 3 commits December 12, 2023 17:28

Merge branch 'develop' into nm/fp8_implementation

08e7735

Fix

82c1df8

Fix tests

096d501

alexsu52 reviewed Dec 13, 2023

View reviewed changes

AlexKoff88 approved these changes Dec 13, 2023

View reviewed changes

Fix for per-channel activations

7512da2

alexsu52 approved these changes Dec 15, 2023

View reviewed changes

Merge openvinotoolkit/develop into nm/fp8_implementation

d6dedc2

nikita-malininn merged commit 5f2c20e into openvinotoolkit:develop Dec 15, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PTQ][OV] FP8 implementation #2283

[PTQ][OV] FP8 implementation #2283

nikita-malininn commented Nov 22, 2023 •

edited

Loading

codecov bot commented Nov 22, 2023 •

edited

Loading

l-bat commented Nov 24, 2023

nikita-malininn commented Nov 24, 2023

alexsu52 left a comment

andreyanufr Nov 28, 2023

alexsu52 left a comment

alexsu52 commented Dec 12, 2023

nikita-malininn commented Dec 12, 2023 •

edited

Loading

nikita-malininn commented Dec 12, 2023

alexsu52 left a comment

alexsu52 Dec 13, 2023

nikita-malininn commented Dec 14, 2023

		@@ -154,6 +169,7 @@ def get_start_nodes_for_activation_path_tracing(nncf_graph: NNCFGraph) -> List[N

		:param nncf_graph: NNCFGraph to get the start nodes.
		:return: List of NNCFNodes to use as start nodes for activation path tracing.

[PTQ][OV] FP8 implementation #2283

[PTQ][OV] FP8 implementation #2283

Conversation

nikita-malininn commented Nov 22, 2023 • edited Loading

Changes

Reason for changes

Related tickets

Tests

codecov bot commented Nov 22, 2023 • edited Loading

Codecov Report

l-bat commented Nov 24, 2023

nikita-malininn commented Nov 24, 2023

alexsu52 left a comment

Choose a reason for hiding this comment

andreyanufr Nov 28, 2023

Choose a reason for hiding this comment

alexsu52 left a comment

Choose a reason for hiding this comment

alexsu52 commented Dec 12, 2023

nikita-malininn commented Dec 12, 2023 • edited Loading

nikita-malininn commented Dec 12, 2023

alexsu52 left a comment

Choose a reason for hiding this comment

alexsu52 Dec 13, 2023

Choose a reason for hiding this comment

nikita-malininn commented Dec 14, 2023

nikita-malininn commented Nov 22, 2023 •

edited

Loading

codecov bot commented Nov 22, 2023 •

edited

Loading

nikita-malininn commented Dec 12, 2023 •

edited

Loading