Export to ExecuTorch: Initial Integration #2090

guangy10 · 2024-11-06T03:00:52Z

What does this PR do?

This PR is the first initiative to create an e2e path for "Export to ExecuTorch".

In this very first revision, I'm focusing outline the skeleton of the integration work:

./setup.py: Specify new dependency in order to "Export to ExecuTorch". In this case, it's adding a new pip package executorch(Beta version) and require the latest released transformfers (models from older versions may not work with ExecuTorch)

Env setup

./optimum/commands/export/executorch.py: To support running export with ExecuTorch backend via cli.

Export AOT(Ahead-of-time)

./optimum/exporters/executorch/: Main entry point to export via ExecuTorch.

optimum/exporters/executorch/
├── __main__.py
├── causal_lm.py
├── convert.py
├── recipe_registry.py
├── task_registry.py
└── xnnpack.py

It contains the coverter.py which defines the common workflow to export a transformers model from 🤗 to ExecuTorch.
The export workflow can be called with different recipes (e.g. quantize and delegate to XNNPACK, Core ML, QNN, MPS, etc.).
For each model that performs different tasks may require different patches, this is handled via registered tasks. For example, CausalLM w/ cache must be loaded with generation_config and exported via transformers.integrations.executorch.

Run with ExecuTorch model

optimum/executorchruntime/modeling_executorch.py defines the python class that wrap the Transformers AutoModelForXXX classes. We start with ExecuTorchModelForCausalLM in this file, which inherits the base OptimizedModel and overrides all abstract methods. This is where the ExecuTorch pybinding and the runtime gets integrated.

optimum/pipelines/pipelines_base.py defines a pipeline for ExecuTorch where everything is put together, deciding which model/task/recipe to load and run. It also consists of tokenizers that paired with the model for encoding/decoding.

CC: @mergennachin

Tests

Export AOT(Ahead-of-time)

Export to ExecuTorch via CLI:
optimum-cli export executorch --model "meta-llama/Llama-3.2-1B" --task "text-generation" --recipe "xnnpack" --output_dir="meta_llama3_2_1b"
It generates the ExecuTorch model to meta_llama3_2_1b/model.pte.

Run with ExecuTorch model (`model.pte`)

python test_executorch.py which essentially runs the following two lines of code:

model = ExecuTorchModelForCausalLM.from_pretrained(
    model="meta_llama3_2_1b", 
    task="text-generation", 
    recipe="xnnpack"
)
print(model.text_generation(
    tokenizer=AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B"), 
    prompt="Hey, can you tell me any fun things to do in New York?"
))

And we got:
"Hey, can you tell me any fun things to do in New York? I’m going to be there for a week and I’m not sure what to do. I’m a little bit of a history buff and I’m also a little bit of a foodie. I’m not sure if I should go to the museum or the zoo or the aquarium. I’m not sure if I should go to the theater or the opera or the ballet..."

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

Who can review?

@michaelbenayoun @echarlaix

optimum/executorchruntime/modeling_executorch.py

guangy10 · 2024-11-06T03:11:53Z

optimum/executorchruntime/modeling_executorch.py

+        return self.et_model.forward(input_ids, cache_position)[0]
+
+    @classmethod
+    def _from_pretrained(


If I understand the example here https://github.com/huggingface/optimum/tree/7e8d857d1ed6be32046324bf8f424690f116b4e9?tab=readme-ov-file#run-the-exported-model-using-onnx-runtime correctly, this method is to load a exported model from local file system, correct?

optimum/executorchruntime/modeling_executorch.py

guangy10 · 2024-11-06T03:24:40Z

optimum/executorchruntime/modeling_executorch.py

+    def generate(
+        self,
+        prompt_tokens: List[int],
+    ) -> List[int]:
+        """
+        Generate a sequence of token ids using the ExecuTorchModule.
+
+        `pipeline()` is where everything puts together. It consists of the tokenizer for encoding the inputs and decoding the model generated outputs.
+        """
+        pass


Idk it this method belongs to this class. This should be the place where we execute the model with delegated kernel libraries via pybinding. For ONNX, it seems to be the ort.InferenceSession (can't find its definition anywhere)

guangy10 · 2024-11-06T03:29:12Z

optimum/pipelines/pipelines_base.py

+def load_executorch_pipeline(
+    model,
+    targeted_task,
+    load_tokenizer,
+    tokenizer,
+    feature_extractor,
+    load_feature_extractor,
+    SUPPORTED_TASKS,
+    subfolder: str = "",
+    token: Optional[Union[bool, str]] = None,
+    revision: str = "main",
+    model_kwargs: Optional[Dict[str, Any]] = None,
+    config: AutoConfig = None,
+    **kwargs,
+):
+    raise NotImplementedError("Executorch pipeline is not implemented yet.")
+
+
 MAPPING_LOADING_FUNC = {
    "ort": load_ort_pipeline,
    "bettertransformer": load_bettertransformer,
+    "executorch": load_executorch_pipeline,
 }


Is this pipeline ever used? If yes, can you show me examples?

The example in the landing page is using the transformers pipeline directly: https://github.com/huggingface/optimum?tab=readme-ov-file#run-the-exported-model-using-onnx-runtime. I fail to understand how it is possible to run the exported model with transformers pipeline. If the exported model may have a modified signature that is different from eager, which won't work with the eager generate I think?

I'll let @echarlaix answer here.
The pipelines should use the proper auto model class for Optimum. About your question for generate, that is true, I think we fail when we do not support exactly the same features as Transformers, is that true @echarlaix ?

yes with the current integration executorch models will not be compatible with transformers pipelines, which is not a big deal as I don't think many people uses pipelines with optimum models

guangy10 · 2024-11-12T00:36:46Z

Fixed the export path. Now optimum-cli export executorch --model "meta-llama/Llama-3.2-1B" --task "text-generation" --recipe "xnnpack" --output_dir="meta_llama3_2_1b" can work as expected

HuggingFaceDocBuilderDev · 2024-11-18T14:50:21Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

michaelbenayoun

Left a few comments. I have to say it is really great work what you have done here since it is not easy to go from our main ONNX codebase and generalize!

Do you think we could also write little tests in the same spirit as what's done for ONNX, where we validate the exported model against a vanilla PyTorch model?

michaelbenayoun · 2024-11-18T14:35:38Z

optimum/commands/export/executorch.py

+        "--recipe",
+        type=str,
+        default="xnnpack",
+        help='Pre-defined recipes for export to ExecuTorch. Defaults to "xnnpack".',


nit: can we link to the list of available recipes?
Is it relevant to limit the field of possibilities by adding a set of allowed choices?

michaelbenayoun · 2024-11-18T14:36:46Z

optimum/executorchruntime/modeling_executorch.py

+    ):
+        super().__init__(model, config)
+        self.et_model = model
+        print(f"DEBUG all static methods: {self.et_model.method_names()}")


nit: forgotten print?

michaelbenayoun · 2024-11-18T14:45:52Z

optimum/executorchruntime/modeling_executorch.py

+        if use_auth_token is not None:
+            warnings.warn(
+                "The `use_auth_token` argument is deprecated and will be removed soon. Please use the `token` argument instead.",
+                FutureWarning,
+            )
+            if token is not None:
+                raise ValueError("You cannot use both `use_auth_token` and `token` arguments at the same time.")
+            token = use_auth_token


Should we support use_auth_token or just skip it since it has been deprecated?

It has not been deprecated in Transformers 4.26.0, which we officially support, but just wanted to raise the question @echarlaix

optimum/executorchruntime/modeling_executorch.py

michaelbenayoun · 2024-11-18T15:18:58Z

optimum/exporters/executorch/convert.py

+        raise RuntimeError(f"The recipe '{recipe}' isn't registered. Detailed error: {e}")
+
+    executorch_prog = recipe_func(model, task, **kwargs)
+    # print(f"Exported program: {executorch_prog.exported_program().graph}")


Suggested change

# print(f"Exported program: {executorch_prog.exported_program().graph}")

michaelbenayoun · 2024-11-18T15:19:16Z

optimum/exporters/executorch/convert.py

+    full_path = os.path.join(f"{output_dir}", "model.pte")
+    with open(full_path, "wb") as f:
+        executorch_prog.write_to_file(f)
+        print(f"Saved exported program to {full_path}")


Can you replace this print by a logger.info please?

michaelbenayoun · 2024-11-19T13:19:47Z

optimum/exporters/executorch/xnnpack.py

+    task: str,
+    **kwargs,
+):
+    print(f"DEBUG: model={model}, task={task}, kwargs={kwargs}")


Suggested change

print(f"DEBUG: model={model}, task={task}, kwargs={kwargs}")

michaelbenayoun · 2024-11-19T13:22:58Z

optimum/pipelines/pipelines_base.py

+def load_executorch_pipeline(
+    model,
+    targeted_task,
+    load_tokenizer,
+    tokenizer,
+    feature_extractor,
+    load_feature_extractor,
+    SUPPORTED_TASKS,
+    subfolder: str = "",
+    token: Optional[Union[bool, str]] = None,
+    revision: str = "main",
+    model_kwargs: Optional[Dict[str, Any]] = None,
+    config: AutoConfig = None,
+    **kwargs,
+):
+    raise NotImplementedError("Executorch pipeline is not implemented yet.")
+
+
 MAPPING_LOADING_FUNC = {
    "ort": load_ort_pipeline,
    "bettertransformer": load_bettertransformer,
+    "executorch": load_executorch_pipeline,
 }


I'll let @echarlaix answer here.
The pipelines should use the proper auto model class for Optimum. About your question for generate, that is true, I think we fail when we do not support exactly the same features as Transformers, is that true @echarlaix ?

michaelbenayoun · 2024-11-19T13:48:36Z

test_executorch.py

This file should be removed right?

echarlaix · 2024-11-20T17:17:02Z

optimum/executorchruntime/modeling_executorch.py

+        return self.et_model.forward((input_ids, cache_position))[0]
+
+    @classmethod
+    def from_pretrained(


here should be _from_pretrained which is used to load the model once exported

Suggested change

def from_pretrained(

def _from_pretrained(

optimum/optimum/modeling_base.py

Line 436 in a7a807c

from_pretrained_method = cls._from_transformers if export else cls._from_pretrained

optimum/executorchruntime/modeling_executorch.py

echarlaix · 2024-11-20T17:30:29Z

optimum/executorchruntime/modeling_executorch.py

+    def from_pretrained(
+        cls,
+        model_dir_path: Union[str, Path],
+        task: str,


I think we can remove, it will always be "text-generation"in this case

Suggested change

task: str,

echarlaix · 2024-11-20T17:32:11Z

optimum/executorchruntime/modeling_executorch.py

+        cls,
+        model_dir_path: Union[str, Path],
+        task: str,
+        recipe: str,


can be removed as well I think as will only be used for export

Suggested change

recipe: str,

echarlaix · 2024-11-20T17:38:54Z

optimum/pipelines/pipelines_base.py

+def load_executorch_pipeline(
+    model,
+    targeted_task,
+    load_tokenizer,
+    tokenizer,
+    feature_extractor,
+    load_feature_extractor,
+    SUPPORTED_TASKS,
+    subfolder: str = "",
+    token: Optional[Union[bool, str]] = None,
+    revision: str = "main",
+    model_kwargs: Optional[Dict[str, Any]] = None,
+    config: AutoConfig = None,
+    **kwargs,
+):
+    raise NotImplementedError("Executorch pipeline is not implemented yet.")
+
+
 MAPPING_LOADING_FUNC = {
    "ort": load_ort_pipeline,
    "bettertransformer": load_bettertransformer,
+    "executorch": load_executorch_pipeline,
 }


yes with the current integration executorch models will not be compatible with transformers pipelines, which is not a big deal as I don't think many people uses pipelines with optimum models

guangy10 commented Nov 6, 2024

View reviewed changes

optimum/executorchruntime/modeling_executorch.py Outdated Show resolved Hide resolved

guangy10 commented Nov 6, 2024

View reviewed changes

optimum/executorchruntime/modeling_executorch.py Show resolved Hide resolved

guangy10 commented Nov 6, 2024

View reviewed changes

optimum/executorchruntime/modeling_executorch.py Show resolved Hide resolved

guangy10 commented Nov 6, 2024

View reviewed changes

optimum/executorchruntime/modeling_executorch.py Show resolved Hide resolved

guangy10 commented Nov 6, 2024

View reviewed changes

guangy10 force-pushed the executorch_integration_skeleton branch 2 times, most recently from 4ad0b3d to 328069d Compare November 12, 2024 00:28

guangy10 force-pushed the executorch_integration_skeleton branch 3 times, most recently from f775391 to 6b38215 Compare November 14, 2024 00:14

Export to ExecuTorch: Initial Integration

757f152

guangy10 force-pushed the executorch_integration_skeleton branch from 6b38215 to 757f152 Compare November 14, 2024 00:37

guangy10 changed the title ~~Export to ExecuTorch: Code Skeleton~~ Export to ExecuTorch: Initial Integration Nov 14, 2024

echarlaix requested a review from michaelbenayoun November 18, 2024 14:06

guangy10 mentioned this pull request Nov 19, 2024

Export to ExecuTorch huggingface/transformers#32253

Open

26 tasks

michaelbenayoun reviewed Nov 19, 2024

View reviewed changes

echarlaix reviewed Nov 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Export to ExecuTorch: Initial Integration #2090

Export to ExecuTorch: Initial Integration #2090

guangy10 commented Nov 6, 2024 •

edited

Loading

guangy10 Nov 6, 2024

guangy10 Nov 6, 2024

guangy10 Nov 6, 2024

michaelbenayoun Nov 19, 2024

echarlaix Nov 20, 2024

guangy10 commented Nov 12, 2024

HuggingFaceDocBuilderDev commented Nov 18, 2024

michaelbenayoun left a comment

michaelbenayoun Nov 18, 2024

michaelbenayoun Nov 18, 2024

michaelbenayoun Nov 18, 2024

michaelbenayoun Nov 18, 2024

michaelbenayoun Nov 18, 2024

michaelbenayoun Nov 19, 2024

michaelbenayoun Nov 19, 2024

michaelbenayoun Nov 19, 2024

echarlaix Nov 20, 2024

echarlaix Nov 20, 2024

echarlaix Nov 20, 2024

echarlaix Nov 20, 2024

Export to ExecuTorch: Initial Integration #2090

Are you sure you want to change the base?

Export to ExecuTorch: Initial Integration #2090

Conversation

guangy10 commented Nov 6, 2024 • edited Loading

What does this PR do?

Env setup

Export AOT(Ahead-of-time)

Run with ExecuTorch model

Tests

Export AOT(Ahead-of-time)

Run with ExecuTorch model (model.pte)

Before submitting

Who can review?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guangy10 commented Nov 12, 2024

HuggingFaceDocBuilderDev commented Nov 18, 2024

michaelbenayoun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guangy10 commented Nov 6, 2024 •

edited

Loading

Run with ExecuTorch model (`model.pte`)