Test more architectures in ORTModel #675
Conversation
fxmarty
requested review from
regisss,
michaelbenayoun,
echarlaix,
JingyaHuang and
mht-sharma
|
The documentation is not available anymore as the PR was closed or merged. |
tests/onnxruntime/test_modeling.py
Outdated
| "bloom", | ||
| "codegen", | ||
| "bigbird_pegasus", | ||
| "bart", |
There was a problem hiding this comment.
Why you put bart, mbart and marian here? They are seq2seq models.
There was a problem hiding this comment.
This was just copy paste, basically because of
As I understand reading modeling_mbart.py, there is a MBartForCausalLM that does not use the encoder, and a MBartForConditionalGeneration that does. I guess that's because of it?
There was a problem hiding this comment.
@JingyaHuang So I removed these models from the CausalLM test. There is an asymmetry because transformers AutoModelForCausalLM (that may have an encoder) and ORTModelForCausalLM (that may not), we may want to fix this later but I'm not sure it's worth it.
|
One remaining issue is that models are exported in many different subtests, so we can expect this test to be super slow now. I'll fix as well in this PR. Edit: done |
fxmarty
changed the title
| "xlm_roberta", | ||
| ] | ||
|
|
||
| ARCH_MODEL_MAP = { |
There was a problem hiding this comment.
What's the use of ARCH_MODEL_MAP?
There was a problem hiding this comment.
Some architectures like perceiver can't have the same model used for different tasks (e.g. sequence classification and image classification). It is similar to
optimum/tests/exporters/exporters_utils.py
Lines 74 to 76 in 6e11941
Currently this is not used because perceiver does not work with ORTModel but it can be useful in the future.
| SUPPORTED_ARCHITECTURES = [ | ||
| "beit", | ||
| "convnext", | ||
| "data2vec_vision", | ||
| "deit", | ||
| "levit", | ||
| "mobilenet_v1", | ||
| "mobilenet_v2", | ||
| "mobilevit", | ||
| # "perceiver", | ||
| "poolformer", | ||
| "resnet", | ||
| "segformer", | ||
| "swin", | ||
| "vit", | ||
| ] |
There was a problem hiding this comment.
Is it not possible to infer that list automatically?
There was a problem hiding this comment.
@michaelbenayoun I would propose in a next PR to introduce in optimum/onnxruntime a dict of officially supported (aka tested) architectures for ORTModel (and raise a warning that unexpected behavior may happen if an other architecture is used). We would then use this dict in here. Does this sound good to you? I can as well add it to this PR directly if you prefer.
We can't use the ones from exporters/tasks.py because some of the models supported in the ONNX export are not in ORTModel.
Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com>
Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com>
A lot of architectures were untested up to now.
Fixes #664