Update/Fix Pipeline Mixins and ORT Pipelines #2021
Conversation
…ace/optimum into auto-diffusion-pipeline
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…eproducibility and comparaison tests (7 failed, 35 passed)
IlyasMoutawwakil
force-pushed
the
update-diffusers-mixins
branch
from
b28ff5a
to
dceccca
Compare
Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
|
I made sure the model also accepts |
There was a problem hiding this comment.
Thanks @IlyasMoutawwakil for raising and working on this PR, it's brilliant! I'm so inspired to improve the diffusers support in Neuron as well!
| components = { | ||
| "vae": self.vae, | ||
| "unet": self.unet, | ||
| "text_encoder": self.text_encoder, | ||
| "text_encoder_2": self.text_encoder_2, | ||
| "safety_checker": self.safety_checker, | ||
| "image_encoder": self.image_encoder, | ||
| } |
There was a problem hiding this comment.
Maybe adding a test leveraging from_pipe() to ensure that we have enough members in the components?
There was a problem hiding this comment.
And for making this to work you might need to allow passing ORTModelXXX to __init__() instead of just bare ort session. Never mind, it's a small nit, we can put it as todo anyways!
There was a problem hiding this comment.
yes this one will need a bit more work for compatibility, I can do it in another PR.
as you said it especially requires passing models and not sessions, which might require deprecating some stuff.
|
I uploaded a model that was exported using optimum 1.22 : I also added the same checks in optimum-intel for missing attributes in model configs, that way even older versions can still work (e.g. I also removed the need for separate |
| @@ -278,14 +284,26 @@ def _from_pretrained( | |||
| else: | |||
| submodels[submodel] = load_method(model_save_path) | |||
|
|
|||
| return cls( | |||
| # same as DiffusionPipeline.from_pretraoned, if called directly, it loads the class in the config | |||
What does this PR do?
This PR allows for using the same modeling in diffusers for ORT diffusion pipelines without maintaining custom mixins.
It also fixes the issues in output reproducibility and numeric consistency vs diffusers observed in #1960.
Breaking changes:
We also conduct a benchmark to prove this change doesn't hurt perf: #2021 (comment)
We also show how reproducibility is impossible with an onnx model that performs random numbers generation at export-time: #2021 (comment)
Before submitting
Who can review?