Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to chose which optypes to merge #93

Open
DrChainsaw opened this issue Jul 14, 2024 · 0 comments
Open

Add option to chose which optypes to merge #93

DrChainsaw opened this issue Jul 14, 2024 · 0 comments

Comments

@DrChainsaw
Copy link
Owner

ONNXNaiveNASflux tries to merge certain op-types into a single vertex. The most prominent example of this is how activation functions are separate nodes in ONNX, but Flux allows them as part of many layer types and fuses the broadcast of the activation and bias addition.

It would be useful to be able to enable/disable this behaviour from the loading API.

One that probably should not be enabled by default is the fusing of Squeeze and Global pooling layers since Flux native global pooling layers do not squeeze the singleton dimensions (current implementation is from before Flux shipped a global pool). The merging is quite useful when doing NAS stuff since one probably don't want to randomly insert operations between the global pool and the squeeze, but I guess most users are not using this library for NAS stuff (and if they do, they would be able to easily re-enable it when loading).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant