Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New layer request. #5

Open
murrellb opened this issue Mar 11, 2024 · 0 comments
Open

New layer request. #5

murrellb opened this issue Mar 11, 2024 · 0 comments

Comments

@murrellb
Copy link
Member

Please can you implement a layer that works like the "value" propagation of IPA, but without keys, queries, and attention. Instead, have an optional "gating" layer that sees both nodes and the edge info, that will multiply the entire message by a scalar (maybe with a default softplus activation to keep it between 0 and inf). It might also be useful to try a version that does this with some structuring similar to the "multi-head" setup, where there are many small layers side by side, each with their own gating, instead of one large one.

We can call it the "Invariant Point Graph Perceptron" or something. The main idea is to try and get as much of the IPA behavior as possible, but remove the strong inductive bias to attending to close neighbours, since we can control that via the graph structure anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant