Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Attention-Based Building Blocks #79

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sanowl
Copy link

@sanowl sanowl commented Apr 4, 2024

This pull request implements attention-based building blocks for neural networks using the tch-rs library. The implemented components include:

GeGlu: Gated Linear Unit activation function.
FeedForward: A feed-forward layer with GeGlu activation.
CrossAttention: Cross-attention layer for query-key-value attention.
BasicTransformerBlock: A basic Transformer block composed of cross-attention and feed-forward layers.
SpatialTransformer: A spatial transformer model (also known as Transformer2DModel) that applies a series of BasicTransformerBlock layers.
AttentionBlock: An attention block that performs self-attention on the input tensor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant