Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add NDTF and mLSTM/sLSTM #149

Merged
merged 6 commits into from
Nov 5, 2024
Merged

add NDTF and mLSTM/sLSTM #149

merged 6 commits into from
Nov 5, 2024

Conversation

AnFreTh
Copy link
Collaborator

@AnFreTh AnFreTh commented Nov 5, 2024

This pull request introduces significant changes to the mambular/arch_utils module, including the addition of new classes, refactoring of existing code, and renaming of files. The most important changes include the introduction of the BlockDiagonal and NeuralEmbeddingTree classes, updates to the EmbeddingLayer class, and substantial modifications to the mLSTMblock and sLSTMblock classes.

New Classes

  • Added BlockDiagonal class in mambular/arch_utils/layer_utils/block_diagonal.py for creating block-diagonal linear layers.
  • Added NeuralEmbeddingTree class in mambular/arch_utils/layer_utils/embedding_tree.py for implementing neural decision trees with hard decision boundaries.

Updates to Existing Classes

  • Updated EmbeddingLayer class to support a new embedding layer type "ndt" and integrated NeuralEmbeddingTree for numerical embeddings. [1] [2] [3]

Refactoring and File Renaming

  • Renamed mambular/arch_utils/embedding_layer.py to mambular/arch_utils/layer_utils/embedding_layer.py and updated imports accordingly.
  • Refactored import paths for normalization layers in multiple files to use layer_utils. [1] [2] [3]

Modifications to LSTM Blocks

  • Added mLSTMblock and sLSTMblock classes in mambular/arch_utils/lstm_utils.py with detailed initialization and forward pass implementations. These blocks include convolutions, gated mechanisms, and projection layers.

Minor Code Cleanups

  • Removed unnecessary blank lines in mambular/arch_utils/mamba_utils/mamba_arch.py to improve code readability. [1] [2] [3]

@AnFreTh AnFreTh merged commit 54ca398 into develop Nov 5, 2024
@AnFreTh AnFreTh deleted the NDTF_LSTM branch November 5, 2024 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant