🚀 Awesome NLP and IR

Welcome to your ultimate resource hub for concepts and papers related to:

📘 Topics Covered

• Natural Language Processing (NLP)
• Large Language Models (LLMs)
• LLM Efficiency and Inference Optimization
• Vector Search and Information Retrieval
• Retrieval Augmented Generation, RAG Optimization and Best Practices

This repository offers a comprehensive and structured guide to the latest advancements, research, techniques, and best practices in NLP and Retrieval-Augmented Generation (RAG). It serves as a valuable resource for researchers, developers, and enthusiasts, covering everything from foundational NLP concepts to advanced RAG techniques. Leverage this repository to deepen your expertise, enhance your projects, and stay abreast of cutting-edge developments in NLP and IR.

Repository Structure

Awesome-NLP-and-IR
│
├── README.md
├── LLM Efficiency and Inference Optimization
│   └── LLM Efficiency and Inference Optimization.md
├── LLMs Pretraining and FineTuning
│   └── LLMs, Pretraining and FineTuning.md
├── NLP
│   └── NLP.md
├── RAG and RAG Optimization
│   └── RAG, Optimization and Best Practices.md
├── Vector Search and IR
│   └── Vector Search and IR.md
└── images

Table of Contents

• Preprocessing
  • Case Folding
  • Contraction Mapping
  • Correcting Spelling Errors
    • Dictionary-based approaches
    • Edit distance algorithms
    • Rule-based methods
  • Deduplication / Duplicate Removal
    • Using Pandas for Deduplication
    • Using Fuzzy Matching for Deduplication
  • Expanding Abbreviations and Acronyms
    • Dictionary-Based Methods
    • Rule-Based Methods
    • Statistical Methods
  • Stemming
  • Lemmatization
    • Rule-Based Lemmatization
    • Lexical Resource-based Lemmatization
  • Noise Removing
    • Stripping HTML Tags
    • Removing Special Characters
    • Removing Stop Words
    • Removing Numerical Values
    • Handling Emojis and Emoticons
    • Removing Non-Linguistic Symbols
  • Tokenization
    • Word Tokenization
    • Subword Tokenization
      • Byte Pair Encoding (BPE)
      • WordPiece Tokenization
      • Unigram Tokenization
      • SentencePiece Tokenization

Statistical NLP

• Naive Bayes
• N-gram Language Model
• Markov Chain
• Hidden Markov Model (HMM)
• Conditional Random Fields (CRFs)

Representation Learning in NLP

• Encoding/Sparse Vectors
  • One Hot Encoding
  • Integer Encoding
  • Bag of Words
  • TF-IDF
  • BM25
• Embedding/Dense Vectors
  • Word2Vec
  • GloVe
  • FastText
  • ELMo
  • BERT
• Hybrid
  • SPLADE
• Encoding vs Embedding Comparision

Deep NLP

• Different Activation Functions

• Optimization Algorithms

  • Comparison of Gradient Descent

• Feedforward Neural Networks (FNN)

• Recurrent Neural Networks (RNN)

  • Long Short-Term Memory (LSTM)
  • Gated Recurrent Unit (GRU)
  • Bidirectional RNNs

• Transformers

  • Key Components of Transformer Architecture
  • Transformer Architectures: A Detailed Comparison

• References

Large Language Models (LLMs)

• Architecture

• LLM Pretraining

  • Self-Supervised Learning
    • Masked Language Modeling
    • Masked Multimodal Language Modeling
    • Next Sentence Prediction
    • Causal Language Modeling
    • Denoising Autoencoders
    • Contrastive Learning

• LLM Fine-Tuning

  • Supervised Fine Tuning
  • Full Fine-Tuning
  • PEFT (Parameter-Efficient Fine-Tuning)
    • Additive PEFT
      • Adapters
      • Soft Prompt-based Fine-tuning
      • IA^3
    • Selective PEFT
      • Unstructured Masking
      • Structured Masking
    • Background of Matrix Decomposition
    • Reparameterized PEFT
      • LoRA: Low-Rank Adaptation
      • DyLoRA: Dynamic LoRA
      • AdaLoRA: Adaptive LoRA
      • DoRA: Weight-Decomposed Low-Rank Adaptation
    • Hybrid PEFT
      • UniPELT
      • S4
      • MAM Adapter
      • LLM-Adapters
      • Neural Architecture Search (NAS)
  • MEFT (Memory-Efficient Fine Tuning)
    • LoRA-FA (LoRA with Frozen Activations)
    • HyperTuning
    • Memory-Efficient Zeroth-Order Optimizer (MeZO)
    • QLoRA: Quantized Low-Rank Adaptation
    • Expert-Specialized Fine-Tuning
    • Sparse Matrix Tuning
    • Representation Finetuning (ReFT)
  • Alignment-Based Fine-Tuning
    • RLHF
    • RLAIF
    • Direct Preference Optimization (DPO)
    • Identity Preference Optimization (IPO)
    • Kahneman-Tversky Optimization (KTO)
    • Odds Ratio Preference Optimization (ORPO)
    • Alignment Techniques Comparison

• References

LLM Efficiency

• LLM Efficiency, Need, and Benefits

• Data Level Optimization

  • Input Compression
    • Prompt Pruning
    • Prompt Summarization
  • Retrieval-Augmented Generation (RAG)
  • Output Organization
    • Skeleton-of-Thought (SoT)
    • SGD (Sub-Problem Directed Graph)

• Model Level Optimization

  • Efficient Structure Design
    • Mixture-of-Experts (MoE)
    • Switch Transformers
    • Differences between MoE and Switch Transformers
  • Efficient Attention Mechanisms
    • Multi Query Attention (MQA)
    • Group Query Attention (GQA)
    • Sliding Window Attention
    • Low-Complexity Attention Models
    • Low-Rank Attention
    • Flash Attention
  • Transformer Alternatives
    • State Space Models (SSM) and Mamba
    • RWKV: Reinventing RNNs for the Transformer Era
    • Extended Long Short-Term Memory (xLSTM)
    • Funnel-Transformer
    • Parameterization Improvements
  • Model Compression Techniques
    • Quantization
      • BitNet: 1-bit Quantization
    • Sparsification
      • Q-Sparse
    • Pruning

• System Level Optimization

  • Inference Engine Optimization
    • Graph and Operator Optimization
      • Runtime Profiling
      • Attention Operator Optimization
      • Linear Operator Optimization
    • Decoding Strategies
      • Autoregressive Decoding
      • Speculative Decoding
      • Lookahead Decoding
      • Multitoken Prediction
    • Graph-Level Optimization
      • Kernel Fusion
      • Layer Fusion
  • Challenges and Solutions in System-Level Optimization
    • Memory Management
    • Continuous Batching
    • Scheduling Strategies
    • Distributed Systems Optimization
    • Hardware Accelerator Design

• References

Vector Search and Information Retrieval

• Vector Search

  • Vector Representation in ML
  • Distance Metrics
    • Dot Product
    • Euclidean Distance
    • Manhattan Distance
    • Cosine Similarity
    • Jaccard Similarity
    • Hamming Distance
    • Earth Mover's Distance (EMD)
  • Vector Search Techniques and Their Applications
  • Nearest Neighbor Search
  • Problems with High Dimensional Data
  • Linear Search
  • Dimensionality Reduction
    • Principal Component Analysis
    • t-Distributed Stochastic Neighbor Embedding (t-SNE)
  • Approximate Nearest Neighbor (ANN) Search
    • Trade-Off Between Accuracy and Efficiency
    • Flat Indexing
    • Inverted Index
    • Locality-Sensitive Hashing (LSH)
    • Quantization and their types
    • Tree-Based Indexing in ANN
    • Random Projection in ANN
    • Graph-based Indexing for ANN Search
    • LSH Forest
    • Composite Indexing in ANN
  • Comparision between Different Indexing Techniques

• References

Retrieval Augmented Generation, RAG Optimization, and Best Practices

• RAG

  • Benefits of RAG
  • Limitations and Challenges Addressed by RAG
  • Types of RAG
    • Simple RAG
    • Simple RAG with Memory
    • Branched RAG
    • Adaptive RAG
    • Corrective RAG
    • Self RAG
    • Agentic RAG

• RAG Optimization and Best Practices

  • Challenges
  • RAG Workflow Components and Optimization
    • Query Classification
    • Document Processing and Indexing
      • Chunking
      • Metadata Addition
      • Embedding Models
      • Embedding Quantization
      • Vector Databases
    • Retrieval Optimization
      • Source Selection and Granularity
      • Retrieval Methods
    • Reranking and Contextual Curation
      • Reranking Methods
      • Repacking and Summarization
        • Repacking
        • Summarization
    • Generation Optimization
      • Language Model Fine-Tuning
      • Co-Training Strategies
    • Advanced Augmentation Techniques
      • Iterative Refinement
      • Recursive Retrieval
      • Hybrid Approaches
    • Evaluation and Optimization Metrics
      • Performance Metrics
      • Benchmark Datasets
    • Tools and Platforms for Optimization
    • Recommendations for Implementing RAG Systems

• References

🤝 Contributions

Contributions are welcome! Please feel free to submit issues, fork the repository, and make pull requests to enhance the repository.

📄 License

This repository is licensed under the Apache License. See the LICENSE file for more information.

{
  "title": "🚀 Awesome NLP and IR",
  "author": "Mohammad Kaif",
  "date": "July 2024",
  "sections": [
    {
      "type": "introduction",
      "content": "Welcome to the ultimate resource hub for:\n\n- **Natural Language Processing (NLP)**\n- **Large Language Models (LLMs)**\n- **LLM Efficiency**\n- **Vector Search and Information Retrieval**\n- **Retrieval Augmented Generation (RAG) Optimization and Best Practices**\n\nThis repository offers a comprehensive, well-structured, and user-friendly guide to the latest advancements, research papers, techniques, and best practices in these fields. Ideal for researchers and developers alike."
    }
  ]
}