NikolasMarkou/dl_techniques

In the rapidly evolving landscape of AI research, groundbreaking techniques are often scattered across countless repositories, disparate implementations, and dense academic papers. dl_techniques emerges as a unified, curated, and production-ready arsenal for advanced deep learning. Pioneered and sponsored by Electi Consulting, this library is more than a collection of components—it is the definitive toolkit for researchers and engineers to design, train, and dissect state-of-the-art neural networks.

We bridge the chasm between theoretical innovation and practical application, providing faithful, efficient, and enterprise-validated components. From next-generation attention mechanisms and graph neural networks to information-theoretic loss functions and an unparalleled model analysis suite, dl_techniques is your strategic advantage for pushing the boundaries of deep learning.

---

Table of Contents

1. Key Features: A Glimpse into the Arsenal.
2. Why dl_techniques?: Your Unfair Advantage in AI R&D.
3. Installation: Get Up and Running in Minutes.
4. Quick Start: See the Library in Action.
5. In-Depth Documentation: From Theory to Tensors.
6. Project Structure: A Tour of the Repository.
7. Contributing: Join Our Research & Development Efforts.
8. License: Understanding the Apache 2.0 License.
9. Acknowledgments: Recognition and Support.
10. Citations & References: The Research That Inspired This Library.

---

Key Features

This library is a comprehensive suite of tools organized into five key pillars, developed through rigorous research and validated in real-world enterprise applications:

1. State-of-the-Art Architectures & Models (102 Leaf Packages, 12 Families)

• Next-Generation Language Models (19 packages): Production-ready implementations of BERT, Gemma, Qwen (dense, Qwen3Next, embeddings), Mamba, ModernBERT, DistilBERT, GPT-2, PtrM, alongside the Byte Latent Transformer (BLT), the Hierarchical Reasoning Model (HRM), H-Net, the Tiny Recursive Model, TopoLM, Tree Transformer, Wave Field, Zamba2, late-interaction retrieval with ColBERT (v1 and v2), and Fourier token mixing in FNet and FFTNet.
• Vision & Multimodal Powerhouses (61 packages across vision + vision_language): A comprehensive suite including CLIP, CliffordCLIP, MobileCLIP v1 and v2 in one package (v2 is the faithful port on FastViT; v1 retained for compatibility), FastVLM (vision-only despite the name), NanoVLM, BiT Diffusion, DINOv1/v2/v3, ViT-SigLIP (a ViT with a two-stage conv patch stem, not SigLIP's sigmoid contrastive objective), BEiT, Swin Transformer, MAE, Video-JEPA, LeVJEPA, the latent-energy world model LEWM, object detection with DETR, YOLOv12, RADConvNet, keypoints with SuperPoint, LightGlue, DaSiamRPN, DeepSORT, SiamFC, MambaLCT, and Segment Anything (SAM 1, 2 and 3).
• Advanced CNNs (50 vision packages): A rich collection including ConvNeXtV1/V2, ConvUNeXt, MobileNetV1-V4, ResNet, AlexNet, the recursively-defined FractalNet, the complex shearlet-based CoShNet, attention-augmented CBAM, and the ultra-efficient SqueezeNet family, plus OmniPoint, RBFProtoNet, AccuNet.
• Time Series & Forecasting (8 packages): State-of-the-art forecasting models including the probabilistic TiRex with quantile prediction, an enhanced implementation of N-BEATS (with NBEATSx), autoregressive DeepAR, PRISM, the novel xLSTM, Mixture Density Networks, Adaptive EMA, and pure Exponential Smoothing (ETS).
• Generative Modeling & Image Restoration: Diffusion and flow-matching transformers (SD3 MMDiT, Ideogram4, BiT Diffusion, DiT), a complete Variational Autoencoder (VAE) framework with VQ-VAE variants (including rotation-based codebook updates), Topographic VAE, arbitrary-scale super-resolution (THERA, PFT-SR), and restoration/denoising backbones (DarkIR, SCUNet, ACC-UNet, PW-FNet, DocRes, DocScanner, and a family of bias-free denoisers).
• Specialized Models: Task-specific models like DepthAnything for monocular depth estimation, full Capsule Networks (CapsNet) with dynamic routing, TabM for tabular data, model-agnostic inference-time power_sampling for any causal LM/VLM, and unsupervised aligners like Mini-Vec2Vec.
• Graph Neural Networks & Special Architectures (3 + 2 + 1 + 1 + 4 packages): Graph family: RELGT, Graph Energy Transformer, SHGCN; Neural Computer: NTM, NAM; Memory: Self-Organizing Maps; Point Cloud: Latent GMM Registration; General Purpose (4): KAN, HKAN, PowerMLP, MothNet; Embeddings Experimental (4): ASCII BERT, ASCII Clifford BERT, ASCII ConvNeXt BERT, and the shared study skeleton.

2. A Modular Arsenal of Advanced Layers (~375 modules, 21+ subpackages)

• Pioneering Attention Mechanisms: Go beyond standard attention with DifferentialMultiHeadAttention, modern HopfieldAttention, GroupQueryAttention, CapsuleRoutingAttention, and efficient alternatives like FNetFourierTransform and RingAttention.
• Unified Factory Architecture: A consistent, powerful factory system for creating and validating 36 attention mechanisms (ATTENTION_REGISTRY), 19 normalization variants (_TYPE_TO_CLASS — including BandRMS, LogitNorm), 22 Feed-Forward Network (FFN) types (FFN_REGISTRY — SwiGLU, GeGLU, OrthoGLU), 25 activations (ACTIVATION_REGISTRY), 15 embeddings (EMBEDDING_REGISTRY), plus registries for Adapters (ADAPTER_REGISTRY: 1), Mixtures (MIXTURE_REGISTRY: 4), Sequence Pooling (SEQUENCE_POOLING_REGISTRY: 4), Logic (LOGIC_REGISTRY: 4), and task heads dispatched by domain via create_head.
• Graph & Structural Primitives: Configurable GNN layers with multiple aggregation strategies (GCN, GAT, GraphSAGE), Relational Graph Transformer (RELGT) blocks, and Entity-Graph Refinement for learning hierarchical relationships.
• Mixture of Experts (MoE) System: A complete MoE implementation with configurable FFN experts, multiple gating strategies (including SoftMoE and Cosine Gating), and integrated training utilities.
• Probabilistic & Statistical Layers: Build models that reason about uncertainty with Mixture Density Networks (MDN), Normalizing Flows, and time series analysis layers for residual autocorrelation (ResidualACFLayer).
• Advanced Embeddings: A full suite of positional and semantic embeddings including standard, sinusoidal, Rotary Position Embedding (RoPE), and its modern variants like DualRotaryPositionEmbedding.

3. A Unified Command Center for Model Analysis & Introspection

• Holistic Model Analysis: A powerful ModelAnalyzer to benchmark models across five critical dimensions: training dynamics, weight health, prediction calibration, information flow, and advanced spectral analysis. Its modular design includes specialized analyzers like CalibrationAnalyzer, WeightAnalyzer, and InformationFlowAnalyzer.
• Publication-Ready Visualizations: Automatically generate insightful visualizations, interactive summary dashboards, and comparative analysis plots with integrated statistical significance testing.
• Predictive Generalization with Spectral Analysis: Integrate the power of WeightWatcher through our SpectralAnalyzer to assess generalization potential by analyzing the spectral properties (eigenvalues) of weight matrices—often without needing test data.
• Deep Diagnostic Toolkit: Move beyond accuracy to diagnose overconfidence (CalibrationAnalyzer), information bottlenecks (InformationFlowAnalyzer), weight decay and similarity (WeightAnalyzer), and learning efficiency (TrainingDynamicsAnalyzer).

4. Next-Generation Loss Functions & Optimization (56 Loss Modules, 17 Optimizers)

• Optimize What Matters with AnyLoss: A groundbreaking framework that transforms any confusion-matrix-based metric (e.g., F1-score, Balanced Accuracy, Matthews Correlation Coefficient) into a differentiable loss function for direct optimization on imbalanced data — 16 losses in ANYLOSS_REGISTRY plus standalone losses.
• Calibration & Robust Losses: Train better-calibrated models with GoodhartAwareLoss (cross-entropy plus a per-sample confidence penalty, with an optional anti-collapse term), calibration-focused losses like BrierScoreLoss, the uncertainty-aware FocalUncertaintyLoss, and DINO's self-distillation loss.
• Domain-Specific Loss Functions: Specialized losses for vision-language (CLIPContrastiveLoss, SigLIPLoss), segmentation (Dice, Focal, Tversky), time series (MASELoss, SMAPELoss), and generative modeling (WassersteinLoss with gradient penalty).
• Advanced Optimization Suite: 17 optimizer/scheduler modules including custom optimizers (Muon, VSGD, SGLD, Gefen, WW-PGD), smart learning rate schedulers like WarmupSchedule, utilities for DeepSupervision in multi-scale architectures, and a suite of advanced regularizers (SoftOrthogonal, SRIP).

5. Enterprise-Grade Training & Deployment Infrastructure

• Accelerated Development with Training Pipelines: ~70 ready-to-use train_*.py entry points across ~74 trainer directories (src/train/, plus the shared src/train/common/ library and 2 family containers train/language/, train/time_series/), establishing standardized and reproducible workflows for training, validation, and testing across domains like NLP, Vision, and Time Series.
• Production-Ready Utilities: A suite of tools including advanced data loaders, augmentation pipelines, a structured visualization and logging manager (VisualizationManager), and enhanced model serialization with custom object support.
• Assured Reliability: An extensive 1,558-module test suite (tests/) ensures the correctness and stability of every component, with dedicated fixtures for mixed-precision and TF32-sensitive regressions. It mirrors src/dl_techniques/ directory-for-directory, with the deliberate exception of tests/test_models/, which stays flat rather than following the model family nesting.
• Verified Against the Source: Where a reference implementation exists, ports are checked against it numerically — several packages commit the reference itself as an executable oracle (e.g. src/dl_techniques/layers/matching/ against tests/lightglue_reference_numpy.py) rather than asserting parity in prose.

---

Why dl_techniques?

• From Theory to Tensors, Instantly: We consolidate the fragmented landscape of AI research. Instead of hunting down dozens of disparate GitHub repos, you get a single, cohesive framework with faithful implementations of cutting-edge research.
• Built for Battle, Validated in the Enterprise: This is not a toy library. Every component has been hardened and validated in demanding enterprise applications, ensuring robustness, efficiency, and production-readiness.
• Unprecedented Introspection: Move beyond accuracy scores. Our first-class analysis toolkit is designed to answer the why behind your model's behavior, providing deep insights that are critical for building trustworthy and reliable AI.
• Engineered for Experimentation: Our innovative factory patterns and modular design are built for rapid prototyping. Swap attention mechanisms, normalization layers, or even entire architectural blocks with a single line of code.
• Modern, Maintainable, and Future-Proof: Built from the ground up for Keras 3 and modern Python, dl_techniques adheres to the highest standards of software engineering, ensuring it's easy to use, extend, and maintain.

---

Installation

Note: This library requires Python 3.11+ and Keras 3.8.0 with the TensorFlow 2.18.0 backend.

1. Clone the repository:

   git clone https://github.com/nikolasmarkou/dl_techniques.git
   cd dl_techniques
   

2. Install dependencies: For standard usage, install the library and its core dependencies:

   pip install .
   

3. Editable Install (for developers): If you plan to contribute or modify the library's source code, install it in editable mode with development tools:

   pip install -e ".[dev]"
   

   This installs additional tools such as pytest, pytest-cov, pylint, and pre-commit.

4. Optional Extras: The Streamlit front-end under src/applications/bias_free_denoiser/ needs an extra that is not part of a core install:

   pip install ".[apps]"
   

   The tokenizer utilities, the HuggingFace/tensorflow-datasets-backed loaders under dl_techniques.datasets and the official DINOv2 checkpoint reader (safetensors, used by DINOv2 and Depth Anything with pretrained=True) need their own extra:

   pip install ".[data]"
   

5. Verify Installation:

   python -c "import dl_techniques; print('Installation successful!')"
   

Development Commands

make test       # full pytest suite (~1.5 hours; also the pre-push hook)
make clean      # remove build artifacts and __pycache__
make structure  # print the src/ tree

Scope pytest to what you changed rather than running the whole suite as a regression check:

pytest tests/test_layers/test_norms/ -vvv

Training scripts need a non-interactive matplotlib backend on headless machines:

MPLBACKEND=Agg python -m train.<pipeline>.train_<script> [args]

No trained weights ship with this repository. Training outputs live under results/, which is gitignored, so a fresh clone contains source, tests and prose only. Any results/<run>/final_model.keras path quoted in a trainer default, a paper or an application is a local artifact — you train it yourself first. See REPO_MAP.md § What a fresh clone actually contains.

---

Quick Start

1. Compose a State-of-the-Art Transformer Block

Effortlessly construct and experiment with modern transformer components using our unified factory system.

import keras
from dl_techniques.layers.attention.factory import create_attention_layer
from dl_techniques.layers.norms.factory import create_normalization_layer
from dl_techniques.layers.ffn.factory import create_ffn_layer

inputs = keras.Input(shape=(1024, 512))

# Use factories for consistent, validated component creation
attention = create_attention_layer(
    'differential',
    dim=512,
    num_heads=8,
    head_dim=64
)
norm = create_normalization_layer('rms_norm', epsilon=1e-6)
ffn = create_ffn_layer('swiglu', output_dim=512)

# Build a modern transformer block
x = attention(inputs)
x = norm(x)
x = ffn(x)

model = keras.Model(inputs, x)
model.summary()

2. Deploy a Probabilistic Time Series Forecaster

Instantiate a state-of-the-art univariate time series model capable of generating robust, uncertainty-aware forecasts.

from dl_techniques.models.time_series.tirex.model import create_tirex_model
from dl_techniques.losses.quantile_loss import QuantileLoss

# TiRex: probabilistic univariate forecasting with quantile prediction
model = create_tirex_model(
    input_length=100,            # length of the input context window
    prediction_length=24,        # forecast horizon
    quantile_levels=[0.1, 0.5, 0.9],  # 80% prediction interval + median
)

# Train directly for calibrated uncertainty with a quantile loss.
# Predictions are (batch, prediction_length, n_quantiles), so a plain
# point-forecast metric like 'mae' would broadcast across the quantile axis.
model.compile(
    optimizer='adamw',
    loss=QuantileLoss(quantiles=[0.1, 0.5, 0.9]),
)

3. Contrastive Vision-Language Learning with CLIP

Map images and text into a single shared embedding space with a CLIP dual encoder, unlocking zero-shot classification and cross-modal retrieval.

import keras
from dl_techniques.models.vision_language.clip import create_clip

# CLIP ViT-B/32: a dual encoder mapping images and text into one space
model = create_clip("ViT-B/32")
model.build({"image": (None, 224, 224, 3), "text": (None, 77)})

images = keras.random.normal((4, 224, 224, 3))
tokens = keras.ops.cast(keras.random.uniform((4, 77), 0, 49408), "int32")

# Full contrastive forward pass
outputs = model({"image": images, "text": tokens}, training=False)
# outputs["image_features"]   -> (4, 512), L2-normalised
# outputs["text_features"]    -> (4, 512), L2-normalised
# outputs["logits_per_image"] -> (4, 4),  similarity matrix

# Or encode each modality independently for retrieval
image_features = model.encode_image(images)
text_features = model.encode_text(tokens)

4. Dissect Model Behavior with the Analysis Engine

Go beyond surface-level metrics and gain deep, actionable insights into your models' performance and behavior.

from dl_techniques.analyzer import ModelAnalyzer, AnalysisConfig, DataInput

# Compare multiple models across a suite of deep diagnostics
models = {'TiRex_Model': tirex_model, 'Baseline_LSTM': lstm_model}
histories = {'TiRex_Model': tirex_history, 'Baseline_LSTM': lstm_history}
test_data = DataInput(x_test, y_test)

# Configure a comprehensive analysis run
config = AnalysisConfig(
    analyze_training_dynamics=True,
    analyze_calibration=True,
    analyze_weights=True,
    analyze_spectral=True, # Unleash spectral analysis for generalization insights
    save_plots=True,
    plot_style='publication'
)

analyzer = ModelAnalyzer(models, config=config, training_history=histories)

# Execute the complete analysis and generate a full suite of visualizations
results = analyzer.analyze(test_data)

# Access detailed, structured metrics programmatically for automated reporting
print(f"Best calibrated model (by ECE): {min(results.calibration_metrics.items(), key=lambda x: x[1]['ece'])}")
print(f"Training efficiency ranking (epochs to converge): {results.training_metrics.epochs_to_convergence}")

5. Optimize Directly for F1-Score on Imbalanced Data

Stop tuning class weights and start optimizing your target metric directly with the AnyLoss framework.

from dl_techniques.losses.any_loss import F1Loss, BalancedAccuracyLoss

# For imbalanced datasets, optimize F1-score directly
model.compile(
    optimizer='adamw',
    loss=F1Loss(from_logits=True),
    metrics=['accuracy', 'precision', 'recall']
)

# Alternatively, optimize for balanced accuracy
model.compile(
    optimizer='adamw',
    loss=BalancedAccuracyLoss(from_logits=True),
    metrics=['accuracy']
)

6. Harness Graph Neural Networks for Relational Data

Unlock insights from graph-structured data with our powerful and configurable GNN implementations.

import keras
from dl_techniques.layers.graphs.graph_neural_network import GraphNeuralNetworkLayer

# Create a Graph Attention Network (GAT) to process relational data
gnn = GraphNeuralNetworkLayer(
    concept_dim=256,
    num_layers=3,
    message_passing='gat',  # Use attention for message passing
    aggregation='attention',
    dropout_rate=0.1
)

# Apply the layer to graph-structured inputs
node_features = keras.Input(shape=(None, 256))  # Variable number of nodes
adjacency_matrix = keras.Input(shape=(None, None))

node_embeddings = gnn([node_features, adjacency_matrix])

---

In-Depth Documentation

This library is engineered to be a living knowledge base, bridging the gap between academia and industry. Our documentation is split into three primary resources: a navigational map of the repository, in-depth theoretical guides, and a comprehensive API reference.

Start Here: REPO_MAP.md

Before anything else, read REPO_MAP.md — a path-verified router for the repository. It answers where code of a given kind lives, how the registry and factory dispatch is wired, which trainer trains which model, and which of the many in-tree docs answers your question.

Tutorials & Deep Dives (research/)

Our research/ directory contains over 120 articles providing the theoretical foundations, implementation details, and best practices behind key components. Highlights include:

• Complete Transformer Guide (2025): A production-focused guide to implementing state-of-the-art Transformer architectures.
• Model Analyzer Guide: A comprehensive tutorial for the advanced model analysis toolkit.
• AnyLoss Framework: A deep dive into the theory and practice of direct metric optimization.
• Chronological Neural Architectures: An extensive chronological guide to influential architectures with implementation notes.
• Band-Constrained Normalization: A novel normalization technique that preserves magnitude information within bounded constraints.
• Mixture Density Networks: The theory and best practices for implementing probabilistic models.

Papers (research/papers/)

Five LaTeX manuscripts written against this codebase live under research/papers/, several with built PDFs: band_rms (band-constrained RMS normalization), bfunet (bias-free denoisers as image priors), cliffordnet_extensions, correlations, and logical_net.

API Reference (per-module docs)

For detailed documentation on every module, class, and function, browse the source tree directly: each subpackage ships a focused README.md (e.g. src/dl_techniques/analyzer/README.md) and a per-package AGENTS.md describing its conventions, patterns, and components. Every one of the 102 leaf model packages carries its own README.md. The research/ guides above complement these with the underlying theory.

---

Project Structure

The repository is organized for clarity, maintainability, and ease of contribution.

src/ is the import root: the library is imported as dl_techniques.*, the training pipelines as train.*, and the applications as applications.*.

dl_techniques/
├── src/dl_techniques/         # THE LIBRARY — 13 subpackages
│   ├── models/                # 102 leaf model packages in 11 FAMILY directories
│   │   ├── vision/            # 50 — resnet, convnext, vit, dino, swin_transformer, beit,
│   │   │                      #      yolo12, detr, vae, vq_vae, depth_anything, alexnet,
│   │   │                      #      and the nested image_restoration/, super_resolution/, keypoints/
│   │   ├── language/          # 19 — bert, gemma, qwen, mamba, modern_bert, gpt2, colbert,
│   │   │                      #      blt, hrm, hnet, ptrm, topolm, wave_field, zamba2, ...
│   │   ├── vision_language/   # 11 — clip, clifford_clip, mobile_clip, fastvlm, nano_vlm,
│   │   │                      #      sd3_mmdit, ideogram4, bit_diffusion, and the nested sam/{sam1,sam2,sam3}
│   │   ├── time_series/       # 8 — tirex, nbeats, xlstm, deepar, prism, mdn, adaptive_ema, ets
│   │   ├── general_purpose/   # 4 — kan, hkan, mothnet, power_mlp
│   │   ├── graph/             # 3 — relgt, graph_energy_transformer, shgcn
│   │   ├── neural_computer/   # 2 — ntm, nam
│   │   ├── common/            # 2 — power_sampling, lm_pretraining
│   │   ├── memory/            # 1 — som
│   │   ├── point_cloud/       # 1 — latent_gmm_registration
│   │   └── tabular/           # 1 — tabm
│   ├── layers/                # ~375 modules — ~200 in 21 themed subpackages,
│   │   │                      # ~75 loose at the top level
│   │   ├── attention/         # 36 registered attention mechanisms (factory)
│   │   ├── ffn/               # 22 registered feed-forward networks (factory)
│   │   ├── activations/       # 25 registered activations (factory)
│   │   ├── norms/             # 19 registered normalization layers (factory)
│   │   ├── embedding/         # 15 registered positional/semantic embeddings (factory)
│   │   ├── transformers/      # Assembled transformer/encoder/decoder blocks
│   │   ├── heads/             # Task heads dispatched by domain (nlp/vision/vlm); time_series heads import directly
│   │   ├── time_series/       # Forecasting-specific layers (heads are in heads/time_series)
│   │   ├── fastvit/           # FastViT primitives (+ a committed reference impl)
│   │   ├── graphs/            # Graph neural network components
│   │   ├── moe/               # Mixture of Experts (MoE) system
│   │   ├── memory/            # External / associative memory layers
│   │   ├── statistics/        # Statistical and probabilistic layers (MDN, Flows)
│   │   └── ...                # fusion, geometric, logic, mixtures, physics,
│   │                          # reasoning, sequence_pooling, tokenizers, adapters
│   ├── losses/                # 56 specialized loss modules (AnyLoss registry: 16 + standalone)
│   ├── metrics/               # Custom Keras metrics (PSNR, SSIM, perplexity, Brier, depth, forecasting)
│   ├── optimization/          # 17 optimizers (Muon, VSGD, SGLD, Gefen, WW-PGD), LR schedules
│   ├── analyzer/              # Comprehensive model analysis toolkit and visualizers
│   ├── visualization/         # Plotting helpers for training and evaluation
│   ├── datasets/              # Dataset loaders and synthetic generators
│   ├── callbacks/             # Reusable Keras callbacks
│   ├── regularizers/          # Advanced regularization techniques (SRIP, Orthogonal)
│   ├── initializers/          # Structured initializers (Gabor, Haar, orthonormal, KAN)
│   ├── constraints/           # Weight constraints
│   └── utils/                 # Core utilities, loggers, masking, and data handlers
├── src/train/                 # ~70 train_*.py entry points across ~74 trainer directories
├── src/applications/          # Deployable apps (today: bias_free_denoiser, Streamlit)
├── research/                  # 120+ in-depth articles, guides, and LaTeX papers
├── tests/                     # 1,558 test modules mirroring src/dl_techniques/
└── REPO_MAP.md

---

Contributing

We welcome contributions from the research community. Whether you are implementing a new technique, improving documentation, or fixing bugs, your input is valuable.

Getting Started

1. Fork & Clone the repository.
2. Set up the development environment: pip install -e ".[dev]".
3. Create a new branch for your feature: git checkout -b feature/new-technique.
4. Adhere to our development standards: Use type hints, write comprehensive tests, and document your code thoroughly.

Development Standards

• Code Quality: Follow PEP 8, use type hints, and rely on centralized logging via dl_techniques.utils.logger (no print).
• Testing: Develop in the .venv environment and write comprehensive tests using pytest, scoped to the modules you change (make test runs the full ~1.5h suite). Set MPLBACKEND=Agg when running training scripts on headless machines.
• Documentation: Document every public symbol, and update relevant guides in the research/ directory. Docstring style is not uniform across this repo — layers/ is predominantly Sphinx/reST (:param:), models/ is measurably mixed with the Sphinx exemplar models/language/bert/model.py as the model for new packages, and losses//metrics//utils//optimization//analyzer//visualization/ are Google-Args:-majority. Match the package you are editing and never convert a file wholesale; the measured counts and the greps that re-derive them are in src/dl_techniques/AGENTS.md § Core Conventions → Code Style.
• Validation: Include benchmarks or comparisons against reference implementations where applicable.

Contribution Types

• New Architectures: Implementations of recent, impactful research papers.
• Performance Improvements: Optimizations that maintain numerical accuracy.
• New Analysis Tools: Additional analyzers or visualizations for the ModelAnalyzer toolkit.
• Enhanced Documentation: Tutorials, guides, or improved API documentation.

---

License

This project is licensed under the Apache License 2.0.

Important Considerations:

• Permissive License: You may use, modify, and distribute the library, including in commercial and closed-source products.
• Attribution: Redistributions must retain the copyright notice and a copy of the license, and modified files must carry a notice stating that they were changed.
• Patent Grant: Contributors grant an express patent license covering their contributions.

See the LICENSE file for complete details.

---

Acknowledgments

This library is proudly sponsored and pioneered by Electi Consulting, a premier AI consultancy specializing in enterprise artificial intelligence, blockchain technology, and cryptographic solutions. The practical validation and enterprise-ready nature of these components has been made possible through Electi's extensive experience deploying state-of-the-art AI solutions across diverse industries:

• Financial Services: High-frequency trading, risk assessment, and fraud detection.
• Maritime Industry: Route optimization, predictive maintenance, and cargo management.
• Healthcare: Diagnostic assistance, treatment optimization, and clinical decision support.
• Manufacturing: Predictive maintenance, quality control, and supply chain optimization.

Special recognition is extended to the open-source community and the many researchers whose groundbreaking work forms the foundation of this library.

---

Citations & References

This library stands on the shoulders of giants. Our implementations are grounded in a rigorous study of the source papers that have defined the field of modern deep learning.

Transformers & Language Models

• Attention Is All You Need (Transformer): Vaswani, A., et al. (2017). NeurIPS.
• BERT: Pre-training of Deep Bidirectional Transformers: Devlin, J., et al. (2018). NAACL.
• RoFormer: Enhanced Transformer with Rotary Position Embedding: Su, J., et al. (2021). ACL.
• Differential Transformer: Ye, T., et al. (2024). ICLR 2025.
• Mamba: Linear-Time Sequence Modeling: Gu, A., & Dao, T. (2023).
• Gemma 3 Technical Report: Google DeepMind (2025).
• Qwen Technical Report: Bai, J., et al. (2023).
• Byte Latent Transformer: Patches Scale Better Than Tokens: Pagnoni, A., et al. (2024).
• ModernBERT: Smarter, Better, Faster, Longer: Warner, B., et al. (2024).
• DistilBERT, a distilled version of BERT: Sanh, V., et al. (2019).
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT: Khattab, O., & Zaharia, M. (2020). SIGIR.
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction: Santhanam, K., et al. (2022). NAACL.
• Tree Transformer: Integrating Tree Structures into Self-Attention: Wang, Y.-S., Lee, H.-Y., & Chen, Y.-N. (2019). EMNLP-IJCNLP.

Vision & Multimodal Models

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT): Dosovitskiy, A., et al. (2020). ICLR.
• A ConvNet for the 2020s (ConvNeXt): Liu, Z., et al. (2022). CVPR.
• ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders: Woo, S., et al. (2023). CVPR.
• MobileNetV2: Inverted Residuals and Linear Bottlenecks: Sandler, M., et al. (2018). CVPR.
• Searching for MobileNetV3: Howard, A., et al. (2019). ICCV.
• MobileNetV4: Universal Inverted Bottleneck and Mobile MQA: Li, Y., et al. (2024).
• DINOv2: Learning Robust Visual Features without Supervision: Oquab, M., et al. (2023).
• Sigmoid Loss for Language Image Pre-Training (SigLIP): Zhai, X., et al. (2023). ICCV.
• Learning Transferable Visual Models From Natural Language Supervision (CLIP): Radford, A., et al. (2021). ICML.
• End-to-End Object Detection with Transformers (DETR): Carion, N., et al. (2020). ECCV.
• FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization: Vasu, P. K. A., et al. (2023).
• BEiT: BERT Pre-Training of Image Transformers: Bao, H., et al. (2021). ICLR.
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows: Liu, Z., et al. (2021). ICCV.
• MobileCLIP2: Improving Multi-Modal Reinforced Training: Faghri, F., et al. (2025).
• Segment Anything: Kirillov, A., et al. (2023). ICCV.
• SAM 2: Segment Anything in Images and Videos: Ravi, N., et al. (2024).
• SuperPoint: Self-Supervised Interest Point Detection and Description: DeTone, D., et al. (2018). CVPR Workshops.
• V-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video: Bardes, A., et al. (2024).
• Masked Autoencoders Are Scalable Vision Learners (MAE): He, K., et al. (2022). CVPR.
• CBAM: Convolutional Block Attention Module: Woo, S., et al. (2018). ECCV.
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data: Yang, L., et al. (2024). CVPR.

Generative Modeling & Image Restoration

• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (SD3 / MMDiT): Esser, P., et al. (2024). ICML.
• Flow Matching for Generative Modeling: Lipman, Y., et al. (2023). ICLR.
• Auto-Encoding Variational Bayes (VAE): Kingma, D. P., & Welling, M. (2013).
• Neural Discrete Representation Learning (VQ-VAE): van den Oord, A., et al. (2017). NeurIPS.
• Robust and Interpretable Blind Image Denoising via Bias-Free Convolutional Neural Networks: Mohan, S., et al. (2020). ICLR.
• Thera: Aliasing-Free Arbitrary-Scale Super-Resolution with Neural Heat Fields: Becker, A., et al. (2025). CVPR.
• DarkIR: Robust Low-Light Image Restoration: Feijoo, D., et al. (2025). CVPR.
• Global Modeling Matters: A Fast, Lightweight and Effective Baseline for Efficient Image Restoration (PW-FNet): Jiang, X., et al. (2025).

Advanced Attention & FFN Mechanisms

• Modern Hopfield Networks is All You Need: Ramsauer, H., et al. (2020). ICML.
• GQA: Training Generalized Multi-Query Transformer Models: Ainslie, J., et al. (2023).
• Ring Attention with Blockwise Transformers for Near-Infinite Context: Liu, H., et al. (2024).
• Rethinking Attention with Performers: Choromanski, K., et al. (2020).
• FNet: Mixing Tokens with Fourier Transforms: Lee-Thorp, J., et al. (2021).
• GLU Variants Improve Transformer: Shazeer, N. (2020).
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer: Shazeer, N., et al. (2017).
• Pay Attention to MLPs (gMLP): Liu, H., et al. (2021). NeurIPS.

Graph Neural Networks & Special Architectures

• Graph Neural Networks: A Review: Wu, Z., et al. (2020).
• Graph Attention Networks: Veličković, P., et al. (2018). ICLR.
• Semi-Supervised Classification with Graph Convolutional Networks: Kipf, T. N., & Welling, M. (2016).
• Dynamic Routing Between Capsules: Sabour, S., et al. (2017). NeurIPS.
• KAN: Kolmogorov-Arnold Networks: Liu, Z., et al. (2024).
• FractalNet: Ultra-Deep Neural Networks without Residuals: Larsson, G., et al. (2016).
• Energy Transformer: Hoover, B., et al. (2023). NeurIPS.
• Neural Turing Machines: Graves, A., et al. (2014).
• Geometric Clifford Algebra Networks: Ruhe, D., et al. (2023). ICML.
• Hyperbolic Graph Convolutional Neural Networks: Chami, I., et al. (2019). NeurIPS.
• sHGCN: Simplified hyperbolic graph convolutional neural networks: Arevalo, P., Molina, A., & Ciudad, A. (2025).
• The Self-Organizing Map: Kohonen, T. (1990). Proceedings of the IEEE, 78(9).

Time Series & Forecasting

• N-BEATS: Neural basis expansion analysis for interpretable time series forecasting: Oreshkin, B. N., et al. (2019). ICLR.
• Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: Lim, B., et al. (2021).
• DeepAR: Probabilistic forecasting with autoregressive recurrent networks: Salinas, D., et al. (2020).
• xLSTM: Extended Long Short-Term Memory: Beck, M., et al. (2024). NeurIPS.
• Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift: Kim, T., et al. (2021). ICLR.
• TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning: Auer, A., et al. (2025).
• PRISM: A hierarchical multiscale approach for time series forecasting: Chen, Z., et al. (2025).

Tabular & Inference-Time Methods

• TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling: Gorishniy, Y., et al. (2024).
• Reasoning with Sampling: Your Base Model is Smarter Than You Think: Karan, A., et al. (2025).

Loss Functions, Optimization, & Regularization

• AnyLoss: A General and Differentiable Framework for Classification Metric Optimization: Han, D., et al. (2024).
• Focal Loss for Dense Object Detection: Lin, T. Y., et al. (2017). ICCV.
• On calibration of modern neural networks: Guo, C., et al. (2017). ICML.
• Wasserstein GAN: Arjovsky, M., et al. (2017). ICML.
• Root Mean Square Layer Normalization: Zhang, B., & Sennrich, R. (2019). NeurIPS.
• Can We Gain More from Orthogonality Regularizations in Training Deep Networks?: Bansal, N., et al. (2018). NeurIPS.
• Predicting the Generalization Gap in Deep Networks with Margin Distributions: Martin, C., & Mahoney, M. W. (2019). ICLR.

Complete bibliographic information is available in the documentation for individual modules.