Furyton/awesome-language-model-analysis
Furyton/awesome-language-model-analysis has 103 GitHub stars and 2 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and 0 in 15 days. Last checked 2026-11-18 04:05 UTC. Last push 2026-07-29.
What you can do
- Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch [paper link] 2026-07-04Mary Letey;Yue M. Lu;Cengiz Pehlevan;Jacob Zavatone-Veth
- The risk of KV cache compression [paper link] 2026-07-01Lukas Haverbeck;Carmen Amo Alonso;Andres Felipe Posada-Moreno;Sebastian Trimpe;Marco Pavone
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation [paper link] 2026-06-30Jiachun Li;David Simchi-Levi
- Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework [paper link] 2026-06-29Haobo Yang
Measured star growth
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- 0
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through theoretical analysis, empirical analysis, or a combination of both.
AIREADME.md
OriginalAwesome Language Model Analysis 
This paper list focuses on the theoretical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through formal/mathematical analysis -- proofs, provable guarantees, bounds, expressivity results, convergence analysis, and similar. Papers that also include supporting experiments still count; purely empirical/observational papers do not.
Scope of this list:
- Currently, this list focuses on transformer-based models.
- We collect papers with a genuine theoretical contribution, instead of purely empirical studies or papers that aim to improve the performance of language models without theoretically analyzing why.
Limitations of this list:
- This list is not exhaustive, and we may miss some very important papers.
- This list is not well-organized yet, and we may need to reorganize the list in the future.
- Some popular topics are not well-covered yet, such as mechanistic engineering, probing, and interpretability.
Statistics of This paper list:
- Total number of different papers: 721
- For more detailed statistics, please refer to the end of this page.
If you have any suggestions or want to contribute, please feel free to open an issue or a pull request.
For details on how to contribute, please refer to the contribution guidelines.
You can also share your thoughts and discuss with others in the Discussions.
[!NOTE]
For uncategorized version, please refer to here.
Table of Content
- Awesome Language Model Analysis
- Table of Content
- Phenomena of Interest
- In-Context Learning
- Chain-of-Thought
- Hallucination
- Reversal Curse
- Scaling Laws / Emergent Abilities / Grokking / etc.
- Knowledge / Memory Mechanisms
- Training Dynamics / Landscape / Optimization / Fine-tuning / etc.
- Learning / Generalization / Reasoning / Weak to Strong Generalization
- Other Phenomena / Discoveries
- Representational Capacity
- Architectural Effectivity
- Training Paradigms
- Mechanistic Engineering / Probing / Interpretability
- Miscellanea
- Phenomena of Interest
Phenomena of Interest
Categories focusing on different phenomena, properties, and behaviors observed in large language models (LLMs) and transformer-based models.
In-Context Learning
Papers focusing on the theoretical and empirical analysis of in-context learning in large language models.
paper list (click to fold / unfold)
A Unified Framework for In-Context Learning with Causal and Masked Language Models [paper link] 2026-07-05
Chenrui Liu;Chuanlong Xie;Falong Tan;Yicheng Zeng;Lixing ZhuSequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch [paper link] 2026-07-04
Mary Letey;Yue M. Lu;Cengiz Pehlevan;Jacob Zavatone-VethBest-of-Better-$N$: Generating Pre-Aligned Responses with In-Context Learning [paper link] 2026-07-03
Eric Lei;Hsiang Hsu;Chun-Fu ChenInduction Heads Interpolate N-Grams [paper link] 2026-07-02
Francesco D'Angelo;Oguz Kaan Yuksel;Swathi Shree Narashiman;Nicolas FlammarionFrom Approximation to Emergence: A Theory of Deep Learning [paper link] 2026-07-01
Zhilin ZhaoGhost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization [paper link] 2026-07-01
Peilin Liu;Ding-Xuan ZhouTransformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation [paper link] 2026-06-30
Jiachun Li;David Simchi-LeviA Theoretical Interpretation of In-Context Learning via Probabilistic Modeling [paper link] 2026-06-27
Zhenyu Liu;Huaze Tang;Shao-Lun HuangWhat Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations? [paper link] 2026-06-18
Sihui Dai;Mann PatelCan In-Context Learning Support Intrinsic Curiosity? [paper link] 2026-06-17
Eric Elmoznino;Sangnie Bhardwaj;Johannes von Oswald;Rajai Nasser;Blaise Agüera y Arcas;João Sacramento;Rif A. Saurous;Guillaume LajoieRevisiting the Systematicity in Negation in the Era of In-Context Learning [paper link] 2026-06-15
Hitomi Yanaka;Taisei YamamotoAdaptive inference and function vectors in deep transformers [paper link] 2026-06-15
Ravin Raj;Gautam ReddyWhere Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function [paper link] 2026-06-11
Atharva Gupta;Dhruv Kumar;Murari Mandal;Saurabh DeshpandeThe Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces [paper link] 2026-06-05
Soo Min Kwon;Alec S. Xu;Can Yaras;Dogyoon Song;Laura Balzano;Qing QuTransformers are Deep Optimizers: Provable In-Context Learning for Deep Model Training [paper link] 2024-11-25
Weimin Wu; Maojiang Su; Jerry Yao-Chieh Hu; Zhao Song; Han LiuCan a Large Language Model Learn Matrix Functions In Context? [paper link] 2024-11-24
Paimon Goulart; Evangelos E. PapalexakisTransformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Models [paper link] 2024-11-13
Chengshuai Shi; Kun Yang; Jing Yang; Cong ShenAdversarial Robustness of In-Context Learning in Transformers for Linear Regression [paper link] 2024-11-07
Usman Anwar; Johannes Von Oswald; Louis Kirsch; David Krueger; Spencer FreiProvable In-Context Learning with Transformers: A Case Study on Linear Regression [paper link] 2024-11-04
Dake Bu; Wei Huang; Andi Han; Atsushi Nitanda; Taiji Suzuki; Qingfu Zhang; Hau-San WongPretrained transformer efficiently learns low-dimensional target functions in-context [paper link] 2024-11-04
Kazusato Oko; Yujin Song; Taiji Suzuki; Denny WuToward Understanding In-context vs. In-weight Learning [paper link] 2024-10-30
Bryan Chan; Xinyi Chen; András György; Dale SchuurmansOn the Role of Depth and Looping for In-Context Learning with Task Diversity [paper link] 2024-10-29
Khashayar Gatmiry; Nikunj Saunshi; Sashank J. Reddi; Stefanie Jegelka; Sanjiv KumarMechanisms of Symbol Processing for In-Context Learning in Transformer Networks [paper link] 2024-10-23
Paul Smolensky; Roland Fernandez; Zhenghao Herbert Zhou; Mattia Opper; Jianfeng GaoCan Transformers In-Context Learn Behavior of a Linear Dynamical System? [paper link] 2024-10-21
Usman Akram; Haris VikaloBayesian scaling laws for in-context learning [paper link] 2024-10-21
Aryaman Arora; Dan Jurafsky; Christopher Potts; Noah D. GoodmanProvable In-context Learning for Mixture of Linear Regressions using Transformers [paper link] 2024-10-18
Yanhao Jin; Krishnakumar Balasubramanian; Lifeng LaiIn-context learning and Occam's razor [paper link] 2024-10-17
Eric Elmoznino; Tom Marty; Tejas Kasetty; Leo Gagnon; Sarthak Mittal; Mahan Fathi; Dhanya Sridhar; Guillaume LajoieContext-Scaling versus Task-Scaling in In-Context Learning [paper link] 2024-10-16
Amirhesam Abedsoltan; Adityanarayanan Radhakrishnan; Jingfeng Wu; Mikhail BelkinBypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent [paper link] 2024-10-15
Bo Chen; Xiaoyu Li; Yingyu Liang; Zhenmei Shi; Zhao SongHow Transformers Implement Induction Heads: Approximation and Optimization Analysis [paper link] 2024-10-15
Mingze Wang; Ruoxi Yu; Weinan E; Lei WuOn the Training Convergence of Transformers for In-Context Classification [paper link] 2024-10-15
Wei Shen; Ruida Zhou; Jing Yang; Cong ShenTransformers learn variable-order Markov chains in-context [paper link] 2024-10-07
Ruida Zhou; Chao Tian; Suhas DiggaviRevisiting In-context Learning Inference Circuit in Large Language Models [paper link] 2024-10-06
Hakaze Cho; Mariko Kato; Yoshihiro Sakai; Naoya InoueTrained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context [paper link] 2024-10-02
Spencer Frei; Gal VardiTransformers Handle Endogeneity in In-Context Linear Regression [paper link] 2024-10-02
Haodong Liang; Krishnakumar Balasubramanian; Lifeng LaiUnveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [paper link] 2024-09-10
Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran YangLearning vs Retrieval: The Role of In-Context Examples in Regression with LLMs [paper link] 2024-09-06
Aliakbar Nafar; Kristen Brent Venable; Parisa KordjamshidiTransformers are Minimax Optimal Nonparametric In-Context Learners [paper link] 2024-08-22
Juno Kim; Tai Nakamaki; Taiji SuzukiMemorisation In In-Context Learning [paper link] 2024-08-21
Shahriar Golchin; Mihai Surdeanu; Steven Bethard; Eduardo Blanco; Ellen RiloffIn-Context Learning with Representations: Contextual Generalization of Trained Transformers [paper link] 2024-08-19
Tong Yang; Yu Huang; Yingbin Liang; Yuejie ChiFast Training Dataset Attribution via In-Context Learning [paper link] 2024-08-14
Milad Fotouhi; Mohammad Taha Bahadori; Oluwaseyi Feyisetan; Payman Arabshahi; David HeckermanHow Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression [paper link] 2024-08-08
Xingwu Chen; Lei Zhao; Difan ZouTransformers are Universal In-context Learners [paper link] 2024-08-02
Takashi Furuya; Maarten V. de Hoop; Gabriel PeyréPolynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment [paper link] 2024-07-27
Max Wilcoxson; Morten Svendgård; Ria Doshi; Dylan Davis; Reya Vir; Anant SahaiUnveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism [paper link] 2024-07-24
Anhao Zhao; Fanghua Ye; Jinlan Fu; Xiaoyu ShenOne-Layer Transformer Provably Learns One-Nearest Neighbor In Context [paper link] 2024-07-24
Zihao Li; Yuan Cao; Cheng Gao; Yihan He; Han Liu; Jason M. Klusowski; Jianqing Fan; Mengdi WangWhen can transformers compositionally generalize in-context? [paper link] 2024-07-17
Seijin Kobayashi; Simon Schug; Yassir Akram; Florian Redhardt; Johannes von Oswald; Razvan Pascanu; Guillaume Lajoie; João SacramentoIn-Context In-Context Learning with Transformer Neural Processes [paper link] 2024-06-19
Matthew Ashman; Cristiana Diaconu; Adrian Weller; Richard E. TurnerProbing the Decision Boundaries of In-context Learning in Large Language Models [paper link] 2024-06-17
Siyan Zhao; Tung Nguyen; Aditya GroverState Soup: In-Context Skill Learning, Retrieval and Mixing [paper link] 2024-06-12
Maciej Pióro; Maciej Wołczyk; Razvan Pascanu; Johannes von Oswald; João SacramentoEstimating the Hallucination Rate of Generative AI [paper link] 2024-06-11
Andrew Jesson; Nicolas Beltran-Velez; Quentin Chu; Sweta Karlekar; Jannik Kossen; Yarin Gal; John P. Cunningham; David BleiBERTs are Generative In-Context Learners [paper link] 2024-06-07
David SamuelEnhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective [paper link] 2024-06-06
Xinhao Yao; Xiaolin Hu; Shenzhi Yang; Yong LiuWhat Do Language Models Learn in Context? The Structured Task Hypothesis [paper link] 2024-06-06
Jiaoda Li; Yifan Hou; Mrinmaya Sachan; Ryan CotterellExact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers [paper link] 2024-06-05
Brian K Chen; Tianyang Hu; Hui Jin; Hwee Kuan Lee; Kenji KawaguchiLearning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks [paper link] 2024-06-04
Tianyu He; Darshil Doshi; Aritra Das; Andrey GromovWhy Larger Language Models Do In-context Learning Differently? [paper link] 2024-05-30
Zhenmei Shi; Junyi Wei; Zhuoyan Xu; Yingyu LiangIs In-Context Learning Sufficient for Instruction Following in LLMs? [paper link] 2024-05-30
Hao Zhao; Maksym Andriushchenko; Francesco Croce; Nicolas FlammarionDoes learning the right latent variables necessarily improve in-context learning? [paper link] 2024-05-29
Sarthak Mittal; Eric Elmoznino; Leo Gagnon; Sangnie Bhardwaj; Dhanya Sridhar; Guillaume LajoieA Theory of In-Context Learning in Transformers [paper link] 2024-05-29
Yifei Wang; Yuyang Wu; Zeming Wei; Stefanie Jegelka; Yisen WangOn Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability [paper link] 2024-05-27
Chenyu Zheng; Wei Huang; Rongzhen Wang; Guoqiang Wu; Jun Zhu; Chongxuan LiTransformer In-Context Learning for Categorical Data [paper link] 2024-05-27
Aaron T. Wang; Ricardo Henao; Lawrence CarinAutomatic Domain Adaptation by Transformers in In-Context Learning [paper link] 2024-05-27
Ryuichiro Hataya; Kota Matsui; Masaaki ImaizumiUnifying Demonstration Selection and Compression for In-Context Learning [paper link] 2024-05-27
Jun GaoOn the Noise Robustness of In-Context Learning for Text Generation [paper link] 2024-05-27
Hongfu Gao; Feipeng Zhang; Wenyu Jiang; Jun Shu; Feng Zheng; Hongxin WeiMLPs Learn In-Context [paper link] 2024-05-24
William L. Tong; Cengiz PehlevanTowards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification [paper link] 2024-05-24
Shang Liu; Zhongze Cai; Guanting Chen; Xiaocheng LiCan Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? [paper link] 2024-05-02
Khashayar Gatmiry; Nikunj Saunshi; Sashank J. Reddi; Stefanie Jegelka; Sanjiv KumarIn-context Learning on Function Classes Unveiled for Transformers [paper link] 2024-05-02
Zhijie Wang; Bo Jiang; Shuai LiIn-Context Learning with Long-Context Models: An In-Depth Exploration [paper link] 2024-04-30
Amanda Bertsch; Maor Ivgi; Uri Alon; Jonathan Berant; Matthew R. Gormley; Graham NeubigWhat needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation [paper link] 2024-04-10
Aaditya K. Singh; Ted Moskovitz; Felix Hill; Stephanie C. Y. Chan; Andrew M. SaxeIs attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability [paper link] 2024-04-01
Ivan Lee; Nan Jiang; Taylor Berg-KirkpatrickTraining Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [paper link] 2024-02-29
Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran YangHow Transformers Learn Causal Structure with Gradient Descent [paper link] 2024-02-22
Eshaan Nichani; Alex Damian; Jason D. LeeIn-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization [paper link] 2024-02-22
Ruiqi Zhang; Jingfeng Wu; Peter L. BartlettIdentifying Semantic Induction Heads to Understand In-Context Learning [paper link] 2024-02-20
Jie Ren; Qipeng Guo; Hang Yan; Dongrui Liu; Xipeng Qiu; Dahua LinHow do Transformers perform In-Context Autoregressive Learning? [paper link] 2024-02-08
Michael E. Sander; Raja Giryes; Taiji Suzuki; Mathieu Blondel; Gabriel PeyréCan Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks [paper link] 2024-02-06
Jongho Park; Jaeseung Park; Zheyang Xiong; Nayoung Lee; Jaewoong Cho; Samet Oymak; Kangwook Lee; Dimitris PapailiopoulosAn Information-Theoretic Analysis of In-Context Learning [paper link] 2024-01-28
Hong Jun Jeon; Jason D. Lee; Qi Lei; Benjamin Van RoyThe Transient Nature of Emergent In-Context Learning in Transformers [paper link] 2023-12-11
Aaditya K. Singh; Stephanie C. Y. Chan; Ted Moskovitz; Erin Grant; Andrew M. Saxe; Felix HillIn-Context Learning Functions with Varying Number of Minima [paper link] 2023-11-21
David Oniani; Yanshan WangExploring the Relationship between In-Context Learning and Instruction Tuning [paper link] 2023-11-17
Hanyu Duan; Yixuan Tang; Yi Yang; Ahmed Abbasi; Kar Yan TamWhen does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks [paper link] 2023-11-15
Hao Peng; Xiaozhi Wang; Jianhui Chen; Weikai Li; Yunjia Qi; Zimu Wang; Zhili Wu; Kaisheng Zeng; Bin Xu; Lei Hou; Juanzi LiIn-context Learning Generalizes, But Not Always Robustly: The Case of Syntax [paper link] 2023-11-13
Aaron Mueller; Albert Webson; Jackson Petty; Tal LinzenTransformers learn to implement preconditioned gradient descent for in-context learning [paper link] 2023-11-09
Kwangjun Ahn; Xiang Cheng; Hadi Daneshmand; Suvrit SraTransformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models [paper link] 2023-10-26
Deqing Fu; Tian-Qi Chen; Robin Jia; Vatsal SharanIn-Context Learning Creates Task Vectors [paper link] 2023-10-24
Roee Hendel; Mor Geva; Amir GlobersonFunction Vectors in Large Language Models [paper link] 2023-10-23
Eric Todd; Millicent L. Li; Arnab Sen Sharma; Aaron Mueller; Byron C. Wallace; David BauIn-context Learning with Transformer Is Really Equivalent to a Contrastive Learning Pattern [paper link] 2023-10-19
Ruifeng Ren; Yong LiuTrained Transformers Learn Linear Models In-Context [paper link] 2023-10-19
Ruiqi Zhang; Spencer Frei; Peter L. BartlettHow Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations [paper link] 2023-10-16
Tianyu Guo; Wei Hu; Song Mei; Huan Wang; Caiming Xiong; Silvio Savarese; Yu BaiUnderstanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions [paper link] 2023-10-13
Satwik Bhattamishra; Arkil Patel; Phil Blunsom; Varun KanadeHow Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [paper link] 2023-10-13
Jingfeng Wu; Difan Zou; Zixiang Chen; Vladimir Braverman; Quanquan Gu; Peter BartlettIn-Context Learning Learns Label Relationships but Is Not Conventional Learning [paper link] 2023-10-13
Jannik Kossen; Yarin Gal; Tom RainforthIn-context Convergence of Transformers [paper link] 2023-10-13
Yu Huang; Yuan Cheng; Yingbin LiangIn-Context Learning through the Bayesian Prism [paper link] 2023-10-13
Madhur Panwar; Kabir Ahuja; Navin GoyalDo pretrained Transformers Really Learn In-context by Gradient Descent? [paper link] 2023-10-12
Lingfeng Shen; Aayush Mishra; Daniel KhashabiWhat and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization [paper link] 2023-10-10
Yufeng Zhang; Fengzhuo Zhang; Zhuoran Yang; Zhaoran WangExplaining Emergent In-Context Learning as Kernel Regression [paper link] 2023-10-05
Chi Han; Ziqi Wang; Han Zhao; Heng JiCausalLM is not optimal for in-context learning [paper link] 2023-09-02
Nan Ding; Tomer Levinboim; Jialin Wu; Sebastian Goodman; Radu SoricutOne Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention [paper link] 2023-07-07
Arvind Mahankali; Tatsunori B. Hashimoto; Tengyu MaTransformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection [paper link] 2023-07-06
Yu Bai; Fan Chen; Huan Wang; Caiming Xiong; Song MeiTransformers Learn In-Context by Gradient Descent [paper link] 2023-06-15
Johannes Von Oswald; Eyvind Niklasson; Ettore Randazzo; Joao Sacramento; Alexander Mordvintsev; Andrey Zhmoginov; Max VladymyrovThe Closeness of In-Context Learning and Weight Shifting for Softmax Regression [paper link] 2023-04-26
Shuai Li; Zhao Song; Yu Xia; Tong Yu; Tianyi ZhouA Theory of Emergent In-Context Learning as Implicit Structure Induction [paper link] 2023-03-14
Michael Hahn; Navin GoyalThe Learnability of In-Context Learning [paper link] 2023-03-14
Noam Wies; Yoav Levine; Amnon ShashuaWhat Can Transformers Learn In-Context? A Case Study of Simple Function Classes [paper link] 2023-01-14
Shivam Garg; Dimitris Tsipras; Percy Liang; Gregory ValiantTransformers generalize differently from information stored in context vs in weights [paper link] 2022-10-13
Stephanie C. Y. Chan; Ishita Dasgupta; Junkyung Kim; Dharshan Kumaran; Andrew K. Lampinen; Felix HillIn-Context Learning and Induction Heads [paper link] 2022-09-24
Catherine Olsson; Nelson Elhage; Neel Nanda; Nicholas Joseph; Nova DasSarma; Tom Henighan; Ben Mann; Amanda Askell; Yuntao Bai; Anna Chen; Tom Conerly; Dawn Drain; Deep Ganguli; Zac Hatfield-Dodds; Danny Hernandez; Scott Johnston; Andy Jones; Jackson Kernion; Liane Lovitt; Kamal Ndousse; Dario Amodei; Tom Brown; Jack Clark; Jared Kaplan; Sam McCandlish; Chris Olah
Chain-of-Thought
Papers analyzing the chain-of-thought phenomenon in large language models, exploring theoretical and empirical perspectives.
paper list (click to fold / unfold)
Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness [paper link] 2026-07-02
Aria Masoomi;Mahsa Bazzaz;Adel Javanmard;Vahab MirrokniWhat LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs [paper link] 2026-06-26
Nhi Nguyen;Shauli Ravfogel;Rajesh RanganathScheduling Thoughts: Learning the Order of Thought in Diffusion Language Models [paper link] 2026-06-22
Jiawei Xu;Minghui Liu;Aakriti Agrawal;Yifan Chen;Furong HuangProvable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently [paper link] 2026-06-22
Stanley Wei;Juno KimA Verifiable Search Is Not a Learnable Chain-of-Thought [paper link] 2026-06-20
Harsh PatelLearning through Internalization [paper link] 2026-06-18
Nikolaos Tsilivis;Nirmit Joshi;Marko Medvedev;Julia Kempe;Nati SrebroWhat Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis [paper link] 2026-06-18
Xinghao Chen;Chak Tou Leong;Wenjin Guo;Jian Wang;Wenjie Li;Xiaoyu ShenEfficiently Representing Algorithms With Chain-of-Thought Transformers [paper link] 2026-06-18
Yanhong Li;Anej Svete;Ashish Sabharwal;William MerrillFree Energy Heuristics: Fast-And-Frugal Cognition as Active Inference Under Uncertain Precision [paper link] 2026-06-14
Alex BogdanTight Sample Complexity of Transformers [paper link] 2026-06-08
Chenxiao Yang;Nathan Srebro;Zhiyuan LiRethinking Thinking Tokens: Understanding Why They Underperform in Practice [paper link] 2024-11-18
Sreeram Vennam; David Valente; David Herel; Ponnurangam KumaraguruWhat Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective [paper link] 2024-10-31
Ming Li; Yanhong Li; Tianyi ZhouA Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration [paper link] 2024-10-21
Yingqian Cui; Pengfei He; Xianfeng Tang; Qi He; Chen Luo; Jiliang Tang; Yue XingTransformers Provably Solve Parity Efficiently with Chain of Thought [paper link] 2024-10-11
Juno Kim; Taiji SuzukiFrom Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency [paper link] 2024-10-07
Kaiyue Wen; Huaqing Zhang; Hongzhou Lin; Jingzhao ZhangTraining Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis [paper link] 2024-10-03
Hongkang Li; Meng Wang; Songtao Lu; Xiaodong Cui; Pin-Yu ChenAutoregressive + Chain of Thought (CoT) ≃ Recurrent: Recurrence's Role in Language Models and a Revist of Recurrent Transformer [paper link] 2024-09-14
Xiang Zhang; Muhammad Abdul-Mageed; Laks V.S. LakshmananUnveiling the Statistical Foundations of Chain-of-Thought Prompting Methods [paper link] 2024-08-25
Xinyang Hu; Fengzhuo Zhang; Siyu Chen; Zhuoran YangDeciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning [paper link] 2024-07-01
Akshara Prabhakar; Thomas L. Griffiths; R. Thomas McCoyOn the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning [paper link] 2024-06-20
Franz Nowak; Anej Svete; Alexandra Butoi; Ryan CotterellIteration Head: A Mechanistic Study of Chain-of-Thought [paper link] 2024-06-04
Vivien Cabannes; Charles Arnal; Wassim Bouaziz; Alice Yang; Francois Charton; Julia KempeLet's Think Dot by Dot: Hidden Computation in Transformer Language Models [paper link] 2024-04-24
Jacob Pfau; William Merrill; Samuel R. BowmanChain of Thought Empowers Transformers to Solve Inherently Serial Problems [paper link] 2024-02-20
Zhiyuan Li; Hong Liu; Denny Zhou; Tengyu MaTowards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective [paper link] 2023-12-22
Guhao Feng; Bohang Zhang; Yuntian Gu; Haotian Ye; Di He; Liwei WangWhy Can Large Language Models Generate Correct Chain-of-Thoughts? [paper link] 2023-10-20
Rasul Tutunov; Antoine Grosnit; Juliusz Ziomek; Jun Wang; Haitham Bou-AmmarHow Large Language Models Implement Chain-of-Thought? [paper link] 2023-10-13
Yiqun Wang; Sile Hu; Yonggang Zhang; Xiang Tian; Xuesong Liu; Yaowu Chen; Xu Shen; Jieping YeThe Expressive Power of Transformers with Chain of Thought [paper link] 2023-10-13
William Merrill; Ashish Sabharwal
Hallucination
Papers examining the hallucination phenomenon in language models, including both theoretical and empirical analysis.
paper list (click to fold / unfold)
Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI [paper link] 2026-07-03
Bing Cheng;Yi-Shuai Niu;Howell Tong;Shing-Tung YauStatistically Meaningful Geometry and Gauge Symmetry Breaking: A Geometric Foundation for Scientific Discovery and Intelligence Emergence [paper link] 2026-07-03
Bing Cheng;Yi-Shuai Niu;Howell Tong;Shing-Tung YauDelayed Verification Destabilizes Multi-Agent LLM Belief: Instability Thresholds and Optimal Corrector Placement [paper link] 2026-06-25
Igor ItkinPre-Generation Hallucination Detection in Large Language Models via Soft-Target Attention Probing [paper link] 2026-06-20
Amina Miftakhova;Alexey ZaytsevQuickest Detection of Hallucination Onset: Delay Bounds and Learned CUSUM Statistics [paper link] 2026-06-10
Igor ItkinHow Many Counterfactuals Does It Take? Probing VLM Hallucinations Through Circuits and Causal Effects [paper link] 2026-06-07
Abhivansh Gupta;Simardeep Singh;Advika Sinha;Shreyansh Modi;Akshat TomarAnalyzing the Correlation Between Hallucinations and Knowledge Conflicts in Large Language Models [paper link] 2026-06-07
Lucrezia Laraspata;Giovanna Castellano;Gennaro VessioOn the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse [paper link] 2024-11-14
Alkis Kalavasis; Anay Mehrotra; Grigoris VelegkasNo Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models [paper link] 2024-10-24
Changlong Wu; Ananth Grama; Wojciech SzpankowskiShared Imagination: LLMs Hallucinate Alike [paper link] 2024-07-23
Yilun Zhou; Caiming Xiong; Silvio Savarese; Chien-Sheng WuEstimating the Hallucination Rate of Generative AI [paper link] 2024-06-11
Andrew Jesson; Nicolas Beltran-Velez; Quentin Chu; Sweta Karlekar; Jannik Kossen; Yarin Gal; John P. Cunningham; David BleiDoes Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? [paper link] 2024-05-09
Zorik Gekhman; Gal Yona; Roee Aharoni; Matan Eyal; Amir Feder; Roi Reichart; Jonathan HerzigMechanisms of non-factual hallucinations in language models [paper link] 2024-03-26
Lei Yu; Meng Cao; Jackie Chi Kit Cheung; Yue DongUnfamiliar Finetuning Examples Control How Language Models Hallucinate [paper link] 2024-03-08
Katie Kang; Eric Wallace; Claire Tomlin; Aviral Kumar; Sergey LevineIn-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation [paper link] 2024-03-05
Shiqi Chen; Miao Xiong; Junteng Liu; Zhengxuan Wu; Teng Xiao; Siyang Gao; Junxian HeCalibrated Language Models Must Hallucinate [paper link] 2023-11-24
Adam Tauman Kalai; Santosh S. VempalaThe Curious Case of Hallucinatory Unanswerablity: Finding Truths in the Hidden States of Over-Confident Large Language Models [paper link] 2023-10-18
Aviv Slobodkin; Omer Goldman; Avi Caciularu; Ido Dagan; Shauli Ravfogel
Reversal Curse
Papers that analyze the reversal curse phenomenon in large language models.
paper list (click to fold / unfold)
Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics [paper link] 2024-05-07
Hanlin Zhu; Baihe Huang; Shaolun Zhang; Michael Jordan; Jiantao Jiao; Yuandong Tian; Stuart RussellThe Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" [paper link] 2024-04-04
Lukas Berglund; Meg Tong; Max Kaufmann; Mikita Balesni; Asa Cooper Stickland; Tomasz Korbak; Owain EvansAn Investigation of LLMs' Inefficacy in Understanding Converse Relations [paper link] 2023-12-01
Chengwen Qi; Bowen Li; Binyuan Hui; Bailin Wang; Jinyang Li; Jinwang Wu; Yuanjun LailiPhysics of Language Models: Part 3.2, Knowledge Manipulation [paper link] 2023-09-25
Zeyuan Allen-Zhu; Yuanzhi LiThe Reversal Curse: Which Tokens You Predict Underlie the Factorization Curse and More [paper link] 2023-06-07
Ouail Kitouni; Niklas Nolte; Diane Bouchacourt; Adina Williams; Mike Rabbat; Mark Ibrahim
Scaling Laws / Emergent Abilities / Grokking / etc.
Papers exploring how model performance scales with model size, data size, or computational resources, and the emergence of unexpected abilities.
paper list (click to fold / unfold)
From Approximation to Emergence: A Theory of Deep Learning [paper link] 2026-07-01
Zhilin ZhaoRadial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization [paper link] 2026-06-30
Srijan Tiwari;Aditya Chauhan;Manjot SinghA Stochastic--Geometric Theory of Scaling Laws in Grokking [paper link] 2026-06-29
Róisín Luo;Christian Gagné;Jonas Ngnawé;Ihsan Ullah;Karyn MorrisseySmooth Scaling Laws Hide Stepwise Token Learning [paper link] 2026-06-29
Pingjie Wang;Zechen Hu;Peiru Yang;Fu Guo;Debing ZhangOn the Nonlinearity of Learning Rate Scaling for LLM Training [paper link] 2026-06-28
Zaiwen Yang;Huaqing Zhang;Jing Xu;Jingzhao ZhangInternal Data Repetition Destroys Language Models [paper link] 2026-06-23
Jessica Chudnovsky;Joshua Kazdan;Noam Levi;Rylan Schaeffer;Yegor Denisov-Blanch;Bo He;Mehmet Donmez;Sanmi Koyejo;David DonohoNoise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks [paper link] 2026-06-15
Ibrahim Talha Ersoy;Karoline WiesnerRethinking the Role of Efficient Attention in Hybrid Architectures [paper link] 2026-06-13
Ziqing Qiao;Yinuo Xu;Chaojun Xiao;Zhou Su;Zihan Zhou;Yingfa Chen;Xiaoyue Xu;Xu Han;Zhiyuan LiuExplaining Data Mixing Scaling Laws [paper link] 2026-06-06
Rui Dai;Shuran ZhengPhase Transitions in Large Language Models and the $O(N)$ Model [paper link] 2025-01-27
Youran Sun; Babak HaghighatUnderstanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data [paper link] 2024-11-11
Alex Havrilla; Wenjing LiaoScaling Laws for Precision [paper link] 2024-11-07
Tanishq Kumar; Zachary Ankner; Benjamin F. Spector; Blake Bordelon; Niklas Muennighoff; Mansheej Paul; Cengiz Pehlevan; Christopher Ré; Aditi RaghunathanUnlocking the Theory Behind Scaling 1-Bit Neural Networks [paper link] 2024-11-03
Majid Daliri; Zhao Song; Chiwun YangHow Does Critical Batch Size Scale in Pre-training? [paper link] 2024-10-29
Hanlin Zhang; Depen Morwani; Nikhil Vyas; Jingfeng Wu; Difan Zou; Udaya Ghai; Dean Foster; Sham KakadeAn Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models [paper link] 2024-10-15
Anuj K. Nayak; Lav R. VarshneyA Hitchhiker's Guide to Scaling Law Estimation [paper link] 2024-10-15
Leshem Choshen; Yang Zhang; Jacob AndreasScaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models [paper link] 2024-10-08
Siqi Wang; Zhengyu Chen; Bei Li; Keqing He; Min Zhang; Jingang WangGrokking at the Edge of Linear Separability [paper link] 2024-10-06
Alon Beck; Noam Levi; Yohai Bar-SinaiAn Empirical Study of Scaling Laws for Transfer [paper link] 2024-08-30
Matthew BarnettA Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language [paper link] 2024-08-22
Ekdeep Singh Lubana; Kyogo Kawaguchi; Robert P. Dick; Hidenori TanakaScaling Law with Learning Rate Annealing [paper link] 2024-08-20
Howe Tissue; Venus Wang; Lu WangPerformance Law of Large Language Models [paper link] 2024-08-19
Chuhan Wu; Ruiming TangInformation-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition [paper link] 2024-08-16
Kenzo Clauw; Sebastiano Stramaglia; Daniele MarinazzoLarge Language Monkeys: Scaling Inference Compute with Repeated Sampling [paper link] 2024-07-31
Bradley Brown; Jordan Juravsky; Ryan Ehrlich; Ronald Clark; Quoc V. Le; Christopher Ré; Azalia MirhoseiniEmergence in non-neural models: grokking modular arithmetic via average gradient outer product [paper link] 2024-07-29
Neil Mallinar; Daniel Beaglehole; Libin Zhu; Adityanarayanan Radhakrishnan; Parthe Pandit; Mikhail BelkinExploring Scaling Trends in LLM Robustness [paper link] 2024-07-25
Nikolaus Howe; Michał Zajac; Ian McKenzie; Oskar Hollinsworth; Tom Tseng; Pierre-Luc Bacon; Adam GleaveUnderstanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT [paper link] 2024-07-25
Muhammad Ali; Swetasudha Panda; Qinlan Shen; Michael Wick; Ari KobrenScaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies [paper link] 2024-07-18
Chaofan Tao; Qian Liu; Longxu Dou; Niklas Muennighoff; Zhongwei Wan; Ping Luo; Min Lin; Ngai WongWhy Do You Grok? A Theoretical Analysis of Grokking Modular Addition [paper link] 2024-07-17
Mohamad Amin Mohamadi; Zhiyuan Li; Lei Wu; Danica J. SutherlandPredicting Emergent Capabilities by Finetuning [paper link] 2024-07-10
Charlie Victor Snell; Eric Wallace; Dan Klein; Sergey LevineResolving Discrepancies in Compute-Optimal Scaling of Language Models [paper link] 2024-06-25
Tomer Porian; Mitchell Wortsman; Jenia Jitsev; Ludwig Schmidt; Yair CarmonScaling Laws for Linear Complexity Language Models [paper link] 2024-06-24
Xuyang Shen; Dong Li; Ruitao Leng; Zhen Qin; Weigao Sun; Yiran ZhongScaling Laws for Fact Memorization of Large Language Models [paper link] 2024-06-22
Xingyu Lu; Xiaonan Li; Qinyuan Cheng; Kai Ding; Xuanjing Huang; Xipeng QiuReconciling Kaplan and Chinchilla Scaling Laws [paper link] 2024-06-12
Tim Pearce; Jinyeop SongDeep Grokking: Would Deep Neural Networks Generalize Better? [paper link] 2024-05-29
Simin Fan; Razvan Pascanu; Martin JaggiLinguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
Robert Wu; Vardan PapyanScaling Laws and Compute-Optimal Training Beyond Fixed Training Durations [paper link] 2024-05-28
Alexander Hägele; Elie Bakouch; Atli Kosson; Loubna Ben Allal; Leandro Von Werra; Martin Jaggigzip Predicts Data-dependent Scaling Laws [paper link] 2024-05-26
Rohan PandeyEmergence of a High-Dimensional Abstraction Phase in Language Transformers [paper link] 2024-05-24
Emily Cheng; Diego Doimo; Corentin Kervadec; Iuri Macocco; Jade Yu; Alessandro Laio; Marco BaroniA rationale from frequency perspective for grokking in training neural network [paper link] 2024-05-24
Zhangchen Zhou; Yaoyu Zhang; Zhi-Qin John XuGrokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization [paper link] 2024-05-23
Boshi Wang; Xiang Yue; Yu Su; Huan SunData Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining [paper link] 2024-05-23
Ce Ge; Zhijian Ma; Daoyuan Chen; Yaliang Li; Bolin Ding4+3 Phases of Compute-Optimal Neural Scaling Laws [paper link] 2024-05-23
Elliot Paquette; Courtney Paquette; Lechao Xiao; Jeffrey PenningtonSlaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models [paper link] 2024-05-22
Raghu Mudumbai; Tyler BellQuantifying Emergence in Large Language Models [paper link] 2024-05-21
Hang Chen; Xinyu Yang; Jiaying Zhu; Wenya WangBeyond Scaling Laws: Understanding Transformer Performance with Associative Memory [paper link] 2024-05-14
Xueyan Niu; Bo Bai; Lei Deng; Wei HanMore Compute Is What You Need [paper link] 2024-04-30
Zhen GuoAn exactly solvable model for emergence and scaling laws [paper link] 2024-04-26
Yoonsoo Nam; Nayara Fonseca; Seok Hyeong Lee; Ard LouisWhy do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck [paper link] 2024-04-11
Nathan Godey; Éric de la Clergerie; Benoît SagotA Large-Scale Exploration of $\mu$-Transfer [paper link] 2024-04-08
Lucas LingleEmergent Abilities in Reduced-Scale Generative Language Models [paper link] 2024-04-02
Sherin Muckatira; Vijeta Deshpande; Vladislav Lialin; Anna RumshiskyUnderstanding Emergent Abilities of Language Models from the Loss Perspective [paper link] 2024-03-23
Zhengxiao Du; Aohan Zeng; Yuxiao Dong; Jie TangUnraveling the Mystery of Scaling Laws: Part I [paper link] 2024-03-21
Hui Su; Zhi Tian; Xiaoyu Shen; Xunliang CaiLanguage models scale reliably with over-training and on downstream tasks [paper link] 2024-03-13
Samir Yitzhak Gadre; Georgios Smyrnis; Vaishaal Shankar; Suchin Gururangan; Mitchell Wortsman; Rulin Shao; Jean Mercat; Alex Fang; Jeffrey Li; Sedrick Keh; Rui Xin; Marianna Nezhurina; Igor Vasiljevic; Jenia Jitsev; Alexandros G. Dimakis; Gabriel Ilharco; Shuran Song; Thomas Kollar; Yair Carmon; Achal Dave; Reinhard Heckel; Niklas Muennighoff; Ludwig SchmidtWhen Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method [paper link] 2024-02-26
Biao Zhang; Zhongtao Liu; Colin Cherry; Orhan FiratInterpreting Grokked Transformers in Complex Modular Arithmetic [paper link] 2024-02-26
Hiroki Furuta; Gouki Minegishi; Yusuke Iwasawa; Yutaka MatsuoA Tale of Tails: Model Collapse as a Change of Scaling Laws [paper link] 2024-02-10
Elvis Dohmatob; Yunzhen Feng; Pu Yang; Francois Charton; Julia KempeScaling Data-Constrained Language Models [paper link] 2023-10-25
Niklas Muennighoff; Alexander M. Rush; Boaz Barak; Teven Le Scao; Aleksandra Piktus; Nouamane Tazi; Sampo Pyysalo; Thomas Wolf; Colin RaffelThe Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning [paper link] 2023-10-06
Tian Jin; Nolan Clement; Xin Dong; Vaishnavh Nagarajan; Michael Carbin; Jonathan Ragan-Kelley; Gintare Karolina DziugaiteAre Emergent Abilities of Large Language Models a Mirage? [paper link] 2023-04-28
Rylan Schaeffer; Brando Miranda; Sanmi KoyejoTraining Compute-Optimal Large Language Models [paper link] 2022-03-29
Jordan Hoffmann; Sebastian Borgeaud; Arthur Mensch; Elena Buchatskaya; Trevor Cai; Eliza Rutherford; Diego de Las Casas; Lisa Anne Hendricks; Johannes Welbl; Aidan Clark; Tom Hennigan; Eric Noland; Katie Millican; George van den Driessche; Bogdan Damoc; Aurelia Guy; Simon Osindero; Karen Simonyan; Erich Elsen; Jack W. Rae; Oriol Vinyals; Laurent SifreScaling Laws for Neural Language Models [paper link] 2020-01-22
Jared Kaplan; Sam McCandlish; Tom Henighan; Tom B. Brown; Benjamin Chess; Rewon Child; Scott Gray; Alec Radford; Jeffrey Wu; Dario Amodei
Knowledge / Memory Mechanisms
Papers focusing on how large language models store, retrieve, and utilize knowledge, analyzing the memory mechanisms involved.
paper list (click to fold / unfold)
Revocable Learned State via Process Sidecars [paper link] 2026-06-29
John SweeneyParallel Manifold Steering: Efficient Adaptation of Large Associative Memories via Residual Energy Shaping [paper link] 2026-06-23
Kanishk AwadhiyaForget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets [paper link] 2026-06-22
Duc Duong;Hoang Anh Duy Le;Jianwen Xie;Anshumali Shrivastava;Zhaozhuo XuA theoretical model for task routing in mixture-of-expert transformers [paper link] 2026-06-12
Vinoth Nandakumar;Yongli Xiang;Yunzhi Yao;Peike Li;Tongliang LiuA Geometric Framework for Understanding Memorization in Generative Models [paper link] 2024-10-31
Brendan Leigh Ross; Hamidreza Kamkari; Tongzi Wu; Rasa Hosseinzadeh; Zhaoyan Liu; George Stein; Jesse C. Cresswell; Gabriel Loaiza-GanemOptimal Memorization Capacity of Transformers [paper link] 2024-09-26
Tokio Kajitsuka; Issei SatoSchrodingers Memory: Large Language Models [paper link] 2024-09-16
Wei Wang; Qing LiSelf-Attention Limits Working Memory Capacity of Transformer-Based Models [paper link] 2024-09-16
Dongyu Gong; Hantao ZhangGreat Memory, Shallow Reasoning: Limits of kNN-LMs [paper link] 2024-08-21
Shangyi Geng; Wenting Zhao; Alexander M RushMemorisation In In-Context Learning [paper link] 2024-08-21
Shahriar Golchin; Mihai Surdeanu; Steven Bethard; Eduardo Blanco; Ellen RiloffGeneralisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks [paper link] 2024-08-09
Verna Dankers; Ivan TitovUnderstanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications [paper link] 2024-07-27
Till Speicher; Mohammad Aflah Khan; Qinyuan Wu; Vedant Nanda; Soumi Das; Bishwamittra Ghosh; Krishna P. Gummadi; Evimaria TerziDemystifying Verbatim Memorization in Large Language Models [paper link] 2024-07-25
Jing Huang; Diyi Yang; Christopher PottsFrom Internal Conflict to Contextual Adaptation of Language Models [paper link] 2024-07-24
Sara Vera Marjanović; Haeun Yu; Pepa Atanasova; Maria Maistro; Christina Lioma; Isabelle AugensteinGeneralization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data [paper link] 2024-07-20
Antonis Antoniades; Xinyi Wang; Yanai Elazar; Alfonso Amayuelas; Alon Albalak; Kexun Zhang; William Yang WangPhysics of Language Models: Part 3.1, Knowledge Storage and Extraction [paper link] 2024-07-16
Zeyuan Allen-Zhu; Yuanzhi LiInduction Heads as an Essential Mechanism for Pattern Matching in In-context Learning [paper link] 2024-07-09
J. Crosbie; E. ShutovaDo LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers [paper link] 2024-06-26
Yibo Jiang; Goutham Rajendran; Pradeep Ravikumar; Bryon AragamScaling Laws for Fact Memorization of Large Language Models [paper link] 2024-06-22
Xingyu Lu; Xiaonan Li; Qinyuan Cheng; Kai Ding; Xuanjing Huang; Xipeng QiuConnecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data [paper link] 2024-06-20
Johannes Treutlein; Dami Choi; Jan Betley; Cem Anil; Samuel Marks; Roger Baker Grosse; Owain EvansUncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Large Language Models [paper link] 2024-06-20
Sunny Duan; Mikail Khona; Abhiram Iyer; Rylan Schaeffer; Ila R FieteUnderstanding Finetuning for Factual Knowledge Extraction [paper link] 2024-06-20
Gaurav Ghosal; Tatsunori Hashimoto; Aditi RaghunathanEstimating Knowledge in Large Language Models Without Generating a Single Token [paper link] 2024-06-18
Daniela Gottesman; Mor GevaHow Do Large Language Models Acquire Factual Knowledge During Pretraining? [paper link] 2024-06-17
Hoyeon Chang; Jinho Park; Seonghyeon Ye; Sohee Yang; Youngkyung Seo; Du-Seong Chang; Minjoon SeoBe like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs [paper link] 2024-06-14
Abhimanyu Hans; Yuxin Wen; Neel Jain; John Kirchenbauer; Hamid Kazemi; Prajwal Singhania; Siddharth Singh; Gowthami Somepalli; Jonas Geiping; Abhinav Bhatele; Tom GoldsteinKnowledge Circuits in Pretrained Transformers [paper link] 2024-05-28
Yunzhi Yao; Ningyu Zhang; Zekun Xi; Mengru Wang; Ziwen Xu; Shumin Deng; Huajun ChenUpper and lower memory capacity bounds of transformers for next-token prediction [paper link] 2024-05-22
Liam Madden; Curtis Fox; Christos ThrampoulidisA Multi-Perspective Analysis of Memorization in Large Language Models [paper link] 2024-05-19
Bowen Chen; Namgi Han; Yusuke MiyaoPhysics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws [paper link] 2024-04-08
Zeyuan Allen-Zhu; Yuanzhi LiMemorization Capacity of Multi-Head Attention in Transformers [paper link] 2024-03-02
Sadegh Mahdavi; Renjie Liao; Christos ThrampoulidisBirth of a Transformer: A Memory Viewpoint [paper link] 2023-11-06
Alberto Bietti; Vivien Cabannes; Diane Bouchacourt; Herve Jegou; Leon BottouPhysics of Language Models: Part 3.2, Knowledge Manipulation [paper link] 2023-09-25
Zeyuan Allen-Zhu; Yuanzhi LiCan Neural Network Memorization Be Localized? [paper link] 2023-07-18
Pratyush Maini; Michael C. Mozer; Hanie Sedghi; Zachary C. Lipton; J. Zico Kolter; Chiyuan ZhangQuantifying Memorization Across Neural Language Models [paper link] 2022-02-15
Nicholas Carlini; Daphne Ippolito; Matthew Jagielski; Katherine Lee; Florian Tramer; Chiyuan Zhang
Training Dynamics / Landscape / Optimization / Fine-tuning / etc.
Papers discussing various aspects of the training process, including optimization, fine-tuning, and the training landscape of large language models.
paper list (click to fold / unfold)
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training [paper link] 2026-07-07
Noel ThomasWhat Does a Discrete Diffusion Model Learn? [paper link] 2026-07-06
Rodrigo Casado Noguerales;Bernhard Schölkopf;Thomas Hofmann;Aran RaoufiTurning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment [paper link] 2026-07-06
Yu Li;Xiuyu Li;Mingyang Yi;Jiaxing Wang; zhangliangxu;Zhaolong Xing;Zhen ChenStatistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI [paper link] 2026-07-03
Bing Cheng;Yi-Shuai Niu;Howell Tong;Shing-Tung YauUnbiased Alignment for Large Language Models with Noisy Preferences [paper link] 2026-07-03
Jialiang Wang;Xianming Liu;Xiong Zhou;Hui Liu;Haoliang LiDemoPSD: Disagreement-Modulated Policy Self-Distillation [paper link] 2026-07-02
Yunhe Li;Hao Shi;Wenhao Liu;Mengzhe Ruan;Hanxu Hou;Zhongxiang Dai;Shuang Qiu;Linqi SongZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces [paper link] 2026-07-01
Xun Dong;Yibo Xu;Naigang Wang;Xin Li;Penghang Yin;Zi YangMeasuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment [paper link] 2026-07-01
Tejas Pradeep ShirodkarGRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity [paper link] 2026-06-30
Yong Yi Bay;Kathleen A. YearickGeometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR [paper link] 2026-06-30
Ruijia Zhang;Jiacheng Zhu;Hanqing Zhu;Laixi ShiCHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield [paper link] 2026-06-30
Dohyeon Kwon;Youngjin ParkBehavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback [paper link] 2026-06-29
Ved Sriraman;Peihan Liu;Daniel Hsu;Adam BlockRevocable Learned State via Process Sidecars [paper link] 2026-06-29
John SweeneyOptimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms [paper link] 2026-06-29
Ziwei Su;Junyu Ren;Victor VeitchMuonSSM: Orthogonalizing State Space Models for Sequence Modeling [paper link] 2026-06-29
Thai-Khanh Nguyen;Ngoc-Bich-Uyen Vo;Thieu N. Vo;Tan M. Nguyen;Cuong PhamOnline Data Selection for Instruction Tuning via Gaussian Processes [paper link] 2026-06-29
Jun Wang;Quoc Phong Nguyen;Julien Monteil;Vu NguyenAdaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models [paper link] 2026-06-28
Gagan JainOn the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse [paper link] 2026-06-28
Amritansh Mishra;Supriyo Chakraborty;Berkcan KapusuzogluDead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks [paper link] 2026-06-28
Tejas Pradeep ShirodkarOn the Nonlinearity of Learning Rate Scaling for LLM Training [paper link] 2026-06-28
Zaiwen Yang;Huaqing Zhang;Jing Xu;Jingzhao ZhangThe Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment [paper link] 2026-06-26
Tianyu Jia;Yue Fang;Hongxin Ding;Rihong Qiu;Zhibang Yang;Zhijing Wu;Xu Chu;Junfeng Zhao;Yasha WangReasoning Quality Emerges Early: Data Curation for Reasoning Models [paper link] 2026-06-25
Hongyi Henry Jin;Wenhan Yang;Meysam Ghaffari;Carlos Morato;Baharan MirzasoleimanCascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry [paper link] 2026-06-25
Huzama Ahmad;Cao Viet Hai Nam;Se-Young YunLearning with a Single Rollout via Monte Carlo Pass@k Critic [paper link] 2026-06-24
Fengdi Che;Yang Liu;Lei Yu;Meng Cao;Tong Che;Rupam Mahmood;Dale SchuurmansTraining for the Model You Return: Improving Optimization for Iterate-Averaged Language Models [paper link] 2026-06-23
Kwok Chun Au;Adam BlockInternal Data Repetition Destroys Language Models [paper link] 2026-06-23
Jessica Chudnovsky;Joshua Kazdan;Noam Levi;Rylan Schaeffer;Yegor Denisov-Blanch;Bo He;Mehmet Donmez;Sanmi Koyejo;David DonohoParallel Manifold Steering: Efficient Adaptation of Large Associative Memories via Residual Energy Shaping [paper link] 2026-06-23
Kanishk AwadhiyaCurvature-Guided Mixing for MLLM Adaptation [paper link] 2026-06-23
Jinglong Yang;Jiaxuan He;Wenjian Huang;Zhan Zhuang;Jianguo ZhangOpen Problem: Is AdamW Effective Under Heavy-Tailed Noise? [paper link] 2026-06-22
Dingzhi Yu;Hongyi Tao;Yuanyu Wan;Luo Luo;Lijun ZhangConvergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime [paper link] 2026-06-22
Yuqing WangA First-Order Mean Field Control Analysis of Transformer Layers under Cross-Entropy Training [paper link] 2026-06-22
Cheng Huan;Hongwei YuanOn the Position Bias of On-Policy Distillation [paper link] 2026-06-21
Yan Xie;Sijie Zhu;Tiansheng Wen;Bo Chen;Yifei WangWhat are Key Factors for Updates in RL for LLM Reasoning? [paper link] 2026-06-21
Peidong Wang;Demi Wang;Xufang Luo;Jiahang Xu;Xiaocui Yang;Shi Feng;Yuqing Yang;Dongsheng LiAsymptotic Signal Subspace Recovery in Softmax Attention Models [paper link] 2026-06-21
Lan V. TruongTowards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective [paper link] 2026-06-19
Tianqi Shen;Jinji Yang;Runze Shi;Jianhao Ma;Jiaye Teng;Ziye MaConservation Laws for Modern Neural Architectures [paper link] 2026-06-16
Viet-Hoang Tran;Vinh Khanh Bui;Tan Lai Ngoc;Nam Nguyen;Tuan Dam;Tan M. NguyenA Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks [paper link] 2026-06-16
Taiki MiyagawaA Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt [paper link] 2026-06-14
Tomoya WakayamaUnderstanding Diversity Collapse in RLVR via the Lens of Overtraining [paper link] 2026-06-13
Suqin Yuan;Jinkun Chen;Jiyang Zheng;Muyang Li;Lei Feng;Dadong Wang;Tao Xiang;Tongliang Liu;Bo AnFree Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success [paper link] 2026-06-12
Florian Hübler;Thomas Pethick;Suvrit SraPhase Transitions in Attention: A Bayesian Theory of Copy Head Emergence [paper link] 2026-06-10
Itay Lavie;Kirsten Fischer;Andrey Lekov;Frederic Van Maele;Zohar Ringel;Moritz HeliasRecoverable but Not Stationary:Local Linear Structures in Weights and Activations [paper link] 2026-06-09
Irina Piontkovskaia;Sergey NikolenkoA Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training [paper link] 2026-06-09
Cheng Huan;Hongfwei YuanMuon Learns More Robust and Transferable Features than Adam [paper link] 2026-06-08
Tianyu Ruan;Fengzhuo Zhang;Shuche Wang;Shihua ZhangUnderstanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin [paper link] 2026-06-08
Hanyang Li;Jianhao Ma;Ying CuiMultilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance [paper link] 2026-06-08
Mikhail Krasitskii;Alexander Gelbukh;Olga Kolesnikova;Grigori SidorovOn the Geometry of On-Policy Distillation [paper link] 2026-06-05
Zhennan Shen;Yanshu Li;Qingyu Yin;Chak Tou Leong;Zhilin Wang;Yanxu Chen;Rongduo Han;Sunbowen Lee;Yi R. FungGradient dynamics for low-rank fine-tuning beyond kernels [paper link] 2024-11-23
Arif Kerem Dayi; Sitan ChenUnraveling the Gradient Descent Dynamics of Transformers [paper link] 2024-11-12
Bingqing Song; Boran Han; Shuai Zhang; Jie Ding; Mingyi HongWhat Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [paper link] 2024-11-12
Katie Kang; Amrith Setlur; Dibya Ghosh; Jacob Steinhardt; Claire Tomlin; Sergey Levine; Aviral KumarTraining Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [paper link] 2024-11-12
Hongru Yang; Bhavya Kailkhura; Zhangyang Wang; Yingbin LiangGlobal Convergence in Training Large-Scale Transformers [paper link] 2024-10-31
Cheng Gao; Yuan Cao; Zihao Li; Yihan He; Mengdi Wang; Han Liu; Jason Matthew Klusowski; Jianqing FanWhat Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective [paper link] 2024-10-31
Ming Li; Yanhong Li; Tianyi ZhouLearning and Transferring Sparse Contextual Bigrams with Linear Transformers [paper link] 2024-10-30
Yunwei Ren; Zixuan Wang; Jason D. LeeAbrupt Learning in Transformers: A Case Study on Matrix Completion [paper link] 2024-10-29
Pulkit Gopalani; Ekdeep Singh Lubana; Wei HuLoRA vs Full Fine-tuning: An Illusion of Equivalence [paper link] 2024-10-28
Reece Shuttleworth; Jacob Andreas; Antonio Torralba; Pratyusha SharmaA distributional simplicity bias in the learning dynamics of transformers [paper link] 2024-10-25
Riccardo Rende; Federica Gerace; Alessandro Laio; Sebastian GoldtActive-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs [paper link] 2024-10-17
Tianyu Guo; Druv Pai; Yu Bai; Jiantao Jiao; Michael I. Jordan; Song MeiHow Transformers Implement Induction Heads: Approximation and Optimization Analysis [paper link] 2024-10-15
Mingze Wang; Ruoxi Yu; Weinan E; Lei WuWhat Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis [paper link] 2024-10-14
Weronika Ormaniec; Felix Dangel; Sidak Pal SinghAdaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? [paper link] 2024-10-08
Fırat Öncel; Matthias Bethge; Beyza Ermis; Mirco Ravanelli; Cem Subakan; Çağatay YıldızOn the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent [paper link] 2024-10-07
Bingrui Li; Wei Huang; Andi Han; Zhanpeng Zhou; Taiji Suzuki; Jun Zhu; Jianfei ChenUnderstanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective [paper link] 2024-10-07
Kaiyue Wen; Zhiyuan Li; Jason Wang; David Hall; Percy Liang; Tengyu MaTraining Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis [paper link] 2024-10-03
Hongkang Li; Meng Wang; Songtao Lu; Xiaodong Cui; Pin-Yu ChenTheoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization [paper link] 2024-10-03
Xinhao Yao; Hongjin Qian; Xiaolin Hu; Gengze Xu; Yong LiuTrained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context [paper link] 2024-10-02
Spencer Frei; Gal VardiTowards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective [paper link] 2024-10-02
Zeyu Gan; Yong LiuInvestigating the Impact of Model Complexity in Large Language Models [paper link] 2024-10-01
Jing Luo; Huiyuan Wang; Weiran HuangBenigh or Not-Benign Overfitting in Token Selection of Attention Mechanism [paper link] 2024-09-26
Keitaro Sakamoto; Issei SatoNon-asymptotic Convergence of Training Transformers for Next-token Prediction [paper link] 2024-09-25
Ruiquan Huang; Yingbin Liang; Jing YangOptimization Hyper-parameter Laws for Large Language Models [paper link] 2024-09-07
Xingyu Xie; Kuangyu Ding; Shuicheng Yan; Kim-Chuan Toh; Tianwen WeiThe AdEMAMix Optimizer: Better, Faster, Older [paper link] 2024-09-05
Matteo Pagliardini; Pierre Ablin; David GrangierClustering and Alignment: Understanding the Training Dynamics in Modular Addition [paper link] 2024-08-18
Tiberiu MusatGlobal Convergence in Training Large-Scale Transformers [paper link] 2024-08
Cheng Gao; Yuan Cao; Zihao Li; Yihan He; Mengdi Wang; Han Liu; Jason M. Klusowski; Jianqing FanOn the Convergence of Encoder-only Shallow Transformers [paper link] 2024-08
Yongtao Wu; Fanghui Liu; Grigorios G Chrysos; Volkan CevherParameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective [paper link] 2024-07-24
Jingren Liu; Zhong Ji; YunLong Yu; Jiale Cao; Yanwei Pang; Jungong Han; Xuelong LiLearning Dynamics of LLM Finetuning [paper link] 2024-07-15
Yi Ren; Danica J. SutherlandDeconstructing What Makes a Good Optimizer for Language Models [paper link] 2024-07-10
Rosie Zhao; Depen Morwani; David Brandfonbrener; Nikhil Vyas; Sham KakadeZero-Shot Generalization during Instruction Tuning: Insights from Similarity and Granularity [paper link] 2024-06-17
Bingxiang He; Ning Ding; Cheng Qian; Jia Deng; Ganqu Cui; Lifan Yuan; Huan-ang Gao; Huimin Chen; Zhiyuan Liu; Maosong SunUnderstanding Linear Probing then Fine-tuning Language Models from NTK Perspective [paper link] 2024-05-27
Akiyoshi Tomihari; Issei SatoInfinite Limits of Multi-head Transformer Dynamics [paper link] 2024-05-24
Blake Bordelon; Hamza Tahir Chaudhry; Cengiz PehlevanTowards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics [paper link] 2024-05-07
Hanlin Zhu; Baihe Huang; Shaolun Zhang; Michael Jordan; Jiantao Jiao; Yuandong Tian; Stuart RussellControl Theoretic Approach to Fine-Tuning and Transfer Learning [paper link] 2024-04-16
Erkan Bayram; Shenyu Liu; Mohamed-Ali Belabbas; Tamer BaşarLook at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think [paper link] 2024-04-12
Xinpeng Wang; Chengzhi Hu; Bolei Ma; Paul Röttger; Barbara PlankOn Training Data Influence of GPT Models [paper link] 2024-04-11
Qingyi Liu; Yekun Chai; Shuohuan Wang; Yu Sun; Keze Wang; Hua WuBest Practices and Lessons Learned on Synthetic Data for Language Models [paper link] 2024-04-11
Ruibo Liu; Jerry Wei; Fangyu Liu; Chenglei Si; Yanzhe Zhang; Jinmeng Rao; Steven Zheng; Daiyi Peng; Diyi Yang; Denny Zhou; Andrew M. DaiHow Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse [paper link] 2024-04-07
Mohamed El Amine Seddik; Suei-Wen Chen; Soufiane Hayou; Pierre Youssef; Merouane DebbahUnveiling the Generalization Power of Fine-Tuned Large Language Models [paper link] 2024-03-14
Haoran Yang; Yumeng Zhang; Jiaqi Xu; Hongyuan Lu; Pheng Ann Heng; Wai LamTransformers Get Stable: An End-to-End Signal Propagation Theory for Language Models [paper link] 2024-03-14
Akhil Kedia; Mohd Abbas Zaidi; Sushil Khyalia; Jungho Jung; Harshith Goka; Haejun LeeLinear Attention is (Maybe) All You Need (to Understand Transformer Optimization) [paper link] 2024-03-13
Kwangjun Ahn; Xiang Cheng; Minhak Song; Chulhee Yun; Ali Jadbabaie; Suvrit SraHallmarks of Optimization Trajectories in Neural Networks and LLMs: The Lengths, Bends, and Dead Ends [paper link] 2024-03-12
Sidak Pal Singh; Bobby He; Thomas Hofmann; Bernhard SchölkopfThe Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models [paper link] 2024-03-06
Adithya Bhaskar; Dan Friedman; Danqi ChenTraining Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [paper link] 2024-02-29
Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran YangHow Transformers Learn Causal Structure with Gradient Descent [paper link] 2024-02-22
Eshaan Nichani; Alex Damian; Jason D. LeeLoRA Training in the NTK Regime has No Spurious Local Minima [paper link] 2024-02-19
Uijeong Jang; Jason D. Lee; Ernest K. RyuOn the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [paper link] 2024-02-06
Zhanpeng Zhou; Zijun Chen; Yilan Chen; Bo Zhang; Junchi YanTransformers learn through gradual rank increase [paper link] 2023-12-10
Enric Boix-Adsera; Etai Littwin; Emmanuel Abbe; Samy Bengio; Joshua SusskindMechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott KruegerConnecting Pre-trained Language Model and Downstream Task via Properties of Representation [paper link] 2023-11-02
Chenwei Wu; Holden Lee; Rong GeScan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer [paper link] 2023-07-02
Yuandong Tian; Yiping Wang; Beidi Chen; Simon DuA Kernel-Based View of Language Model Fine-Tuning [paper link] 2023-06-15
Sadhika Malladi; Alexander Wettig; Dingli Yu; Danqi Chen; Sanjeev AroraA Stability Analysis of Fine-Tuning a Pre-Trained Model [paper link] 2023-01-24
Zihao Fu; Anthony Man-Cho So; Nigel Collier
Learning / Generalization / Reasoning / Weak to Strong Generalization
Papers analyzing the learning capabilities and generalization performance of language models, from weak to strong generalization.
paper list (click to fold / unfold)
Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees [paper link] 2026-07-05
Sijin Dong;Hiroyuki ShinnouStatistically Meaningful Geometry and Gauge Symmetry Breaking: A Geometric Foundation for Scientific Discovery and Intelligence Emergence [paper link] 2026-07-03
Bing Cheng;Yi-Shuai Niu;Howell Tong;Shing-Tung YauGrounding LLM Reasoning under Incomplete Graph Evidence [paper link] 2026-06-29
Jiaqi Li;Fanghui SongGeneralization Analysis of Transformers in Distribution Regression [paper link] 2026-06-28
Peilin Liu;Ding-Xuan ZhouWhen Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation [paper link] 2026-06-27
Varun KotteReasoning Quality Emerges Early: Data Curation for Reasoning Models [paper link] 2026-06-25
Hongyi Henry Jin;Wenhan Yang;Meysam Ghaffari;Carlos Morato;Baharan MirzasoleimanVeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Tools [paper link] 2026-06-17
Amirul Rahman;Mohammed Sabih AlsharariFrom Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning [paper link] 2026-06-16
Lingjing Kong;Xin Liu;Guangyi Chen;Martin Q. Ma;Xiangchen Song;Yuekai Sun;Mikhail Yurochkin;Taylor W. Killian;Ruslan Salakhutdinov;Kun Zhang;Eric P. Xing;Zhengzhong LiuIs Code Better Than Language for Algorithmic Reasoning [paper link] 2026-06-14
Terry Tong;Yu Feng;Surbhi Goel;Dan RothCausally Evaluating the Learnability of Formal Language Tasks [paper link] 2026-06-08
Vésteinn Snæbjarnarson;Anej Svete;Josef Valvoda;Reda Boumasmoud;Brian DuSell;Ryan CotterellHow Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures [paper link] 2026-06-04
Tanvi Thoria;Kiana Jafari;Marc R. Schlichting;Mykel J. KochenderferThe Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models [paper link] 2026-06-04
Jinyang Zhang;Hongxin Ding;Yue Fang;Weibin Liao;Muyang Ye;Junfeng Zhao;Yasha WangAn In-depth Investigation of Sparse Rate Reduction in Transformer-like Models [paper link] 2024-11-26
Yunzhe Hu; Difan Zou; Dong XuWhat Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [paper link] 2024-11-12
Katie Kang; Amrith Setlur; Dibya Ghosh; Jacob Steinhardt; Claire Tomlin; Sergey Levine; Aviral KumarGeneralization and Risk Bounds for Recurrent Neural Networks [paper link] 2024-11-05
Xuewei Cheng; Ke Huang; Shujie MaProvable Length Generalization in Sequence Prediction via Spectral Filtering [paper link] 2024-11-01
Annie Marsden; Evan Dogariu; Naman Agarwal; Xinyi Chen; Daniel Suo; Elad HazanRL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner [paper link] 2024-10-31
Fu-Chieh Chang; Yu-Ting Lee; Hui-Ying Shih; Pei-Yuan WuMixture of Parrots: Experts improve memorization more than reasoning [paper link] 2024-10-24
Samy Jelassi; Clara Mohri; David Brandfonbrener; Alex Gu; Nikhil Vyas; Nikhil Anand; David Alvarez-Melis; Yuanzhi Li; Sham M. Kakade; Eran MalachHow Numerical Precision Affects Mathematical Reasoning Capabilities of LLMs [paper link] 2024-10-17
Guhao Feng; Kai Yang; Yuntian Gu; Xinyue Ai; Shengjie Luo; Jiacheng Sun; Di He; Zhenguo Li; Liwei WangOn Rank-Dependent Generalisation Error Bounds for Transformers [paper link] 2024-10-15
Lan V. TruongBenign Overfitting in Single-Head Attention [paper link] 2024-10-10
Roey Magen; Shuning Shang; Zhiwei Xu; Spencer Frei; Wei Hu; Gal VardiDynamics of Concept Learning and Compositional Generalization [paper link] 2024-10-10
Yongyi Yang; Core Francisco Park; Ekdeep Singh Lubana; Maya Okawa; Wei Hu; Hidenori TanakaBenign Overfitting for Regression with Trained Two-Layer ReLU Networks [paper link] 2024-10-08
Junhyung Park; Patrick Bloebaum; Shiva Prasad KasiviswanathanProvable Weak-to-Strong Generalization via Benign Overfitting [paper link] 2024-10-06
David X. Wu; Anant SahaiA Formal Framework for Understanding Length Generalization in Transformers [paper link] 2024-10-03
Xinting Huang; Andy Yang; Satwik Bhattamishra; Yash Sarrof; Andreas Krebs; Hattie Zhou; Preetum Nakkiran; Michael HahnTrained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context [paper link] 2024-10-02
Spencer Frei; Gal VardiLines of Thought in Large Language Models [paper link] 2024-10-02
Raphaël Sarfati; Toni J. B. Liu; Nicolas Boullé; Christopher J. EarlsInvestigating the Impact of Model Complexity in Large Language Models [paper link] 2024-10-01
Jing Luo; Huiyuan Wang; Weiran HuangBenign or Not-Benign Overfitting in Token Selection of Attention Mechanism [paper link] 2024-09-26
Keitaro Sakamoto; Issei SatoUnderstanding Simplicity Bias towards Compositional Mappings via Learning Dynamics [paper link] 2024-09-15
Yi Ren; Danica J. SutherlandUnforgettable Generalization in Language Models [paper link] 2024-09-03
Eric Zhang; Leshem Chosen; Jacob AndreasThe Many Faces of Optimal Weak-to-Strong Learning [paper link] 2024-08-30
Mikael Møller Høgsgaard; Kasper Green Larsen; Markus Engelund MathiasenPhysics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems [paper link] 2024-08-29
Tian Ye; Zicheng Xu; Yuanzhi Li; Zeyuan Allen-ZhuOut-of-distribution generalization via composition: a lens through induction heads in Transformers [paper link] 2024-08-18
Jiajun Song; Zhuoyan Xu; Yiqiao ZhongOn the Generalization of Preference Learning with DPO [paper link] 2024-08-06
Shawn Im; Yixuan LiInductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs [paper link] 2024-07-31
Kewei Cheng; Jingfeng Yang; Haoming Jiang; Zhengyang Wang; Binxuan Huang; Ruirui Li; Shiyang Li; Zheng Li; Yifan Gao; Xian Li; Bing Yin; Yizhou SunPhysics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process [paper link] 2024-07-29
Tian Ye; Zicheng Xu; Yuanzhi Li; Zeyuan Allen-ZhuUnlocking Tokens as Data Points for Generalization Bounds on Larger Language Models [paper link] 2024-07-25
Sanae Lotfi; Yilun Kuang; Brandon Amos; Micah Goldblum; Marc Finzi; Andrew Gordon WilsonOn Initialization of Transformers with Pre-trained Embeddings [paper link] 2024-07-17
Ha Young Kim; Niranjan Balasubramanian; Byungkon KangWhen can transformers compositionally generalize in-context? [paper link] 2024-07-17
Seijin Kobayashi; Simon Schug; Yassir Akram; Florian Redhardt; Johannes von Oswald; Razvan Pascanu; Guillaume Lajoie; João SacramentoReasoning in Large Language Models: A Geometric Perspective [paper link] 2024-07-02
Romain Cosentino; Sarath ShekkizharLearning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [paper link] 2024-06-24
Hongkang Li; Meng Wang; Shuai Zhang; Sijia Liu; Pin-Yu ChenHow Truncating Weights Improves Reasoning in Language Models [paper link] 2024-06-05
Lei Chen; Joan Bruna; Alberto BiettiUnderstanding Transformer Reasoning Capabilities via Graph Algorithms [paper link] 2024-05-28
Clayton Sanford; Bahare Fatemi; Ethan Hall; Anton Tsitsulin; Mehran Kazemi; Jonathan Halcrow; Bryan Perozzi; Vahab MirrokniLinguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
Robert Wu; Vardan PapyanReality Only Happens Once: Single-Path Generalization Bounds for Transformers [paper link] 2024-05-26
Yannick Limmer; Anastasis Kratsios; Xuwei Yang; Raeid Saqur; Blanka HorvathA statistical framework for weak-to-strong generalization [paper link] 2024-05-25
Seamus Somerstep; Felipe Maia Polo; Moulinath Banerjee; Ya'acov Ritov; Mikhail Yurochkin; Yuekai SunTheoretical Analysis of Weak-to-Strong Generalization [paper link] 2024-05-25
Hunter Lang; David Sontag; Aravindan VijayaraghavanQuantifying the Gain in Weak-to-Strong Generalization [paper link] 2024-05-24
Moses Charikar; Chirag Pabbaraju; Kirankumar ShiragurTowards Understanding How Transformer Perform Multi-step Reasoning with Matching Operation [paper link] 2024-05-24
Zhiwei Wang; Yunji Wang; Zhongwang Zhang; Zhangchen Zhou; Hui Jin; Tianyang Hu; Jiacheng Sun; Zhenguo Li; Yaoyu Zhang; Zhi-Qin John XuInitialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing [paper link] 2024-05-08
Zhongwang Zhang; Pengxiao Lin; Zhiwei Wang; Yaoyu Zhang; Zhi-Qin John XuOn the Empirical Complexity of Reasoning and Planning in LLMs [paper link] 2024-04-17
Liwei Kang; Zirui Zhao; David Hsu; Wee Sun LeeWhen can transformers reason with abstract symbols? [paper link] 2024-04-16
Enric Boix-Adsera; Omid Saremi; Emmanuel Abbe; Samy Bengio; Etai Littwin; Joshua SusskindA Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task [paper link] 2024-02-19
Jannik Brinkmann; Abhay Sheshadri; Victor Levoso; Paul Swoboda; Christian BarteltProvably learning a multi-head attention layer [paper link] 2024-02-06
Sitan Chen; Yuanzhi LiMechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott KruegerThe Impact of Depth and Width on Transformer Language Model Generalization [paper link] 2023-10-30
Jackson Petty; Sjoerd van Steenkiste; Ishita Dasgupta; Fei Sha; Dan Garrette; Tal LinzenImplicit meta-learning may lead language models to trust more reliable sources [paper link] 2023-10-23
Dmitrii Krasheninnikov; Egor Krasheninnikov; Bruno Mlodozeniec; Tegan Maharaj; David KruegerOn the Optimization and Generalization of Multi-head Attention [paper link] 2023-10-19
Puneesh Deora; Rouzbeh Ghaderi; Hossein Taheri; Christos ThrampoulidisLarge Language Models Cannot Self-Correct Reasoning Yet [paper link] 2023-10-13
Jie Huang; Xinyun Chen; Swaroop Mishra; Huaixiu Steven Zheng; Adams Wei Yu; Xinying Song; Denny ZhouHow Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition [paper link] 2023-10-09
Guanting Dong; Hongyi Yuan; Keming Lu; Chengpeng Li; Mingfeng Xue; Dayiheng Liu; Wei Wang; Zheng Yuan; Chang Zhou; Jingren ZhouA Theory for Emergence of Complex Skills in Language Models [paper link] 2023-07-29
Sanjeev Arora; Anirudh GoyalOn the Power of Foundation Models [paper link] 2023-07-03
Yang YuanTask-Specific Skill Localization in Fine-tuned Language Models [paper link] 2023-06-15
Abhishek Panigrahi; Nikunj Saunshi; Haoyu Zhao; Sanjeev AroraTowards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks [paper link] 2023-02-11
Jiachun Pan; Pan Zhou; Shuicheng YanSame Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models [paper link] 2022-10-25
Hong Liu; Sang Michael Xie; Zhiyuan Li; Tengyu MaWhy Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning [paper link] 2022-04-20
Colin Wei; Sang Michael Xie; Tengyu MaA Mathematical Exploration of Why Language Models Help Solve Downstream Tasks [paper link] 2021-04-14
Nikunj Saunshi; Sadhika Malladi; Sanjeev AroraIntrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning [paper link] 2020-12-22
Armen Aghajanyan; Luke Zettlemoyer; Sonal GuptaHow fine can fine-tuning be? Learning efficient language models [paper link] 2020-06-03
Evani Radiya-Dixit; Xin Wang
Other Phenomena / Discoveries
Papers discussing other interesting phenomena or discoveries related to the behavior and properties of language models.
paper list (click to fold / unfold)
The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment [paper link] 2026-07-06
Haonan HuangSocial Networks of LLM Agents [paper link] 2026-07-04
Kaixuan Liu;Guojun Xiong;Weinan Zhang;Shengpu TangHow Much of the Routing Gap Is Real? Decomposing the Router-to-Oracle Gap into Reproducible Specialist Advantage and Single-Draw Label Noise [paper link] 2026-07-03
Teng-Ruei ChenSpectral Signatures of Large Language Models [paper link] 2026-07-03
Zhuoying Zhang;Ishan V. Prasad;Yuanzhe Hu;Zihang Liu;Hengrui Luo;Pu Ren;Yaoqing YangMeasuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment [paper link] 2026-07-01
Tejas Pradeep ShirodkarSemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models [paper link] 2026-06-30
Jian Gu;Aldeida Aleti;Chunyang Chen;Hongyu ZhangWhen Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs [paper link] 2026-06-29
Zhichao Yang;Caiqi Zhang;Ruihan Yang;Chengzu Li;Nigel Collier;Deqing YangSpectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantization [paper link] 2026-06-26
Rahid Zahid Alekberli;Hikmat KarimovWhy Do Accumulated Transformations Extrapolate? [paper link] 2026-06-23
Mahesh GodavartiDeeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding [paper link] 2026-06-20
Xuanming Zhang;Sining Zhoubian;Yuxuan Chen;Tianyi Tang;An Yang;Sean Du;Chujie Zheng;Fei Huang;Dayiheng Liu;Gao Huang;Jingren ZhouBeyond Hooking Onto the World: Referential Profiles and the Numerical Structure of LLM Grounding [paper link] 2026-06-19
Joo Yull RheeContagion Networks: Evaluator Preference Propagation in Multi-Agent LLM Systems [paper link] 2026-06-18
Zewen LiuFrom Drift to Coherence: Stabilizing Beliefs in LLMs [paper link] 2026-06-16
SongEun Kim;Seungyoo Lee;Edwin Fong;Hyungi Lee;Juho LeeService-Induced Congestion in Memory-Constrained LLM Serving [paper link] 2026-06-14
Ruicheng Ao;Jing Dong;Gan Luo;David Simchi-LeviBeyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective [paper link] 2026-06-13
Tao Jing;Ningxin Wu;Chen Kang;Dong Yu;Changliang Li;Pengyuan LiuMultilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance [paper link] 2026-06-08
Mikhail Krasitskii;Alexander Gelbukh;Olga Kolesnikova;Grigori SidorovOn the loss of context-awareness in general instruction fine-tuning [paper link] 2024-11-05
Yihan Wang; Andrew Bai; Nanyun Peng; Cho-Jui HsiehWeight decay induces low-rank attention layers [paper link] 2024-10-31
Seijin Kobayashi; Yassir Akram; Johannes Von OswaldAll or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling [paper link] 2024-10-30
Emanuele Marconato; Sébastien Lachapelle; Sebastian Weichwald; Luigi GreseleLooking Beyond The Top-1: Transformers Determine Top Tokens In Order [paper link] 2024-10-26
Daria Lioubashevski; Tomer Schlank; Gabriel Stanovsky; Ariel GoldsteinActive-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs [paper link] 2024-10-17
Tianyu Guo; Druv Pai; Yu Bai; Jiantao Jiao; Michael I. Jordan; Song MeiEmergent properties with repeated examples [paper link] 2024-10-09
François Charton; Julia KempeMasked Mixers for Language Generation and Retrieval [paper link] 2024-09-02
Benjamin L. BadgerMonotonic Representation of Numeric Properties in Language Models [paper link] 2024-08-15
Benjamin Heinzerling; Kentaro InuiDoes Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models [paper link] 2024-08-12
Hila Gonen; Terra Blevins; Alisa Liu; Luke Zettlemoyer; Noah A. SmithLarge Language Monkeys: Scaling Inference Compute with Repeated Sampling [paper link] 2024-07-31
Bradley Brown; Jordan Juravsky; Ryan Ehrlich; Ronald Clark; Quoc V. Le; Christopher Ré; Azalia MirhoseiniTransformers on Markov Data: Constant Depth Suffices [paper link] 2024-07-25
Nived Rajaraman; Marco Bondaschi; Kannan Ramchandran; Michael Gastpar; Ashok Vardhan MakkuvaOn the Benefits of Rank in Attention Layers [paper link] 2024-07-23
Noah Amsel; Gilad Yehudai; Joan BrunaTransformer Alignment in Large Language Models [paper link] 2024-07-10
Murdock Aubry; Haoming Meng; Anton Sugolov; Vardan PapyanUnderstanding Transformers via N-gram Statistics [paper link] 2024-06-30
Timothy NguyenLarge Vocabulary Size Improves Large Language Models [paper link] 2024-06-24
Sho Takase; Ryokan Ri; Shun Kiyono; Takuya KatoConnecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data [paper link] 2024-06-20
Johannes Treutlein; Dami Choi; Jan Betley; Cem Anil; Samuel Marks; Roger Baker Grosse; Owain EvansDistributional reasoning in LLMs: Parallel reasoning processes in multi-hop reasoning [paper link] 2024-06-19
Yuval Shalev; Amir Feder; Ariel GoldsteinTranscendence: Generative Models Can Outperform The Experts That Train Them [paper link] 2024-06-17
Edwin Zhang; Vincent Zhu; Naomi Saphra; Anat Kleiman; Benjamin L. Edelman; Milind Tambe; Sham M. Kakade; Eran MalachTaking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens [paper link] 2024-06-16
Weiyao Luo; Suncong Zheng; Heming Xia; Weikang Wang; Yan Lei; Tianyu Liu; Shuang Chen; Zhifang SuiAnisotropy is Not Inherent to Transformers [paper link] 2024-06
Anemily Machina; Robert MercerLinguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
Robert Wu; Vardan PapyanExploring Activation Patterns of Parameters in Language Models [paper link] 2024-05-28
Yudong Wang; Damai Dai; Zhifang SuiImplicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs [paper link] 2024-05-26
Mustafa Shukor; Matthieu CordYour Transformer is Secretly Linear [paper link] 2024-05-19
Anton Razzhigaev; Matvey Mikhalchuk; Elizaveta Goncharova; Nikolai Gerasimenko; Ivan Oseledets; Denis Dimitrov; Andrey KuznetsovThe Platonic Representation Hypothesis [paper link] 2024-05-13
Minyoung Huh; Brian Cheung; Tongzhou Wang; Phillip IsolaBy Tying Embeddings You Are Assuming the Distributional Hypothesis [paper link] 2024-05-02
Francesco Bertolotti; Walter CazzolaEmergent Representations of Program Semantics in Language Models Trained on Programs [paper link] 2024-05-02
Charles Jin; Martin RinardAlgorithmic progress in language models [paper link] 2024-03-09
Anson Ho; Tamay Besiroglu; Ege Erdil; David Owen; Robi Rahman; Zifan Carl Guo; David Atkinson; Neil Thompson; Jaime SevillaMassive Activations in Large Language Models [paper link] 2024-02-27
Mingjie Sun; Xinlei Chen; J. Zico Kolter; Zhuang LiuOn the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [paper link] 2024-02-06
Zhanpeng Zhou; Zijun Chen; Yilan Chen; Bo Zhang; Junchi YanAnisotropy Is Inherent to Self-Attention in Transformers [paper link] 2024-01-24
Nathan Godey; Éric de la Clergerie; Benoît SagotThe Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers [paper link] 2023-02-01
Zonglin Li; Chong You; Srinadh Bhojanapalli; Daliang Li; Ankit Singh Rawat; Sashank J. Reddi; Ke Ye; Felix Chern; Felix Yu; Ruiqi Guo; Sanjiv Kumar
Representational Capacity
Categories focused on the representational capacities and limitations of transformers and language models.
What Can Transformer Do? / Properties of Transformer
Papers providing positive results into the capabilities and properties of transformer-based models, e.g., expressiveness and learning abilities.
paper list (click to fold / unfold)
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch [paper link] 2026-07-04
Mary Letey;Yue M. Lu;Cengiz Pehlevan;Jacob Zavatone-VethThe risk of KV cache compression [paper link] 2026-07-01
Lukas Haverbeck;Carmen Amo Alonso;Andres Felipe Posada-Moreno;Sebastian Trimpe;Marco PavoneTransformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation [paper link] 2026-06-30
Jiachun Li;David Simchi-LeviTransformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework [paper link] 2026-06-29
Haobo YangWhen Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding [paper link] 2026-06-29
Aaryam SharmaExploring the Cryptographic Limits of Transformer Networks [paper link] 2026-06-28
Stefan Domunco;Andis Draguns;Philip Torr;Isaac Robinson;Christian Schroeder de WittGeneralization Analysis of Transformers in Distribution Regression [paper link] 2026-06-28
Peilin Liu;Ding-Xuan ZhouOn the Expressive Power of Weight Quantization in Large Language Models [paper link] 2026-06-20
Shao-Qun ZhangGeometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts [paper link] 2026-06-17
Tho Tran Huu;Huu-Tuan Nguyen;Thien-Hai Nguyen;Nhat-Tri Ho;Viet-Hoang Tran;Tho Quan;Tan Minh NguyenFunctional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity [paper link] 2026-06-16
Viet-Hoang Tran;Vinh Khanh Bui;Van-Hoan Trinh;Tan Lai Ngoc;Tan M. NguyenAn expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars [paper link] 2026-06-16
Vinoth Nandakumar;Qiang Qu;Pramod Thebe;Sakshi Khachariya;Tongliang LiuAdaptive inference and function vectors in deep transformers [paper link] 2026-06-15
Ravin Raj;Gautam ReddyHow Linear Is a Transformer Feed-Forward Block? Per-Block Linear Recoverability Is Learned, Not Architectural [paper link] 2026-06-12
Stuart WhippTowards Tight Bounds for Streaming Attention [paper link] 2026-06-05
Justin Y. Chen;Ying Feng;Piotr Indyk;Michael Kapralov;Ekaterina Kochetkova;Boris ProkhorovFundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency [paper link] 2024-11-25
Jerry Yao-Chieh Hu; Wei-Po Wang; Ammar Gilani; Chenyang Li; Zhao Song; Han LiuMechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers [paper link] 2024-11-18
Tiberiu MusatMeasure-to-measure interpolation using Transformers [paper link] 2024-11-07
Borjan Geshkovski; Philippe Rigollet; Domènec Ruiz-BaletAsk, and it shall be given: Turing completeness of prompting [paper link] 2024-11-04
Ruizhong Qiu; Zhe Xu; Wenxuan Bao; Hanghang TongProvable Optimal Transport with Transformers: The Essence of Depth and Prompt Engineering [paper link] 2024-10-25
Hadi DaneshmandOn the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery [paper link] 2024-10-17
Renpu Liu; Ruida Zhou; Cong Shen; Jing YangTheoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding [paper link] 2024-10-16
Daichi Hayakawa; Issei SatoMemory-augmented Transformers can implement Linear First-Order Optimization Methods [paper link] 2024-10-08
Sanchayan Dutta; Suvrit SraTransformers are Efficient Compilers, Provably [paper link] 2024-10-07
Xiyu Zhai; Runlong Zhou; Liao Zhang; Simon Shaolei DuFundamental Limitations on Subquadratic Alternatives to Transformers [paper link] 2024-10-05
Josh Alman; Hantao YuAutoregressive Large Language Models are Computationally Universal [paper link] 2024-10-04
Dale Schuurmans; Hanjun Dai; Francesco ZaniniCan Transformers Learn n-gram Language Models? [paper link] 2024-10-03
Anej Svete; Nadav Borenstein; Mike Zhou; Isabelle Augenstein; Ryan CotterellTowards Understanding the Universality of Transformers for Next-Token Prediction [paper link] 2024-10-03
Michael E. Sander; Gabriel PeyréLarge Language Models as Markov Chains [paper link] 2024-10-03
Oussama Zekri; Ambroise Odonnat; Abdelhakim Benechehab; Linus Bleistein; Nicolas Boullé; Ievgen RedkoOn Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding [paper link] 2024-10-02
Kevin Xu; Issei SatoAttention layers provably solve single-location regression [paper link] 2024-10-02
Pierre Marion; Raphaël Berthier; Gérard Biau; Claire BoyerTransformers in Uniform TC0 [paper link] 2024-09-20
David ChiangHow Transformers Learn Structured Data: Insights from Hierarchical Filtering [paper link] 2024-08-27
Jerome Garnier-Brun; Marc Mézard; Emanuele Moscato; Luca SagliettiImplicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations [paper link] 2024-08-27
Yize Zhao; Tina Behnia; Vala Vakilian; Christos ThrampoulidisA Law of Next-Token Prediction in Large Language Models [paper link] 2024-08-24
Hangfeng He; Weijie J. SuTransformers As Approximations of Solomonoff Induction [paper link] 2024-08-22
Nathan Young; Michael WitbrockLearning Randomized Algorithms with Transformers [paper link] 2024-08-20
Johannes von Oswald; Seijin Kobayashi; Yassir Akram; Angelika StegerAttention is a smoothed cubic spline [paper link] 2024-08-19
Zehua Lai; Lek-Heng Lim; Yucong LiuWhy Transformers are Obviously Good Models of Language [paper link] 2024-08-07
Felix HillCan LLMs predict the convergence of Stochastic Gradient Descent? [paper link] 2024-08-03
Oussama Zekri; Abdelhakim Benechehab; Ievgen RedkoTransformers on Markov Data: Constant Depth Suffices [paper link] 2024-07-25
Nived Rajaraman; Marco Bondaschi; Kannan Ramchandran; Michael Gastpar; Ashok Vardhan MakkuvaDo Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability [paper link] 2024-07-22
Zhuoyan Xu; Zhenmei Shi; Yingyu LiangUniversal Approximation Theory: The basic theory for large language models [paper link] 2024-07-01
Wei Wang; Qing LiSeperations in the Representational Capabilities of Transformers and Recurrent Architectures [paper link] 2024-06-13
Satwik Bhattamishra; Michael Hahn; Phil Blunsom; Varun KanadeTransformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot [paper link] 2024-06-11
Zixuan Wang; Stanley Wei; Daniel Hsu; Jason D. LeeWhat Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages [paper link] 2024-06-07
Nadav Borenstein; Anej Svete; Robin Chan; Josef Valvoda; Franz Nowak; Isabelle Augenstein; Eleanor Chodroff; Ryan CotterellPhysics of Language Models: Part 1, Learning Hierarchical Language Structures [paper link] 2024-06-02
Zeyuan Allen-Zhu; Yuanzhi LiTransformers Can Do Arithmetic with the Right Embeddings [paper link] 2024-05-27
Sean McLeish; Arpit Bansal; Alex Stein; Neel Jain; John Kirchenbauer; Brian R. Bartoldson; Bhavya Kailkhura; Abhinav Bhatele; Jonas Geiping; Avi Schwarzschild; Tom GoldsteinA One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness [paper link] 2024-05-27
Yuhao Zhang; Aws Albarghouthi; Loris D'AntoniThe Power of Hard Attention Transformers on Data Sequences: A Formal Language Theoretic Perspective [paper link] 2024-05-25
Pascal Bergsträßer; Chris Köcher; Anthony Widjaja Lin; Georg ZetzscheTransformers represent belief state geometry in their residual stream [paper link] 2024-05-24
Adam S. Shai; Sarah E. Marzen; Lucas Teixeira; Alexander Gietelink Oldenziel; Paul M. RiechersALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Models [paper link] 2024-05-15
Siwei Wang; Yifei Shen; Shi Feng; Haoran Sun; Shang-Hua Teng; Wei ChenWhat Formal Languages Can Transformers Express? A Survey [paper link] 2024-05-06
Lena Strobl; William Merrill; Gail Weiss; David Chiang; Dana AngluinTransformers Can Represent $n$-gram Language Models [paper link] 2024-04-23
Anej Svete; Ryan CotterellMechanics of Next Token Prediction with Self-Attention [paper link] 2024-04-18
Yingcong Li; Yixiao Huang; Muhammed E. Ildiz; Ankit Singh Rawat; Samet OymakWhen can transformers reason with abstract symbols? [paper link] 2024-04-16
Enric Boix-Adsera; Omid Saremi; Emmanuel Abbe; Samy Bengio; Etai Littwin; Joshua SusskindThe Illusion of State in State-Space Models [paper link] 2024-04-12
William Merrill; Jackson Petty; Ashish SabharwalLanguage Generation in the Limit [paper link] 2024-04-10
Jon Kleinberg; Sendhil MullainathanAttention is Naturally Sparse with Gaussian Distributed Input [paper link] 2024-04-03
Yichuan Deng; Zhao Song; Chiwun YangWhat Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks [paper link] 2024-04-01
Xingwu Chen; Difan ZouThe Topos of Transformer Networks [paper link] 2024-03-27
Mattia Jacopo Villani; Peter McBurneySimulating Weighted Automata over Sequences and Trees with Transformers [paper link] 2024-03-12
Michael Rizvi; Maude Lizaire; Clara Lacroce; Guillaume RabusseauSimplicity Bias of Transformers to Learn Low Sensitivity Functions [paper link] 2024-03-11
Bhavya Vasudeva; Deqing Fu; Tianyi Zhou; Elliott Kau; Youqi Huang; Vatsal SharanOn the Origins of Linear Representations in Large Language Models [paper link] 2024-03-06
Yibo Jiang; Goutham Rajendran; Pradeep Ravikumar; Bryon Aragam; Victor VeitchHow Well Can Transformers Emulate In-context Newton's Method? [paper link] 2024-03-05
Angeliki Giannou; Liu Yang; Tianhao Wang; Dimitris Papailiopoulos; Jason D. LeeRNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval [paper link] 2024-02-29
Kaiyue Wen; Xingyu Dang; Kaifeng LyuImplicit Bias of Next-Token Prediction [paper link] 2024-02-28
Christos ThrampoulidisOn the Expressive Power of a Variant of the Looped Transformer [paper link] 2024-02-21
Yihang Gao; Chuanyang Zheng; Enze Xie; Han Shi; Tianyang Hu; Yu Li; Michael K. Ng; Zhenguo Li; Zhaoqiang LiuFrom Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers [paper link] 2024-02-20
M. Emrullah Ildiz; Yixiao Huang; Yingcong Li; Ankit Singh Rawat; Samet OymakTransformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context [paper link] 2024-02-15
Xiang Cheng; Yuxin Chen; Suvrit SraCompositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks [paper link] 2024-02-05
Rahul Ramesh; Ekdeep Singh Lubana; Mikail Khona; Robert P. Dick; Hidenori TanakaAre Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? [paper link] 2024-01-29
Tokio Kajitsuka; Issei SatoTransformers are Multi-State RNNs [paper link] 2024-01-11
Matanel Oren; Michael Hassid; Yossi Adi; Roy SchwartzHow Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks [paper link] 2023-12-12
Rahul Ramesh; Mikail Khona; Robert P. Dick; Hidenori Tanaka; Ekdeep Singh LubanaTransformers can optimally learn regression mixture models [paper link] 2023-11-14
Reese Pathak; Rajat Sen; Weihao Kong; Abhimanyu DasThe Expressive Power of Low-Rank Adaptation [paper link] 2023-10-26
Yuchen Zeng; Kangwook LeeWhat Algorithms can Transformers Learn? A Study in Length Generalization [paper link] 2023-10-24
Hattie Zhou; Arwen Bradley; Etai Littwin; Noam Razin; Omid Saremi; Josh Susskind; Samy Bengio; Preetum NakkiranTransformers as Support Vector Machines [paper link] 2023-09-07
Davoud Ataee Tarzanagh; Yingcong Li; Christos Thrampoulidis; Samet OymakHow Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [paper link] 2023-06-15
Yuchen Li; Yuanzhi Li; Andrej RisteskiTighter Bounds on the Expressivity of Transformer Encoders [paper link] 2023-06-15
David Chiang; Peter Cholak; Anand PillayFast Attention Requires Bounded Entries [paper link] 2023-02-26
Josh Alman; Zhao SongTransformers Learn Shortcuts to Automata [paper link] 2023-02-01
Bingbin Liu; Jordan T. Ash; Surbhi Goel; Akshay Krishnamurthy; Cyril ZhangTransformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems [paper link] 2022-11-17
Dan Navon; Alex M. BronsteinSmall Transformers Compute Universal Metric Embeddings [paper link] 2022-10-18
Anastasis Kratsios; Valentin Debarnot; Ivan DokmanićThe Lipschitz Constant of Self-Attention [paper link] 2021-06-09
Hyunjik Kim; George Papamakarios; Andriy MnihOn Identifiability in Transformers [paper link] 2020-02-07
Gino Brunner; Yang Liu; Damián Pascual; Oliver Richter; Massimiliano Ciaramita; Roger Wattenhofer
What Can Transformer Not Do? / Limitation of Transformer
Papers investigating the limitations of transformer-based models, including expressiveness and learning constraints, e.g., limitations in reasoning.
paper list (click to fold / unfold)
Why Do Few-Step Text Latents Fail When Image Latents Work? Non-Commitment at Sharp Categorical Readouts [paper link] 2026-06-29
Zhongyao WangGrounding LLM Reasoning under Incomplete Graph Evidence [paper link] 2026-06-29
Jiaqi Li;Fanghui SongExploring the Cryptographic Limits of Transformer Networks [paper link] 2026-06-28
Stefan Domunco;Andis Draguns;Philip Torr;Isaac Robinson;Christian Schroeder de WittOn the Inseparability of Instructions and Data in Shared-Embedding Sequence Models [paper link] 2026-06-25
Dewank Pant;Shruti Lohani;Avijit KumarCascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT [paper link] 2026-06-25
Jinghan Wang;Yanjun Chen;Wei Zhang;Xiaotong Huang;Tianchen Liu;Gaoliang PengWhy Do Accumulated Transformations Extrapolate? [paper link] 2026-06-23
Mahesh GodavartiOn the Limits of Prompt-Conditioned Language Models as General-Purpose Learners [paper link] 2026-06-22
David Mguni;Julian Ma;Jun WangUnderstanding the Parameter Space Geometry of Transformers Encoding Boolean Functions [paper link] 2026-06-07
Blanka Köver;Alexandra Butoi;Anej Svete;Michael Hahn;Ryan CotterellTowards Tight Bounds for Streaming Attention [paper link] 2026-06-05
Justin Y. Chen;Ying Feng;Piotr Indyk;Michael Kapralov;Ekaterina Kochetkova;Boris ProkhorovCircuit Complexity Bounds for RoPE-based Transformer Architecture [paper link] 2024-11-12
Bo Chen; Xiaoyu Li; Yingyu Liang; Jiangxuan Long; Zhenmei Shi; Zhao SongConsistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness [paper link] 2024-11
Georgi Shopov; Stefan GerdjikovHow Numerical Precision Affects Mathematical Reasoning Capabilities of LLMs [paper link] 2024-10-17
Guhao Feng; Kai Yang; Yuntian Gu; Xinyue Ai; Shengjie Luo; Jiacheng Sun; Di He; Zhenguo Li; Liwei WangSelf-Attention Limits Working Memory Capacity of Transformer-Based Models [paper link] 2024-09-16
Dongyu Gong; Hantao ZhangOne-layer transformers fail to solve the induction heads task [paper link] 2024-08-26
Clayton Sanford; Daniel Hsu; Matus TelgarskyYour Context Is Not an Array: Unveiling Random Access Limitations in Transformers [paper link] 2024-08-10
MohammadReza Ebrahimi; Sunny Panchal; Roland MemisevicWhen Can Transformers Count to n? [paper link] 2024-07-21
Gilad Yehudai; Haim Kaplan; Asma Ghandeharioun; Mor Geva; Amir GlobersonWhen can transformers compositionally generalize in-context? [paper link] 2024-07-17
Seijin Kobayashi; Simon Schug; Yassir Akram; Florian Redhardt; Johannes von Oswald; Razvan Pascanu; Guillaume Lajoie; João SacramentoHopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries [paper link] 2024-06-18
Eden Biran; Daniela Gottesman; Sohee YangHow Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad [paper link] 2024-06-10
Emmanuel Abbe; Samy Bengio; Aryo Lotfi; Colin Sandon; Omid SaremiTransformers Need Glasses! Information Over-squashing in Language Tasks [paper link] 2024-06-06
Federico Barbero; Andrea Banino; Steven Kapturowski; Dharshan Kumaran; João G.M. Araújo; Alex Vitvitskyi; Razvan Pascanu; Petar VeličkovićOn Limitation of Transformer for Learning HMMs [paper link] 2024-06-06
Jiachen Hu; Qinghua Liu; Chi JinLanguage Models Need Inductive Biases to Count Inductively [paper link] 2024-05-30
Yingshan Chang; Yonatan BiskLimits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory [paper link] 2024-05-26
Nikola Zubić; Federico Soldá; Aurelio Sulser; Davide ScaramuzzaAttention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers [paper link] 2024-05-22
Tobias Leemann; Alina Fastowski; Felix Pfeiffer; Gjergji KasneciCollapse of Self-trained Language Models [paper link] 2024-04-02
David Herel; Tomas MikolovThe pitfalls of next-token prediction [paper link] 2024-03-11
Gregor Bachmann; Vaishnavh NagarajanWhy are Sensitive Functions Hard for Transformers? [paper link] 2024-03-03
Michael Hahn; Mark RofinTransformers are Expressive, But Are They Expressive Enough for Regression? [paper link] 2024-02-23
Swaroop Nath; Harshad Khadilkar; Pushpak BhattacharyyaLimits of Transformer Language Models on Learning Algorithmic Compositions [paper link] 2024-02-13
Jonathan Thomm; Aleksandar Terzic; Geethan Karunaratne; Giacomo Camposampiero; Bernhard Schölkopf; Abbas RahimiRepresentational Strengths and Limitations of Transformers [paper link] 2023-11-16
Clayton Sanford; Daniel Hsu; Matus TelgarskyLarge Language Models Cannot Self-Correct Reasoning Yet [paper link] 2023-10-13
Jie Huang; Xinyun Chen; Swaroop Mishra; Huaixiu Steven Zheng; Adams Wei Yu; Xinying Song; Denny ZhouAttention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth [paper link] 2023-08-01
Yihe Dong; Jean-Baptiste Cordonnier; Andreas LoukasLimits for Learning with Language Models [paper link] 2023-06-21
Nicholas Asher; Swarnadeep Bhar; Akshay Chaturvedi; Julie Hunter; Soumya PaulYour Transformer May Not be as Powerful as You Expect [paper link] 2022-10-31
Shengjie Luo; Shanda Li; Shuxin Zheng; Tie-Yan Liu; Liwei Wang; Di HeThe Devil in Linear Transformer [paper link] 2022-10-19
Zhen Qin; XiaoDong Han; Weixuan Sun; Dongxu Li; Lingpeng Kong; Nick Barnes; Yiran ZhongOn the Ability and Limitations of Transformers to Recognize Formal Languages [paper link] 2020-09-23
Satwik Bhattamishra; Kabir Ahuja; Navin Goyal
Architectural Effectivity
Categories analyzing different architectural components and their effects in transformer models.
Layer-normalization
Papers discussing the role, effects, and optimization of layer normalization in transformer models.
paper list (click to fold / unfold)
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale [paper link] 2026-06-17
Tejas Pradeep Shirodkar;P. J. NarayananRe-Introducing LayerNorm: Geometric Meaning, Irreversibility and a Comparative Study with RMSNorm [paper link] 2024-09-19
Akshat Gupta; Atahan Ozdemir; Gopala AnumanchipalliOn the Role of Attention Masks and LayerNorm in Transformers [paper link] 2024-05-29
Xinyi Wu; Amir Ajorlou; Yifei Wang; Stefanie Jegelka; Ali JadbabaieThe Expressive Power of Tuning Only the Normalization Layers [paper link] 2023-07-12
Angeliki Giannou; Shashank Rajput; Dimitris PapailiopoulosResiDual: Transformer with Dual Residual Connections [paper link] 2023-04-28
Shufang Xie; Huishuai Zhang; Junliang Guo; Xu Tan; Jiang Bian; Hany Hassan Awadalla; Arul Menezes; Tao Qin; Rui YanDeepNet: Scaling Transformers to 1,000 Layers [paper link] 2022-03-01
Hongyu Wang; Shuming Ma; Li Dong; Shaohan Huang; Dongdong Zhang; Furu WeiOn Layer Normalization in the Transformer Architecture [paper link] 2020-06-29
Ruibin Xiong; Yunchang Yang; Di He; Kai Zheng; Shuxin Zheng; Chen Xing; Huishuai Zhang; Yanyan Lan; Liwei Wang; Tie-Yan Liu
Tokenization / Embedding
Papers focused on tokenization, embedding strategies, and input representations in language models.
paper list (click to fold / unfold)
Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms [paper link] 2026-06-29
Ziwei Su;Junyu Ren;Victor VeitchGeometry of Semantic Space: Comparative Study of Discrete and Continuous Models [paper link] 2026-06-05
Gabriel Bounias;Sabine PlouxTheoretical Analysis of Byte-Pair Encoding [paper link] 2024-11-13
László Kozma; Johannes VoderholzerCounting Ability of Large Language Models and Impact of Tokenization [paper link] 2024-10-25
Xiang Zhang; Juntai Cao; Chenyu YouTokenization as Finite-State Transduction [paper link] 2024-10-21
Marco Cognetta; Naoaki OkazakiTokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5 [paper link] 2024-10-15
Thao Anh Dang; Limor Raviv; Lukas GalkeFrom Tokens to Words: On the Inner Lexicon of LLMs [paper link] 2024-10-08
Guy Kaplan; Matanel Oren; Yuval Reif; Roy SchwartzNorm of Mean Contextualized Embeddings Determines their Variance [paper link] 2024-09-17
Hiroaki Yamagiwa; Hidetoshi ShimodairaWhere is the signal in tokenization space? [paper link] 2024-08-16
Renato Lui Geh; Honghua Zhang; Kareem Ahmed; Benjie Wang; Guy Van den BroeckMonotonic Representation of Numeric Properties in Language Models [paper link] 2024-08-15
Benjamin Heinzerling; Kentaro InuiReconsidering Token Embeddings with the Definitions for Pre-trained Language Models [paper link] 2024-08-02
Ying Zhang; Dongyuan Li; Manabu OkumuraData Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? [paper link] 2024-07-23
Jonathan Hayase; Alisa Liu; Yejin Choi; Sewoong Oh; Noah A. SmithScaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies [paper link] 2024-07-18
Chaofan Tao; Qian Liu; Longxu Dou; Niklas Muennighoff; Zhongwei Wan; Ping Luo; Min Lin; Ngai WongOn Initialization of Transformers with Pre-trained Embeddings [paper link] 2024-07-17
Ha Young Kim; Niranjan Balasubramanian; Byungkon KangAn Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models [paper link] 2024-07-08
Nandini Mundra; Aditya Nanda Kishore; Raj Dabre; Ratish Puduppully; Anoop Kunchukuttan; Mitesh M. KhapraUnderstanding and Mitigating Tokenization Bias in Language Models [paper link] 2024-06-24
Buu Phan; Marton Havasi; Matthew Muckley; Karen UllrichLarge Vocabulary Size Improves Large Language Models [paper link] 2024-06-24
Sho Takase; Ryokan Ri; Shun Kiyono; Takuya KatoTransformers Can Do Arithmetic with the Right Embeddings [paper link] 2024-05-27
Sean McLeish; Arpit Bansal; Alex Stein; Neel Jain; John Kirchenbauer; Brian R. Bartoldson; Bhavya Kailkhura; Abhinav Bhatele; Jonas Geiping; Avi Schwarzschild; Tom GoldsteinBy Tying Embeddings You Are Assuming the Distributional Hypothesis [paper link] 2024-05-02
Francesco Bertolotti; Walter CazzolaToward a Theory of Tokenization in LLMs [paper link] 2024-04-12
Nived Rajaraman; Jiantao Jiao; Kannan RamchandranOn the Effect of (Near) Duplicate Subwords in Language Modelling [paper link] 2024-04-09
Anton Schäfer; Thomas Hofmann; Imanol Schlag; Tiago PimentelTokenization Is More Than Compression [paper link] 2024-02-28
Craig W. Schmidt; Varshini Reddy; Haoran Zhang; Alec Alameddine; Omri Uzan; Yuval Pinter; Chris TannerSmall Transformers Compute Universal Metric Embeddings [paper link] 2022-10-18
Anastasis Kratsios; Valentin Debarnot; Ivan Dokmanić
Linear Attention / State Space Models / Recurrent Language Models / etc.
Papers analyzing alternative architectures to the standard transformer models, such as linear attention and state space models.
paper list (click to fold / unfold)
When Does Tool Use Increase the Expressive Power of Finite-Precision Recurrent Models? [paper link] 2026-07-07
Nikola Zubić;Qian Li;Yuyi Wang;Davide ScaramuzzaGhost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization [paper link] 2026-07-01
Peilin Liu;Ding-Xuan ZhouMuonSSM: Orthogonalizing State Space Models for Sequence Modeling [paper link] 2026-06-29
Thai-Khanh Nguyen;Ngoc-Bich-Uyen Vo;Thieu N. Vo;Tan M. Nguyen;Cuong PhamCARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention [paper link] 2026-06-25
Sayak DuttakNN Attention Demystified: A Theoretical Exploration for Scalable Transformers [paper link] 2024-11-06
Themistoklis HarisFundamental Limitations on Subquadratic Alternatives to Transformers [paper link] 2024-10-05
Josh Alman; Hantao YuAutoregressive + Chain of Thought (CoT) ≃ Recurrent: Recurrence's Role in Language Models and a Revist of Recurrent Transformer [paper link] 2024-09-14
Xiang Zhang; Muhammad Abdul-Mageed; Laks V.S. LakshmananTheory, Analysis, and Best Practices for Sigmoid Self-Attention [paper link] 2024-09-06
Jason Ramapuram; Federico Danieli; Eeshan Dhekane; Floris Weers; Dan Busbridge; Pierre Ablin; Tatiana Likhomanenko; Jagrit Digani; Zijin Gu; Amitis Shidani; Russ WebbRecurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations [paper link] 2024-08-20
Róbert Csordás; Christopher Potts; Christopher D. Manning; Atticus GeigerTransformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models [paper link] 2024-08-19
Aviv Bick; Kevin Y. Li; Eric P. Xing; J. Zico Kolter; Albert GuJust read twice: closing the recall gap for recurrent language models [paper link] 2024-07-07
Simran Arora; Aman Timalsina; Aaryan Singhal; Benjamin Spector; Sabri Eyuboglu; Xinyi Zhao; Ashish Rao; Atri Rudra; Christopher RéParallelizing Linear Transformers with the Delta Rule over Sequence Length [paper link] 2024-06-10
Songlin Yang; Bailin Wang; Yu Zhang; Yikang Shen; Yoon KimTransformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality [paper link] 2024-05-31
Tri Dao; Albert Gu
Training Paradigms
Categories discussing various training methodologies and paradigms for language models.
paper list (click to fold / unfold)
Trust Region Policy Distillation [paper link] 2026-07-06
Zhengpeng Xie;Li Lyna Zhang;Zeke Xie;Mao YangDemoPSD: Disagreement-Modulated Policy Self-Distillation [paper link] 2026-07-02
Yunhe Li;Hao Shi;Wenhao Liu;Mengzhe Ruan;Hanxu Hou;Zhongxiang Dai;Shuang Qiu;Linqi SongOn-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity [paper link] 2026-06-24
Andrei Liviu Nicolicioiu;Mohammad Pezeshki;Aaron CourvilleOn the Position Bias of On-Policy Distillation [paper link] 2026-06-21
Yan Xie;Sijie Zhu;Tiansheng Wen;Bo Chen;Yifei WangOn the Geometry of On-Policy Distillation [paper link] 2026-06-05
Zhennan Shen;Yanshu Li;Qingyu Yin;Chak Tou Leong;Zhilin Wang;Yanxu Chen;Rongduo Han;Sunbowen Lee;Yi R. FungKnowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget [paper link] 2024-04-30
Minh Duc Bui; Fabian David Schmidt; Goran Glavaš; Katharina von der WenseWhy are Adaptive Methods Good for Attention Models? [paper link] 2020-10-23
Jingzhao Zhang; Sai Praneeth Karimireddy; Andreas Veit; Seungyeon Kim; Sashank J. Reddi; Sanjiv Kumar; Suvrit Sra
Mechanistic Engineering / Probing / Interpretability
Categories exploring the internal mechanisms and interpretability of language models.
paper list (click to fold / unfold)
Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation [paper link] 2026-07-05
Kargi Chauhan;Aditya ShahSpectral Signatures of Large Language Models [paper link] 2026-07-03
Zhuoying Zhang;Ishan V. Prasad;Yuanzhe Hu;Zihang Liu;Hengrui Luo;Pu Ren;Yaoqing YangDistill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation [paper link] 2026-07-01
Shayan Talaei;Abhinav Chinta;Devvrit Khatri;Amin Karbasi;Azalia Mirhoseini;Amin SaberiShapley in Context: Explaining Financial Language with Domain Expertise [paper link] 2026-07-01
Dangxing Chen;Pengzhan GuoIG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients [paper link] 2026-06-29
Duc Anh NguyenWhat LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs [paper link] 2026-06-26
Nhi Nguyen;Shauli Ravfogel;Rajesh RanganathVision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models [paper link] 2026-06-26
Niclas Lietzow;Danielle Bitterman;Carsten Eickhoff;William Rudman;Michal GolovanevskyEvidence for feature-specific error correction in LLMs [paper link] 2026-06-23
Francisco Ferreira da Silva;Stefan HeimersheimPre-Generation Hallucination Detection in Large Language Models via Soft-Target Attention Probing [paper link] 2026-06-20
Amina Miftakhova;Alexey ZaytsevBeyond Hooking Onto the World: Referential Profiles and the Numerical Structure of LLM Grounding [paper link] 2026-06-19
Joo Yull RheeLeverage Is Not Reach: A Control-Window Law for Single-Neuron Steering in Language Models [paper link] 2026-06-18
Hongliang LiuFrom Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability [paper link] 2026-06-16
Dibyanayan Bandyopadhyay;Asif EkbalRevisiting the Systematicity in Negation in the Era of In-Context Learning [paper link] 2026-06-15
Hitomi Yanaka;Taisei YamamotoRethinking the Role of Efficient Attention in Hybrid Architectures [paper link] 2026-06-13
Ziqing Qiao;Yinuo Xu;Chaojun Xiao;Zhou Su;Zihan Zhou;Yingfa Chen;Xiaoyue Xu;Xu Han;Zhiyuan LiuBeyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective [paper link] 2026-06-13
Tao Jing;Ningxin Wu;Chen Kang;Dong Yu;Changliang Li;Pengyuan LiuTransformers Learn the Mestre-Nagao Heuristic [paper link] 2026-06-13
Pranav Venkata KondaBeyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Components [paper link] 2026-06-12
Yifeng Guo;Jin-Hong Du;Xiang ChenHow Linear Is a Transformer Feed-Forward Block? Per-Block Linear Recoverability Is Learned, Not Architectural [paper link] 2026-06-12
Stuart WhippWhere Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function [paper link] 2026-06-11
Atharva Gupta;Dhruv Kumar;Murari Mandal;Saurabh DeshpandeCan Editing 1 Neuron Fix Repetition Loops in LLMs? [paper link] 2026-06-09
Aristotelis Lazaridis;Aman Sharma;Dylan Bates;Brian King;Vincent Lu;Jack FitzGeraldA retrieval conditioned rebinding circuit for dynamic entity tracking in large language models [paper link] 2026-06-07
Soyoung Oh;Vera DembergWhen Attribution Patching Lies: Diagnosis and a Second-Order Correction [paper link] 2026-06-05
Luyang Zhang;Jialu WangHow Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures [paper link] 2026-06-04
Tanvi Thoria;Kiana Jafari;Marc R. Schlichting;Mykel J. KochenderferThe Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models [paper link] 2026-06-04
Jinyang Zhang;Hongxin Ding;Yue Fang;Weibin Liao;Muyang Ye;Junfeng Zhao;Yasha WangMechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads [paper link] 2026-06-04
Ruoxi Sun;Quantong Qiu;Juntao Li;Zecheng Tang;Yihang Lou;Min ZhangHow Transformers Solve Propositional Logic Problems: A Mechanistic Analysis [paper link] 2024-11-06
Guan Zhe Hong; Nishanth Dikkala; Enming Luo; Cyrus Rashtchian; Xin Wang; Rina PanigrahyTowards Interpreting Language Models: A Case Study in Multi-Hop Reasoning [paper link] 2024-11-06
Mansi SakarvadiaMechanisms of Symbol Processing for In-Context Learning in Transformer Networks [paper link] 2024-10-23
Paul Smolensky; Roland Fernandez; Zhenghao Herbert Zhou; Mattia Opper; Jianfeng GaoInterpreting Affine Recurrence Learning in GPT-style Transformers [paper link] 2024-10-22
Samarth Bhargav; Alexander GuExtracting Finite State Machines from Transformers [paper link] 2024-10-08
Rik Adriaensen; Jaron MaeneOptimal ablation for interpretability [paper link] 2024-09-16
Maximilian Li; Lucas JansonSelf-Attention Limits Working Memory Capacity of Transformer-Based Models [paper link] 2024-09-16
Dongyu Gong; Hantao ZhangExplaining Datasets in Words: Statistical Models with Natural Language Parameters [paper link] 2024-09-13
Ruiqi Zhong; Heng Wang; Dan Klein; Jacob SteinhardtExtracting Paragraphs from LLM Token Activations [paper link] 2024-09-10
Nicholas Pochinkov; Angelo Benoit; Lovkush Agarwal; Zainab Ali Majid; Lucile Ter-MinassianModularity in Transformers: Investigating Neuron Separability & Specialization [paper link] 2024-08-30
Nicholas Pochinkov; Thomas Jones; Mohammed Rashidur RahmanA Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models [paper link] 2024-08-16
Geonhee Kim; Marco Valentino; André FreitasMonotonic Representation of Numeric Properties in Language Models [paper link] 2024-08-15
Benjamin Heinzerling; Kentaro InuiThe Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights [paper link] 2024-08-05
Nura Aljaafari; Danilo S. Carvalho; André FreitasAnswer, Assemble, Ace: Understanding How Transformers Answer Multiple Choice Questions [paper link] 2024-07-21
Sarah Wiegreffe; Oyvind Tafjord; Yonatan Belinkov; Hannaneh Hajishirzi; Ashish SabharwalLLM Circuit Analyses Are Consistent Across Training and Scale [paper link] 2024-07-15
Curt Tigges; Michael Hanna; Qinan Yu; Stella BidermanTransformer Layers as Painters [paper link] 2024-07-12
Qi Sun; Marc Pickett; Aakash Kumar Nain; Llion JonesTransformer Circuit Faithfulness Metrics are not Robust [paper link] 2024-07-11
Joseph Miller; Bilal Chughtai; William SaundersMonitoring Latent World States in Language Models with Propositional Probes [paper link] 2024-06-27
Jiahai Feng; Stuart Russell; Jacob SteinhardtClustering in pure-attention hardmax transformers and its role in sentiment analysis [paper link] 2024-06-26
Albert Alcalde; Giovanni Fantuzzi; Enrique ZuazuaInterpreting Attention Layer Outputs with Sparse Autoencoders [paper link] 2024-06-25
Connor Kissane; Robert Krzyzanowski; Joseph Isaac Bloom; Arthur Conmy; Neel NandaLarge Language Models are Interpretable Learners [paper link] 2024-06-25
Ruochen Wang; Si Si; Felix Yu; Dorothea Wiesmann; Cho-Jui Hsieh; Inderjit DhillonTransformer Normalisation Layers and the Independence of Semantic Subspaces [paper link] 2024-06-25
Stephen Menary; Samuel Kaski; Andre FreitasConfidence Regulation Neurons in Language Models [paper link] 2024-06-24
Alessandro Stolfo; Ben Wu; Wes Gurnee; Yonatan Belinkov; Xingyi Song; Mrinmaya Sachan; Neel NandaFinding Transformer Circuits with Edge Pruning [paper link] 2024-06-24
Adithya Bhaskar; Alexander Wettig; Dan Friedman; Danqi ChenUnlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models [paper link] 2024-06-23
Tianyi Men; Pengfei Cao; Zhuoran Jin; Yubo Chen; Kang Liu; Jun ZhaoInsights into LLM Long-Context Failures: When Transformers Know but Don't Tell [paper link] 2024-06-20
Taiming Lu; Muhan Gao; Kuai Yu; Adam Byerly; Daniel KhashabiFrom RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries [paper link] 2024-06-18
Hitesh Wadhwa; Rahul Seetharaman; Somyaa Aggarwal; Reshmi Ghosh; Samyadeep Basu; Soundararajan Srinivasan; Wenlong Zhao; Shreyas Chaudhari; Ehsan AghazadehRefusal in Language Models Is Mediated by a Single Direction [paper link] 2024-06-17
Andy Arditi; Oscar Obeso; Aaquib Syed; Daniel Paleka; Nina Panickssery; Wes Gurnee; Neel NandaTalking Heads: Understanding Inter-layer Communication in Transformer Language Models [paper link] 2024-06-13
Jack Merullo; Carsten Eickhoff; Ellie PavlickScaling and evaluating sparse autoencoders [paper link] 2024-06-06
Leo Gao; Tom Dupré la Tour; Henk Tillman; Gabriel Goh; Rajan Troll; Alec Radford; Ilya Sutskever; Jan Leike; Jeffrey WuObservable Propagation: Uncovering Feature Vectors in Transformers [paper link] 2024-06-04
Jacob Dunefsky; Arman CohanFrom Neurons to Neutrons: A Case Study in Interpretability [paper link] 2024-05-27
Ouail Kitouni; Niklas Nolte; Víctor Samuel Pérez-Díaz; Sokratis Trifinopoulos; Mike WilliamsMechanistic Interpretability of Binary and Ternary Transformers [paper link] 2024-05-27
Jason LiInversionView: A General-Purpose Method for Reading Information from Neural Activations [paper link] 2024-05-27
Xinting Huang; Madhur Panwar; Navin Goyal; Michael HahnNot All Language Model Features Are Linear [paper link] 2024-05-23
Joshua Engels; Isaac Liao; Eric J. Michaud; Wes Gurnee; Max TegmarkAttention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers [paper link] 2024-05-22
Tobias Leemann; Alina Fastowski; Felix Pfeiffer; Gjergji KasneciSparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models [paper link] 2024-05-21
Charles O'Neill; Thang BuiAnchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions [paper link] 2024-05-06
Ruizhe Li; Yanjun GaoA Primer on the Inner Workings of Transformer-based Language Models [paper link] 2024-05-02
Javier Ferrando; Gabriele Sarti; Arianna Bisazza; Marta R. Costa-jussàGiLOT: Interpreting Generative Language Models via Optimal Transport [paper link] 2024-05-02
Xuhong Li; Jiamin Chen; Yekun Chai; Haoyi XiongTalking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs [paper link] 2024-04-25
Valeriia Cherepanova; James ZouInterpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions [paper link] 2024-02-22
Clement Neo; Shay B. Cohen; Fazl BarezUniversal Neurons in GPT2 Language Models [paper link] 2024-01-22
Wes Gurnee; Theo Horsley; Zifan Carl Guo; Tara Rezaei Kheirkhah; Qinyi Sun; Will Hathaway; Neel Nanda; Dimitris BertsimasInterpretability Illusions in the Generalization of Simplified Models [paper link] 2023-12-06
Dan Friedman; Andrew Lampinen; Lucas Dixon; Danqi Chen; Asma GhandehariounTransformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars [paper link] 2023-12-03
Kaiyue Wen; Yuchen Li; Bingbin Liu; Andrej RisteskiWhite-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? [paper link] 2023-11-22
Yaodong Yu; Sam Buchanan; Druv Pai; Tianzhe Chu; Ziyang Wu; Shengbang Tong; Hao Bai; Yuexiang Zhai; Benjamin D. Haeffele; Yi MaMechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott KruegerUnderstanding the Mechanics and Dynamics of Memorisation in Large Language Models: A Case Study with Random Strings [paper link] 2023-10-13
Till Speicher; Aflah Mohammad Khan; Qinyuan Wu; Vedant Nanda; Soumi Das; Bishwamittra Ghosh; Krishna P. Gummadi; Evimaria Terzi
Miscellanea
Categories for papers that do not fit neatly into other classifications but discuss theoretical or empirical aspects of language models.
paper list (click to fold / unfold)
Quantitative Gaussian-Process limits of Tensor Programs [paper link] 2026-07-07
Andrea Agazzi;Eloy Mosig García;Dario TrevisanFrom Application-Layer Simulation to Native Meta-Architecture: Structural Tension as an Endogenous Driver for Heterogeneous AI Evolution [paper link] 2026-07-07
Heting MaoSafe Inference-Time Alignment via Lagrangian Reward Augmentation [paper link] 2026-07-02
Yaswanth Chittepu;Ativ Joshi;Sohini Chintala;Scott NiekumWeave of Formal Thought [paper link] 2026-06-24
Alexandre BouayadSVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression [paper link] 2026-06-22
Mahmoud Safari;Frank HutterAn In-depth Investigation of Sparse Rate Reduction in Transformer-like Models [paper link] 2024-11-26
Yunzhe Hu; Difan Zou; Dong XuOn the goals of linguistic theory: Revisiting Chomskyan theories in the era of AI [paper link] 2024-11-15
Eva Portelance; Masoud JasbiLength-Induced Embedding Collapse in Transformer-based Models [paper link] 2024-10-31
Yuqi Zhou; Sunhao Dai; Zhanshuo Cao; Xiao Zhang; Jun XuAnalyzing & Reducing the Need for Learning Rate Warmup in GPT Training [paper link] 2024-10-31
Atli Kosson; Bettina Messmer; Martin JaggiA Theoretical Perspective for Speculative Decoding Algorithm [paper link] 2024-10-30
Ming Yin; Minshuo Chen; Kaixuan Huang; Mengdi WangInevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models [paper link] 2024-10-27
Zhengmian Hu; Heng HuangOptimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection [paper link] 2024-10-18
Aaron Alvarado Kristanto Julistiono; Davoud Ataee Tarzanagh; Navid AzizanFine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes [paper link] 2024-10-12
Xiaoyu Li; Yingyu Liang; Zhenmei Shi; Zhao Song; Yufa ZhouMind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers [paper link] 2024-10-10
Alireza Naderi; Thiziri Nait Saada; Jared TannerDynamic metastability in the self-attention model [paper link] 2024-10-09
Borjan Geshkovski; Hugo Koubbi; Yury Polyanskiy; Philippe RigolletDecoding Game: On Minimax Optimality of Heuristic Text Generation Strategies [paper link] 2024-10-04
Sijin Chen; Omar Hagrass; Jason M. KlusowskiHow to Train Long-Context Language Models (Effectively) [paper link] 2024-10-03
Tianyu Gao; Alexander Wettig; Howard Yen; Danqi Chensoftmax is not enough (for sharp out-of-distribution) [paper link] 2024-10-01
Petar Veličković; Christos Perivolaropoulos; Federico Barbero; Razvan PascanuOn the Implicit Relation Between Low-Rank Adaptation and Differential Privacy [paper link] 2024-09-26
Saber Malekmohammadi; Golnoosh FarnadiA Controlled Study on Long Context Extension and Generalization in LLMs [paper link] 2024-09-18
Yi Lu; Jing Nathan Yan; Songlin Yang; Justin T. Chiu; Siyu Ren; Fei Yuan; Wenting Zhao; Zhiyong Wu; Alexander M. RushBeyond Parameter Count: Implicit Bias in Soft Mixture of Experts [paper link] 2024-09-02
Youngseog Chung; Dhruv Malik; Jeff Schneider; Yuanzhi Li; Aarti SinghReframing Data Value for Large Language Models Through the Lens of Plausability [paper link] 2024-08-30
Mohamad Rida Rammal; Ruida Zhou; Suhas DiggaviMulti-Layer Transformers Gradient Can be Approximated in Almost Linear Time [paper link] 2024-08-23
Yingyu Liang; Zhizhou Sha; Zhenmei Shi; Zhao Song; Yufa ZhouA Tighter Complexity Analysis of SparseGPT [paper link] 2024-08-22
Xiaoyu Li; Yingyu Liang; Zhenmei Shi; Zhao SongGreat Memory, Shallow Reasoning: Limits of kNN-LMs [paper link] 2024-08-21
Shangyi Geng; Wenting Zhao; Alexander M RushLearning Randomized Algorithms with Transformers [paper link] 2024-08-20
Johannes von Oswald; Seijin Kobayashi; Yassir Akram; Angelika StegerLanguage Models as Models of Language [paper link] 2024-08-13
Raphaël MillièreEnhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Models [paper link] 2024-08-09
Yiming Luo; Patrick Cheong-Iao; Shanton ChangData Debugging is NP-hard for Classifiers Trained with SGD [paper link] 2024-08-02
Zizheng Guo; Pengyu Chen; Yanzhang Fu; Dongjing MiaoDancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models [paper link] 2024-07-31
Zhengxuan Wu; Yuhao Zhang; Peng Qi; Yumo Xu; Rujun Han; Yian Zhang; Jifan Chen; Bonan Min; Zhiheng HuangOn the Benefits of Rank in Attention Layers [paper link] 2024-07-23
Noah Amsel; Gilad Yehudai; Joan BrunaFundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models [paper link] 2024-07-22
Adway Girish; Alliot Nagle; Marco Bondaschi; Michael Gastpar; Ashok Vardhan Makkuva; Hyeji KimIn-Context Probing Approximates Influence Function for Data Valuation [paper link] 2024-07-17
Cathy Jiao; Gary Gao; Chenyan XiongOn Initialization of Transformers with Pre-trained Embeddings [paper link] 2024-07-17
Ha Young Kim; Niranjan Balasubramanian; Byungkon KangOn Exact Bit-level Reversible Transformers Without Changing Architectures [paper link] 2024-07-12
Guoqiang Zhang; J.P. Lewis; W. B. KleijnImplicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations [paper link] 2024-07-10
Yize Zhao; Tina Behnia; Vala Vakilian; Christos ThrampoulidisUniversal Length Generalization with Turing Programs [paper link] 2024-07-03
Kaiying Hou; David Brandfonbrener; Sham Kakade; Samy Jelassi; Eran MalachEfficient Training of Language Models with Compact and Consistent Next Token Distributions [paper link] 2024-07-03
Ashutosh Sathe; Sunita SarawagiUnderstanding Transformers via N-gram Statistics [paper link] 2024-06-30
Timothy NguyenEvaluating n-Gram Novelty of Language Models Using Rusty-DAWG [paper link] 2024-06-25
William Merrill; Noah A. Smith; Yanai ElazarA Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens [paper link] 2024-06-25
Zhijie Nie; Richong Zhang; Zhanyu WuConnecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data [paper link] 2024-06-20
Johannes Treutlein; Dami Choi; Jan Betley; Cem Anil; Samuel Marks; Roger Baker Grosse; Owain EvansDemystifying Forgetting in Language Model Fine-Tuning with Statistical Analysis of Example Associations [paper link] 2024-06-20
Xisen Jin; Xiang RenOn Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier [paper link] 2024-06-20
Jiachen Jiang; Jinxin Zhou; Zhihui ZhuHow to Compute the Probability of a Word [paper link] 2024-06-20
Tiago Pimentel; Clara MeisterToward Infinite-Long Prefix in Transformer [paper link] 2024-06-20
Jiuxiang Gu; Yingyu Liang; Zhenmei Shi; Zhao Song; Chiwun YangUnveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis [paper link] 2024-06-19
Rachel S.Y. Teo; Tan M. NguyenTextual Unlearning Gives a False Sense of Unlearning [paper link] 2024-06-19
Jiacheng Du; Zhibo Wang; Kui RenAttention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters [paper link] 2024-06-18
Zhiyu Guo; Hidetaka Kamigaito; Taro WatanabeExploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance [paper link] 2024-06-18
Anna C. Marbut; John W. Chandler; Travis J. WheelerUnderstanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models [paper link] 2024-06-13
Sarah Ball; Frauke Kreuter; Nina RimskyInterpretability of Language Models via Task Spaces [paper link] 2024-06-10
Lucas Weber; Jaap Jumelet; Elia Bruni; Dieuwke HupkesHow Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States [paper link] 2024-06-09
Zhenhong Zhou; Haiyang Yu; Xinghua Zhang; Rongwu Xu; Fei Huang; Yongbin LiAttention as a Hypernetwork [paper link] 2024-06-09
Simon Schug; Seijin Kobayashi; Yassir Akram; João Sacramento; Razvan PascanuVerbalized Machine Learning: Revisiting Machine Learning with Language Models [paper link] 2024-06-06
Tim Z. Xiao; Robert Bamler; Bernhard Schölkopf; Weiyang LiuLocal to Global: Learning Dynamics and Effect of Initialization for Transformers [paper link] 2024-06-05
Ashok Vardhan Makkuva; Marco Bondaschi; Chanakya Ekbote; Adway Girish; Alliot Nagle; Hyeji Kim; Michael GastparPre-trained Large Language Models Use Fourier Features to Compute Addition [paper link] 2024-06-05
Tianyi Zhou; Deqing Fu; Vatsal Sharan; Robin JiaComputational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models [paper link] 2024-06-05
Jerry Yao-Chieh Hu; Maojiang Su; En-Jui Kuo; Zhao Song; Han LiuRethinking Spiking Neural Networks as State Space Models [paper link] 2024-06-05
Malyaban Bal; Abhronil SenguptaLongSSM: On the Length Extension of State-space Models in Language Modelling [paper link] 2024-06-04
Shida WangOn Affine Homotopy between Language Encoders [paper link] 2024-06-04
Robin SM Chan; Reda Boumasmoud; Anej Svete; Yuxin Ren; Qipeng Guo; Zhijing Jin; Shauli Ravfogel; Mrinmaya Sachan; Bernhard Schölkopf; Mennatallah El-Assady; Ryan CotterellAnisotropy is Not Inherent to Transformers [paper link] 2024-06
Anemily Machina; Robert MercerA Theory of In-Context Learning in Transformers [paper link] 2024-05-29
Yifei Wang; Yuyang Wu; Zeming Wei; Stefanie Jegelka; Yisen WangLower Bounds on the Expressivity of Recurrent Neural Language Models [paper link] 2024-05-29
Anej Svete; Franz Nowak; Anisha Mohamed Sahabdeen; Ryan CotterellDemystifying amortized causal discovery with transformers [paper link] 2024-05-27
Francesco Montagna; Max Cairney-Leeming; Dhanya Sridhar; Francesco LocatelloUnlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective [paper link] 2024-05-27
Zhen Qin; Xuyang Shen; Dong Li; Weigao Sun; Stan Birchfield; Richard Hartley; Yiran ZhongCan Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? [paper link] 2024-05-27
Gal Yona; Roee Aharoni; Mor GevaTowards Understanding How Transformer Perform Multi-step Reasoning with Matching Operation [paper link] 2024-05-24
Zhiwei Wang; Yunji Wang; Zhongwang Zhang; Zhangchen Zhou; Hui Jin; Tianyang Hu; Jiacheng Sun; Zhenguo Li; Yaoyu Zhang; Zhi-Qin John XuDissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers [paper link] 2024-05-24
Lorenzo Tiberi; Francesca Mignacco; Kazuki Irie; Haim SompolinskyAttention as an RNN [paper link] 2024-05-22
Leo Feng; Frederick Tung; Hossein Hajimirsadeghi; Mohamed Osama Ahmed; Yoshua Bengio; Greg MoriSurgical Feature-Space Decomposition of LLMs: Why, When and How? [paper link] 2024-05-17
Arnav Chavan; Nahush Lele; Deepak GuptaDynamic Activation Pitfalls in LLaMA Models: An Empirical Study [paper link] 2024-05-15
Chi Ma; Mincong Huang; Chao Wang; Yujie Wang; Lei YuChallenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis [paper link] 2024-05-14
Yao FuUnderstand LLMs Requires More Than Statistical Generalization [paper link] 2024-05-03
Patrik Reizinger; Szilvia Ujváry; Anna Mészáros; Anna Kerekes; Wieland Brendel; Ferenc HuszárViewing Transformers Through the Lens of Long Convolutions Layers [paper link] 2024-05-02
Itamar Zimerman; Lior WolfModeling Language Tokens as Functionals of Semantic Fields [paper link] 2024-05-02
Zhengqi Pei; Anran Zhang; Shuhui Wang; Qingming HuangCompression Represents Intelligence Linearly [paper link] 2024-04-15
Yuzhen Huang; Jinghan Zhang; Zifei Shan; Junxian HeLanguage Generation in the Limit [paper link] 2024-04-10
Jon Kleinberg; Sendhil MullainathanDo language models plan ahead for future tokens? [paper link] 2024-03-31
Wilson Wu; John X. Morris; Lionel LevineWhat's In My Big Data? [paper link] 2024-03-05
Yanai Elazar; Akshita Bhagia; Ian Magnusson; Abhilasha Ravichander; Dustin Schwenk; Alane Suhr; Pete Walsh; Dirk Groeneveld; Luca Soldaini; Sameer Singh; Hanna Hajishirzi; Noah A. Smith; Jesse DodgeDo Efficient Transformers Really Save Computation? [paper link] 2024-02-21
Kai Yang; Jan Ackermann; Zhenyu He; Guhao Feng; Bohang Zhang; Yunzhen Feng; Qiwei Ye; Di He; Liwei WangLong Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning [paper link] 2024-02-07
Hao Zhao; Maksym Andriushchenko; Francesco Croce; Nicolas FlammarionProvably learning a multi-head attention layer [paper link] 2024-02-06
Sitan Chen; Yuanzhi LiAnisotropy Is Inherent to Self-Attention in Transformers [paper link] 2024-01-24
Nathan Godey; Éric de la Clergerie; Benoît SagotUniversality and Limitations of Prompt Tuning [paper link] 2023-11-16
Yihan Wang; Jatin Chauhan; Wei Wang; Cho-Jui HsiehData Similarity is Not Enough to Explain Language Model Performance [paper link] 2023-11-15
Gregory Yauney; Emily Reif; David MimnoSimplifying Transformer Blocks [paper link] 2023-11-03
Bobby He; Thomas HofmannCausal Interpretation of Self-Attention in Pre-Trained Transformers [paper link] 2023-10-31
Raanan Y. Rohekar; Yaniv Gurwicz; Shami NisimovHow do Language Models Bind Entities in Context? [paper link] 2023-10-26
Jiahai Feng; Jacob SteinhardtUnderstanding prompt engineering may not require rethinking generalization [paper link] 2023-10-13
Victor Akinwande; Yiding Jiang; Dylan Sam; J. Zico KolterUnderstanding Catastrophic Forgetting in Language Models via Implicit Inference [paper link] 2023-09-18
Suhas Kotha; Jacob Mitchell Springer; Aditi RaghunathanAttention-Only Transformers and Implementing MLPs with Attention Heads [paper link] 2023-09-15
Robert Huben; Valerie MorrisOn the Role of Attention in Prompt-tuning [paper link] 2023-06-15
Samet Oymak; Ankit Singh Rawat; Mahdi Soltanolkotabi; Christos Thrampoulidis
Detailed Statistics
Phenomena of Interest:
In-Context Learning: 109
Chain-of-Thought: 27
Hallucination: 17
Reversal Curse: 5
Scaling Laws / Emergent Abilities / Grokking / etc.: 62
Knowledge / Memory Mechanisms: 34
Training Dynamics / Landscape / Optimization / Fine-tuning / etc.: 102
Learning / Generalization / Reasoning / Weak to Strong Generalization: 70
Other Phenomena / Discoveries: 48
Representational Capacity:
What Can Transformer Do? / Properties of Transformer: 85
What Can Transformer Not Do? / Limitation of Transformer: 36
Architectural Effectivity:
Layer-normalization: 7
Tokenization / Embedding: 23
Linear Attention / State Space Models / Recurrent Language Models / etc.: 13
Training Paradigms: 7
Mechanistic Engineering / Probing / Interpretability: 73
Miscellanea: 93
Related links:
Contact
- Shiguang Wu, furyton AT outlook.com / shiguang.wu AT mail.sdu.edu.cn