Research Interests: Learning Algorithms Beyond the Gradient

References

(Download as pdf)

This document collects my current research interests and related papers. Interested in similar topics? Let's chat!

Natural gradient descent and second-order optimization for LLMs [1]–[3], distributed training [4]
Automatic differentiation [5], specifically computing higher-order derivatives for scientific ML [6], [7]
Curvature/Covariance (Hessian, Fisher, …) matrix estimation [8]–[10] with structured matrices
Curvature information outside optimization [11] (e.g. training data attribution [12], [13])
Symbolic computing, compute graph surgery, ML compilers [14], [15]
Randomized linear algebra [16], [17]
Neural network parameter space symmetries [18], [19]

References

[1]

K. Jordan et al., “Muon: An optimizer for hidden layers in neural networks,” 2024.

[2]

N. Vyas et al., “Soap: Improving and stabilizing shampoo using adam,” Arxiv, 2025.

[3]

V. Gupta, T. Koren, and Y. Singer, “Shampoo: Preconditioned stochastic tensor optimization.” 2018.

[4]

H.-J. M. Shi et al., “A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale.” 2023.

[5]

A. Griewank and A. Walther, Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.

[6]

M. J. Johnson, J. Bettencourt, D. Maclaurin, and D. Duvenaud, “Taylor-mode higher-order automatic differentiation.” 2021.

[7]

R. Li et al., “Forward laplacian: A new computational framework for neural network-based variational monte carlo.” 2023.

[8]

J. Martens and R. Grosse, “Optimizing neural networks with Kronecker-factored approximate curvature,” 2015.

[9]

T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent, “Fast approximate natural gradient descent in a kronecker-factored eigenbasis,” Advances in neural information processing systems (neurips), 2018.

[10]

W. Lin, S. C. Lowe, F. Dangel, R. Eschenhagen, Z. Xu, and R. B. Grosse, “Understanding and improving shampoo and soap via kullback-leibler minimization,” Arxiv, 2025.

[11]

F. Dangel, R. Eschenhagen, W. Ormaniec, A. Fernandez, L. Tatzel, and A. Kristiadi, “Position: Curvature matrices should be democratized via linear operators,” Arxiv, 2025.

[12]

R. Grosse et al., “Studying large language model generalization with influence functions.” 2023.

[13]

P. W. Koh and P. Liang, “Understanding black-box predictions via influence functions,” 2017.

[14]

V. Roulet and A. Agarwala, “Per-example gradients: a new frontier for understanding and improving optimizers,” Arxiv, 2025.

[15]

F. Dangel, T. Siebert, M. Zeinhofer, and A. Walther, “Collapsing taylor mode automatic differentiation,” 2025.

[16]

A. Guzmán-Cordero, F. Dangel, G. Goldshlager, and M. Zeinhofer, “Improving energy natural gradient descent through woodbury, momentum, and randomization,” 2025.

[17]

A. Fernandez, F. Dangel, P. Hennig, and F. Schneider, “Sketching low-rank plus diagonal matrices,” Arxiv, 2025.

[18]

M. F. da Silva, F. Dangel, and S. Oore, “Hide & seek: Transformer symmetries obscure sharpness & riemannian geometry finds it,” 2025.

[19]

D. Kunin, J. Sagastuy-Brena, S. Ganguli, D. L. Yamins, and H. Tanaka, “Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics,” 2021.

Research Interests: Learning Algorithms Beyond the Gradient

Table of Contents

References