Skip to content
プロジェクト

samuela/git-re-basin has 518 GitHub stars and 44 forks. GitHot measured -1 stars in 24 hours. Last checked 2026-11-20 04:38 UTC. Last push 2023-03-07.

この数値の測り方

計測したスター増加

24時間
-1
7日間
履歴を収集中
15日間
履歴を収集中
1か月
履歴を収集中

期間ごとの純増数です。期間は重複しており、日別の値ではありません。未計測の値は空欄です。スター数は注目度の指標であり、品質の保証ではありません。

概要

Code release for "Git Re-Basin: Merging Models modulo Permutation Symmetries"

機械学習

Git Re-Basin: Merging Models modulo Permutation Symmetries

Video demonstrating the effect of our permutation matching algorithm on the loss landscape throughout training.

Code for the paper Git Re-Basin: Merging Models modulo Permutation Symmetries.

Abstract:

The success of deep learning is thanks to our ability to solve certain massive non-convex optimization problems with relative ease. Despite non-convex optimization being NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes contain (nearly) a single basin, after accounting for all possible permutation symmetries of hidden units. We introduce three algorithms to permute the units of one model to bring them into alignment with units of a reference model. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10 and CIFAR-100. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity across a variety of models and datasets. Finally, we discuss shortcomings of a single basin theory, including a counterexample to the linear mode connectivity hypothesis.