Skip to content
Projekte

cmsflash/efficient-attention

Auf GitHub ansehen

cmsflash/efficient-attention has 332 GitHub stars and 26 forks. Star changes are not measured yet. Last checked 2026-11-18 05:59 UTC. Last push 2020-11-30.

So werden diese Zahlen gemessen

Gemessener Sternzuwachs

in 24h
Aktualisierung verzögert
in 7 days
Aktualisierung verzögert
in 15 days
Aktualisierung verzögert
in 1 month
Aktualisierung verzögert

Nettoänderungen in überlappenden Zeiträumen, keine Tagessummen. Fehlende Messwerte bleiben leer. Sterne zeigen Interesse, nicht Qualität.

Über uns

An implementation of the efficient attention module.

Maschinelles Lernen

README.md

Original

Efficient Attention

An implementation of the efficient attention module.

Description

Efficient attention is an attention mechanism that substantially optimizes the memory and computational efficiency while retaining exactly the same expressive power as the conventional dot-product attention. The illustration above compares the two types of attention. The efficient attention module is a drop-in replacement for the non-local module (Wang et al., 2018), while it:

  • uses less resources to achieve the same accuracy;
  • achieves higher accuracy with the same resource constraints (by allowing more insertions); and
  • is applicable in domains and models where the non-local module is not (due to resource constraints).

Resources

YouTube:

bilibili (for users in Mainland China):

Implementation details

This repository implements the efficient attention module with softmax normalization, output reprojection, and residual connection.

Features not in the paper

This repository implements additionally implements the multi-head mechanism which was not in the paper. To learn more about the mechanism, refer to Vaswani et al.

Citation

The paper will appear at WACV 2021. If you use, compare with, or refer to this work, please cite

@inproceedings{shen2021efficient,
    author = {Zhuoran Shen and Mingyuan Zhang and Haiyu Zhao and Shuai Yi and Hongsheng Li},
    title = {Efficient Attention: Attention with Linear Complexities},
    booktitle = {WACV},
    year = {2021},
}