Skip to content
プロジェクト

cmsflash/efficient-attention

GitHubで見る

cmsflash/efficient-attention has 332 GitHub stars and 26 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 04:17 UTC. Last push 2020-11-30.

この数値の測り方

計測したスター増加

24時間
0
7日間
履歴を収集中
15日間
履歴を収集中
1か月
履歴を収集中

期間ごとの純増数です。期間は重複しており、日別の値ではありません。未計測の値は空欄です。スター数は注目度の指標であり、品質の保証ではありません。

概要

An implementation of the efficient attention module.

機械学習

Efficient Attention

An implementation of the efficient attention module.

Description

Efficient attention is an attention mechanism that substantially optimizes the memory and computational efficiency while retaining exactly the same expressive power as the conventional dot-product attention. The illustration above compares the two types of attention. The efficient attention module is a drop-in replacement for the non-local module (Wang et al., 2018), while it:

  • uses less resources to achieve the same accuracy;
  • achieves higher accuracy with the same resource constraints (by allowing more insertions); and
  • is applicable in domains and models where the non-local module is not (due to resource constraints).

Resources

YouTube:

bilibili (for users in Mainland China):

Implementation details

This repository implements the efficient attention module with softmax normalization, output reprojection, and residual connection.

Features not in the paper

This repository implements additionally implements the multi-head mechanism which was not in the paper. To learn more about the mechanism, refer to Vaswani et al.

Citation

The paper will appear at WACV 2021. If you use, compare with, or refer to this work, please cite

@inproceedings{shen2021efficient,
    author = {Zhuoran Shen and Mingyuan Zhang and Haiyu Zhao and Shuai Yi and Hongsheng Li},
    title = {Efficient Attention: Attention with Linear Complexities},
    booktitle = {WACV},
    year = {2021},
}