Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity - nikkie-memos

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

https://arxiv.org/abs/2101.03961

T5をベースにフィードフォワード層にMoEを利用とのこと

Figure 2

検証記事 https://www.ogis-ri.co.jp/otc/hiroba/technical/similar-document-search/part16.html