知の物理学研究センター / Institute for Physics of Intelligence (iπ)
【日時/Date】
2023年12月5日（火）10時30分～12時 / Dec. 5, 2023, 10:30 – 12:00 (JST)
【発表者/Speaker】
太田 敏博 氏（ サイバーエージェント AI Lab）
【タイトル/Title】
“Hopfield/Mixer correspondence: towards a better understanding of MetaFormers architecture design”
【概要/Abstract】
In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models (MetaFormers) that compete with Transformers, such as the MLP-Mixer. Despite their weak induced bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this talk, we discuss how the modern Hopfield networks may be useful for a unified understanding of MetaFormer. In particular, we propose Hopfield/Mixer correspondence as a new direction for MetaFormers architecture design, and introduce a novel MetaFormer model theoretically derived from the correspondence. Finally, possible extensions and their prospects will be discussed.
※この講演に関するZoomのリンク等の案内を受け取ることを希望されるかたは、下記のgoogle formからメールアドレスをご記入ください。こちらに登録頂いた情報は、案内の配信のみに利用いたします。
登録フォーム：https://forms.gle/xnLmd9Kc1BaaNPgq8
世話人：知の物理学研究センター 髙橋昂, 中西健