MinD-Vis
https://gyazo.com/5af3e3ad9555acfe057ab411736b8538
Abstract
Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, due to the scarcity of data annotations and the complexity of underlying brain information, it is challenging to decode images with faithful details and meaningful semantics. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Specifically, by boosting the information capacity of feature representations learned from a large-scale resting-state fMRI dataset, we show that our MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings with very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. 画像を見せて、脳のどこが活性化しているか？を調べたfMRIデータセットがある
深層学習によって、逆にこのfMRIから対応する画像を復元できるようになった ただし、データセットの不足とfMRIから直接、複雑な神経活動を復号するために有用な生物学的指針(？)がないため、出来上がる画像は不鮮明
個人差によるばらつきもある
これをなんちゃらかんちゃらして精度を上げた
(降参ですnomadoor.icon)