from Hacker News

Three things everyone should know about Vision Transformers

by reqo on 4/24/25, 3:53 PM with 17 comments

• by Centigonal on 4/24/25, 4:14 PM

  There's something that tickles me about this paper's title. The thought that everyone should know these three things. The idea of going to my neighbor who's a retired K-12 teacher and telling her about how adding MLP-based patch pre-processing layers improves Bert-like self-supervised training based on patch masking.

• by i5heu on 4/24/25, 4:38 PM

  I put this paper into 4o so i can check if it is relevant, so that you do not have to do this too here are the bullet points:

  - Vision Transformers can be parallelized to reduce latency and improve optimization without sacrificing accuracy.

  - Fine-tuning only the attention layers is often sufficient for adapting ViTs to new tasks or resolutions, saving compute and memory.

  - Using MLP-based patch preprocessing improves performance in masked self-supervised learning by preserving patch independence.