from Hacker News

DeepSeek's Multi-Head Latent Attention

by the_origami_fox on 3/3/25, 7:42 AM with 1 comments

  • by fspeech on 3/3/25, 11:06 PM

    Matrix absorption is unnecessary. What is needed is the order of multiplication associates towards the direction of the absorption. This and the modified Rope are needed to make the caching work.