by tester89 on 3/3/24, 8:55 PM with 4 comments
by light_hue_1 on 3/4/24, 2:22 AM
> Hebbian learning, can implement a transformer-like attention computation if the synaptic weight changes are large and rapidly induced
Well this has no analog in the brain either.
In any case. How is this transformer like attention? Not all attention mechanisms lead to a transformer. Certainly the two are not synonymous.
> While it is tempting to assume that cortical rhythms might play a role, we have found that match windows around 0.5—2 s are necessary for reducing noise in the spike train comparisons, a timescale much longer than the cycles of theta, beta or gamma rhythms found in the cortex.
Well that's totally the wrong timescale.
There's a glut of these "abuse some random feature of the brain that already serves a different purpose to implement something that clearly doesn't work but is vaguely reminiscent of something that happens in machine learning so we'll call the two the same". They contribute nothing.
The few somewhat worthwhile actually show a working network for a real task. The real breakthrough will be a paper that actually works for real, on real data, and can be implemented in the brain; we've got nothing like that. This isn't one of the good ones.
by adamnemecek on 3/4/24, 2:26 AM