from Hacker News

X-Transformers: A fully-featured transformer with experimental features

by blackcat201 on 5/8/21, 5:38 PM with 37 comments

by throwawaybbq1 on 5/9/21, 12:10 AM
FYI .. I work in deep learning and lucidrains is becoming a legend in my line of work. Seems someone who is obsessed about transformers (the deep learning ones, and rightly so, they are amazing). To the author (if you are reading this on HN), I want to thank you for the amazing work you have done!
For the non-DL crowd, Transformers are a tsunami in deep learning for the past few years. They are topping benchmarks in many subfields. I do research professionally and this work is amazingly useful for people like me.
by fao_ on 5/8/21, 9:57 PM
As others have mentioned, anything obscure like this should literally come with a Wikipedia (or other such) link to explain what it is, what it does. This is the primary problem with small project READMEs, imo. They assume you're already familiar with them and know what the hell they are. Like, take Ironhide:
```
    https://github.com/MrMEEE/ironhide
    Optimus Support for Linux Through VirtualGL - PPA version also available
```
That's... great. So it's doing something with GL, and it's running on Linux, but uhhh.
```
    my branch of the original bumblebee project..
```
What is Optimus? What is Bumblebee? The trick of it is that it links to a blog where neither of these terms are ever explained. Maybe it's to just look impressive on someone's CV? How could I even tell the difference?
Likewise for this project, all you need in the README is one line that's like:
```
   X-Transformers is a re-implementation of Machine Learning Transformers that has been built based on experimental Arxiv papers
```
It's a one-line fix but it'll stop people like me being confused as to whether or not you're implementing a new HTTP header
by bratao on 5/9/21, 2:24 AM
lucidrains and Ice Cream are my references in terms of research, knowledge and productivity. Phil was always available to guide and hear me. One time I told him about an underground research in another language and he was kind enough to check if it had any merit.
About X-Transformers, it is a very great piece of engineering that implemented almost of all possible improvements in transformers. But according to my experience and Phil himself, only the Feedforward GLU and RoPe (Rotary Positional Embeddings) works (or to be fair, they show improvements in more general use-cases)
by argvargc on 5/8/21, 9:19 PM
Unfortunately for me, I genuinely thought this was going to be a DIY robot build that could disguise itself as something else.
by adontz on 5/8/21, 10:25 PM
I have expected to see a 3D model for Optimus Prime.
by bravura on 5/9/21, 9:57 AM
What do you use for images that don’t have identical height and width? It seems the image transformer here expects square images.
by krick on 5/8/21, 10:43 PM
That's really cool. Now I need a bunch of pre-trained models for this...
by mrfusion on 5/8/21, 8:48 PM
Explain like I’m a first year CS major?
by shayankh on 5/8/21, 9:44 PM
absolutely fucking amazing