from Hacker News

twitter/the-algorithm

by johns on 4/25/22, 9:01 PM with 380 comments

• by johns on 4/25/22, 9:17 PM

  Will Norris who works on OSS at Twitter posted this[0]: "watch this space https://github.com/twitter/the-algorithm"

  [0]: https://twitter.com/willnorris/status/1518694675909013504

• by Irishsteve on 4/25/22, 9:12 PM

  Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.

• by jdrc on 4/25/22, 9:21 PM

  I know the algorithm i use, it ends with ORDER BY date DESC.

• by shrimpx on 4/25/22, 9:39 PM

  I don't understand the concept of open-sourcing "the algorithm".

  First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic.

  And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neural nets without the training data is mostly meaningless from a transparency perspective?

• by mrintellectual on 4/25/22, 9:14 PM

  This seems to be a practical joke by a Twitter engineer as opposed to an actual release.

• by axg11 on 4/25/22, 9:42 PM

  I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense.

  Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of everything. What most people think of as "the algorithm" is a complex system that receives many inputs (maybe hundreds) and has dependencies on many other internal Twitter services. Tweets likely pass through multiple filtering steps as well as scoring before you ever see them. Each of these steps is highly contextual, depending on: location, past tweets, verification status, etc. You can attempt to predict the effect of a certain change, but you never know the actual outcome until you test it.

  I think what will ultimately happen is that _some_ details will be published. Elon will parade that around as a victory for free speech as Twitter is now more "open". In reality, nothing of value will be gained as "the algorithm" isn't a simple function.

• by transitivebs on 4/25/22, 9:11 PM

  Is this supposed to be a joke? It's clearly an empty repo.

  Either this is a mistake, or this is a really, really misguided attempt at a joke from Twitter.

• by mrkramer on 4/25/22, 9:30 PM

  Imagine having something like this for Google's and YouTube's algorithms; $100bn+ SEO industry would go bankrupt or at least they would pivot to some sort of advising but there wouldn't be the mayhem that we have today.

• by CincinnatiMan on 4/25/22, 9:14 PM

  Makes me wonder how Twitter employees internally are handling the news. If they are celebrating or commiserating?

• by standyro on 4/25/22, 9:34 PM

  I tried to make a pull request already, haha.

  error forking repo: HTTP 403: The repository exists, but it contains no Git content. Empty repositories cannot be forked. (https://api.github.com/repos/twitter/the-algorithm/forks)

  My thoughts:

  - Explicit rules for temporary and permanent bans

  - Edit button

  - More fun and thoughtful conversations like HN

  - Less thought bubble Brooklyn based reporters, less VC and side grind hustle snake oil, maybe more comedians and memes?

• by edouard-harris on 4/26/22, 1:58 AM

  Assuming Twitter is serious about publishing their feed algorithm [1], it's possible they're merely anticipating the EU's upcoming Digital Services Act which was finalized over the weekend. Among other things, the Act will compel large platforms to "make the working of their recommender algorithms (used for sorting content on the News Feed or suggesting TV shows on Netflix) transparent to users." [2]

  Twitter's EU user base is probably [3] above the 45 million threshold that triggers the strictest transparency requirements under the Act. So perhaps they figure if they're going to be forced to disclose anyway, they might as well do it proactively.

  [1] If it's even coherent to talk about their feed ranking system as a single algorithm — see the other comments in this thread.

  [2] https://www.theverge.com/2022/4/23/23036976/eu-digital-servi...

  [3] https://www.statista.com/statistics/242606/number-of-active-...

• by nighthawk454 on 4/25/22, 9:14 PM

  Seems weird to start as a non-private repo until there's some content. Also bit of an unusual name. Can't tell if this is internal trolling or the future

• by nickysielicki on 4/25/22, 9:22 PM

  Surely you guys don’t think that twitters sorting algorithm is already factored out into its own repo. Of course it’s empty.

  That doesn’t mean it’s a joke, I see it as a show of goodwill — that there are a handful of people inside Twitter that are excited for transparency and for a revenue model that isn’t entirely based on ads, that are excited to get to work on this right away.

• by pddpro on 4/25/22, 9:18 PM

  Does it remind anyone of Po and the Dragon scroll from Kung Fu Panda?

• by rickreynoldssf on 4/25/22, 10:12 PM

  Wait until Musk finds out its a bunch of gnarly PHP 5.4 code much of which is a black box everyone is afraid to touch.

• by paxys on 4/25/22, 9:36 PM

  I'm going to guess some engineers at Twitter with Github org permissions are having fun with the "release the algorithm" discussion.

• by Barrin92 on 4/25/22, 9:37 PM

  whatever will show up in this repo, I hope people realize that depending on what data you put into some algorithm you can get whatever output you want, and twitter is never going to (and neither can or should they) publish everyone's personal information and interaction on the site.

  So I'm not sure what the ultimate point of this exercise is other than producing faux-transparency.

• by xena on 4/25/22, 9:35 PM

  I can't believe they missed the chance to make it a rick roll. Such a wasted opportunity.

• by NaturalPhallacy on 4/25/22, 10:33 PM

  Not "the algorithm", but you can check if twitter is silently suppressing your account here: https://taishin-miyamoto.com/ShadowBan/

• by unethical_ban on 4/25/22, 9:22 PM

  There are elements of their algo that I think should be openly defined, and perhaps there should be some regulatory branch that reports to Congress that has full access. However, obfuscation is often necessary to countering bad actors.

• by newbamboo on 4/25/22, 10:31 PM

  The government, at federal, state and local levels, all rely on Twitter to conduct official taxpayer funded work. Taxpayer funded work should not happen on proprietary systems that operate with zero oversight or public transparency.

  Elon polled Twitter users about this and the response was overwhelmingly in favor of open source and transparency. Everyone on Twitter got a vote.

  If you oppose transparency, as many now are, you lose your credibility. So it’s another one of Elon’s people hacks, and look at all the morons falling for it.

• by EMIRELADERO on 4/25/22, 9:20 PM

  Kind of unrealistic but I hope Twitter now open-sources not only the algorithm but also the Rails monolith itself. Would be kind of interesting to see how everything is done

• by qgin on 4/25/22, 9:45 PM

  I have literally no idea how a "twitter algorithm" could be published on github. Maybe I've been doing recommender systems wrong.

• by bpodgursky on 4/25/22, 10:31 PM

  I'm very technical and I think it would still be valuable to have a list of all the things that weight into the timeline view, even without the models or underlying data.

  Like, there's no public admission right now of whether "shadow banning" or "ghost banning" is even officially a thing!

  Some transparency seems unquestionably more powerful than none, and we can work from there.

• by yabones on 4/25/22, 9:15 PM

  There is something vaguely threatening about this.

• by rvz on 4/25/22, 9:20 PM

  Perhaps Twitter will be the new Mozilla if it decides to open-source 'everything' then.

  Maybe that is where it is going.

• by holtkam2 on 4/25/22, 9:13 PM

  I don't get it ¯\_(ツ)_/¯

• by Traster on 4/25/22, 10:29 PM

  At the time of posting, Will Norris (the open source lead at twitter, admin of their github account presumably) posted this. It has 44 retweets, 193 likes, 17 quote tweets, on github it has 1.6k stars.

  That seems... bizarre to me?

• by sakopov on 4/26/22, 1:21 AM

  I agree that there is no such thing as "the algorithm." It is Twitter in its entirety. And with that I have a wild question. Can Musk make Twitter fully open-source on GitHub?

• by g105b on 4/25/22, 9:21 PM

  Can someone explain this to me? All I can see from this link is an empty GitHub repository. Not sure what I'm missing here.

• by threeseed on 4/25/22, 9:32 PM

  Anyone who actually uses Twitter already knows the algorithm:

  * Chronological - reverse sort by date

  * Home - for all of the followed topics, recommended topics, retweets and tweets in the past day determine the estimated level of engagement, include the highest and reverse sort by date. This is likely to be a fairly basic ML model.

  It will be uncontroversial, technically unsophisticated and of no practical use to anyone - users, developers or researchers.

  This is not going to be PageRank where some genuine new insight was discovered.

• by Synaesthesia on 4/25/22, 9:22 PM

  So nobody is being shadowbanned or suppressed?

• by hazb on 4/26/22, 12:47 AM

  "The algorithm" could mean a lot of things. Whatever it means, it probably spans hundreds or even thousands of services. That doesn't mean it cannot be made open-source.

  I imagine they'd probably start with documentation and white-papers that communicate "here's how we intend for it to work".

  It's seriously unlikely anyone in Twitter knows actually works how any non-trivial algorithm in the company works. To figure THAT out, they could decide to do a company-wide documentation and instrumentation push like they probably would've had to do for GDPR anyway, which is painful and boring and going to take a very long time.

  Failing that, they could just say 'the algorithm as it stands is no longer fit for purpose, given part of its core requirement has become that it needs to be transparent and publishable, and presumably legible. We need to make a new one. Publish the core algorithm. We probably won't deploy it in that exact state, it's going to span multi-services and so on, you obviously don't get the data we used to train the models, but we will work backwards from it and here's an open mechanism to measure how true-to-form it actually is'

• by u1tron on 4/26/22, 5:36 AM

  It's already gone.

• by tmaly on 4/26/22, 1:58 AM

  I could see GPT-3 being added in the empty space.

• by minroot on 4/25/22, 9:58 PM

  Why do we want to know the "algorithm"?

• by zelon88 on 4/25/22, 9:18 PM

  if ($has_blue_checkmark) show_post_to($everyone);

• by qudat on 4/25/22, 10:10 PM

  I’ve spent the better part of a decade writing open source projects for few to see. An empty repo gets hundreds of stars immediately. It’s all a popularity contest.

• by drnonsense42 on 4/25/22, 9:50 PM

  Apples are red. The sky is blue. Twitter shadowbans and tinkers with who sees who. I wonder what the old guard will do with the codebase over the next few months.

• by LugarOS on 4/25/22, 9:11 PM

  It's empty.

• by a-dub on 4/25/22, 10:18 PM

  it's probably just a ripoff of pagerank with a separate spam filtering and banning system along with an army of contractors manually fixing it up.

  if twitter is a game, sinking $43bn into it is kinda like winning or losing the grand final boss level. (unclear which)

  wish elon would get back to facilitating the building of useful things. we still don't have a great clean energy generation story.

• by TrapLord_Rhodo on 4/25/22, 9:23 PM

  Musks first order of business?

• by oxplot on 4/25/22, 10:40 PM

  Musk has repeatedly talked about "open sourcing" twitter's algorithm. Given Musk is (understandably) super impatient, this repo may be his first move. I expect this to start with bunch of readme and other high level docs and evolve into details and eventually code.

• by asd88 on 4/25/22, 9:41 PM

  #drama?

• by 4e530344963049 on 4/25/22, 10:10 PM

  Nice, making it much easier to game!

• by arthurcolle on 4/25/22, 9:58 PM

  is this performance art?

• by ArtWomb on 4/25/22, 9:14 PM

  It was all in your head ;)