from Hacker News

Is Hacker News a good predictor of future tech trends?

by drpancake on 8/24/21, 1:45 PM with 264 comments

by MattGaiser on 8/24/21, 2:01 PM
1. For things like remote work, it only became a trend because of the pandemic. You would have needed to predict the pandemic to predict that management would permit remote work. It was not really a trend, but rather an emergency measure that morphed into a desirable benefit.
2. This only looks at successful things. We must examine unsuccessful things to see if there is any real predictive ability.
by afavour on 8/24/21, 2:09 PM
Feels like a great case study in confirmation bias: you only looked at the topics that ended up being successful. How many tech trends did Hacker News hype up only to have them go nowhere? Without that info we don’t really know what HN’s success rate is.
by prewett on 8/24/21, 2:53 PM
HN is pretty good at identifying trends. I generally read some comment here way before I hear about it somewhere else. Does HN accurately predict where the trend goes? Not necessarily. But if something actually does trend, I will have heard low-level chatter about it on HN for a long time.
Remote work trended because of the pandemic, for example, but there was constant chatter about remote work anytime the topic of employment came up, and the monthly job match-making posts even post whether the job / worker is open to remote. Functional programming has had low-level chatter from the beginning (given that PG viewed it as one of the reasons for the success of his company), and indeed, functional programming has grown more mainstream. I started exploring Python for my Perl-type tasks because it kept getting good mentions on HN.
I don't think any of the above ever got to the point where it got so many mentions it would hit a strong 100 in his methodology compared to Google trends. I'm also not sure you could identify successful trends early on, but I think you hear chatter on HN long before other places.
by mettamage on 8/24/21, 2:59 PM
From the top of my head (a very biased account):
* Dolphin emulator progress reports (I love it when they show up)
* Random posts on x86-64 emulation or emulating a gameboy or something like that. Fabrice Bellard is a hero.
* People who post their very first side project ever made either via no-code solutions or some React/Bootstrap type of thing. In many cases, these things are job boards for a specific niche.
* HN praising Apple for standing up to the FBI (or CIA?). HN hating Apple because of recent events with scanning pictures on iCloud and all that.
* All the things the blog post talks about
* Hardcore learning resources on many topics, one that I actually used: http://neuralnetworksanddeeplearning.com/ (tons of books on Calculus as well, I remember a 1900 textbook called Calculus Made Easy)
* The random physics or biological discovery
* And so much more
The way I see it: HN talks about tech (and interesting stuff outside tech), it's bound to cover tech that eventually becomes popular. But will the Dolphin progress reports become as popular as Bitcoin? I don't think so. It'd be interesting times if they would ;-)
by shotta on 8/24/21, 2:05 PM
This only looked at submissions it seems. I gather a lot more useful information lurking the comments. I think discussions in the comments here are far ahead of the mainstream.
by bane on 8/24/21, 3:17 PM
HN is fantastic for surfacing up and coming technologies. Reading HN daily is almost like a superpower.
The HN community is...not so great...at predicting which things are exciting and the industry should adopt. If it were the industry would have settled on a LISP dialect long ago, the web would have adopted good readability standards (based on thousands of meta-complaints on virtually every top rated post on any topic), and mobile app development would still be a growing and profitable business (and Apple would finally be catering to niche HN reader needs).
by wiradikusuma on 8/24/21, 2:31 PM
Hacker News, like any news aggregator, pops up new stuff all the time.
The hardest part is keeping an open mind for jumping on early. Hard because 99% turn out to be dud anyway, not to mention our crowd are skeptical bunch.
My biggest "regret" is ignoring Bitcoin/Ether.
Saw Bitcoin posts when it was $5.. nah it's a scam.
Oh it's gone up to $10.. nah it won't go up further.
Went up further.. nah won't buy bcoz already too expensive.
Ether announced.. lolz nobody going to buy another coin since there's already Bitcoin..
Ether goes up.. WTF.
by OJFord on 8/24/21, 3:16 PM
> Remote Work [...] I thought [in contradistinction to the graph] that Hacker News would have been more ahead of the trend on this one.
> For each topic I counted how many times it had been mentioned in a post title
I suspect that's the problem. A lot of trendy discussion happens in comments; I think that's going to be particularly true for a topic like remote work.
People are going to extol its virtues when it's tangentially related to a submission, but especially pre-pandemic there's only so many stories you could submit about it? 'Company goes full-remote'? 'How we handle remote working at Company'?
It'd be interesting to search comments too I think. hn.algolia.com supports it.
by hackitup7 on 8/24/21, 5:29 PM
I find that when Hacker News really likes a piece of highly technical SaaS technology it's usually pretty good. The best example I can think of is Datadog which got rave reviews here years ago and has continued to be enormously successful. Particularly for enterprise tech as there tends to be less herd behavior up-market. If I'm diligencing a technology I'll often actually look at HN to see the opinions.
by keewee7 on 8/24/21, 3:11 PM
The mood on Hacker News (and reddit/twitter) has become too pessimistic to be a good predictor of the future.
The long-term trend is that the world is becoming a better place. Pessimistic people are biased against that fundamental trend that other trends will develop on. Groups of pessimistic people will not be good predictors of the future.
by leugim on 8/24/21, 1:56 PM
I'd say no. Lots of (successful) ideas have been dismissed by this community.
And that is okey.
by New_California on 8/24/21, 2:33 PM
Bitcoin was completely and totally missed by HN and more broadly by Silicon Valley.
Only recently the SV/HN community is catching up and waking up to cryptocurrencies - and still hesitantly.
by yuchi on 8/24/21, 3:09 PM
Only tangential but still interesting.
During the infamous leap second fiasco of Java in 2012 which brought down all JVM servers by saturating the CPU, I saved the day because I was reading about it here on HN while my colleagues were struggling to undestrand why all our environments stopped working.
It took me a few minutes to realize that we were indeed hit by what I just read, look at ten of my colleagues each closely inspecting logs at different ssh sessions, and gleefully give them a precise description of the problem, an approach to verify it and a solution, adding “it’s all over the news guys, you should keep yourself updated”.
by jsnell on 8/24/21, 6:20 PM
One thing worth nothing is that HN didn't launch publicly until 2007 February. There is some older content in the DB, but it looks mostly like very low volume manual testing. So the data set definitely shouldn't be extended that far back, and is affecting a few of the be graphs.
by kkdaemas on 8/24/21, 1:57 PM
In case of NFTs these were called "Colored Coins" much earlier.
by asciimov on 8/24/21, 2:28 PM
No. This is just confirmation bias.
The author didn't control for what was on the front page, how many points the article received, or the amount of discussion on the topic. Not to mention the myriad of technologies that are on Hacker News that fizzle.
by rafale on 8/24/21, 2:17 PM
They missed crypto and NFTs. For some reason HN doesn't appreciate the potential of these tech. Even when some prominent Silicon Valley investors do. I wouldn't have expected this from a technical and freedom minded audience.
by username91 on 8/24/21, 2:15 PM
I'm not aware of HN doing any predicting at large. If you mean 'will the average person later act like a given segment of HN people do now?' - then it depends on how good you are at selecting your segment, which makes it no better or worse than using any other sample of people for prediction.
Less abstractly, productisation is what takes things from obscurity (e.g. HN) to mainstream audiences, so future trends might grow out of stuff that you see here, but the people who can figure out whether certain tech can become a product (would-be predictors) -hopefully- also have the means to turn it into a product themselves.
by f154hfds on 8/24/21, 3:06 PM
Two things I learned on HN before they became mainstream:
* Rust
* Covid
For the latter I remember thinking that it was going to become a big deal soon but no one seemed to be paying attention except for those paranoid HNers.
by hnbad on 8/24/21, 2:00 PM
"My conclusion: Hacker News is typically ahead of the mainstream, often by a few years, but you would need to be paying very close attention to catch the early mentions of a new tech trend. Most of the linked posts and comments have very few upvotes and probably wouldn't even make it to the front page."
That's a long way of saying "no".
by reducesuffering on 8/24/21, 7:58 PM
If you payed attention and believed in HN, you would've invested in:
AMD, Square, Cloudflare, Fiverr, Apple, Nvidia, TSM, Twilio, CrowdStrike, Zoom, Atlassian, Moderna, and Shopify.
Maybe a 5 year 500% return compared to the general market's 100%?
I'd say your fat stacks of $ would have you believing it's a pretty good predictor of something.
by p0nce on 8/24/21, 5:36 PM
Hacker News skews towards richer, more technical, and higher-educated and then makes systematic bias against pedestrian, working-class, and uncomplicated solutions. If it's too popular on HN I'd say it's a red flag.
by lend000 on 8/24/21, 6:06 PM
I've read about an indie trading algorithm that works like this. It's basically a PageRank for professional forums, which apparently did quite well for the author. A more fundamental-based form of sentiment analysis.
by oakpond on 8/25/21, 1:02 PM
Regarding your conclusion:
> My conclusion: Hacker News is typically ahead of the mainstream, often by a few years, but you would need to be paying very close attention to catch the early mentions of a new tech trend. Most of the linked posts and comments have very few upvotes and probably wouldn't even make it to the front page.
HN also tracks front-page history: news.ycombinator.com/front. To go back to a particular date all you need to do is give the date, i.e. news.ycombinator.com/front?day=2020-08-24. Seems to go back to 2006-10-09.
by modeless on 8/24/21, 4:33 PM
HN is the best way of learning about future tech trends before they happen. But it's not because there's any kind of prescient consensus here. The important stuff is a small minority. You have to do your own filtering.
I was ahead of the curve on Bitcoin, VR, and deep learning because I read about them early here, but they weren't the most popular things at the time. HN hates Bitcoin to this day. But I learned about it from HN and decided that HN was wrong about it. When lots of people are wrong about something, I see it as an opportunity.
by yablak on 8/24/21, 3:54 PM
https://en.wikipedia.org/wiki/Survivorship_bias
by slingnow on 8/24/21, 1:59 PM
Using total number of mentions as a way to predict future tech trends seems dubious at best. I understand coming up with a better metric would be difficult, but this one seems useless.
For example, with this metric, how can we tell the difference between people constantly dismissing Bitcoin or people constantly praising it? Just because you mention it doesn't mean you're mentioning it in a positive light.
by turdnagel on 8/24/21, 2:59 PM
What is the prediction being made? Whether or not something is a "trend"? What is the definition of a trend?
> Can I convince myself that checking the HN front page multiple times a day is a useful and productive exercise?
That seems to be a different topic entirely and should probably be the headline of the article. Also, it's clearly a bit cheeky, but confirmation bias abounds nonetheless.
by ChicagoBoy11 on 8/24/21, 3:06 PM
Of things which are "interesting," absolutely. But to the stickiness of these things I'd say the record is mixed. If you lurk here chances are you'll hear about any single tech topic earlier than most other places. But that the ones you hear about here (or are most upvoted here) also tend to be the ones with greater stickiness is not clear to me at all.
by FridayoLeary on 8/24/21, 1:55 PM
I like to see analysis like this that sheds light on the collective 'mind' of HN. Are there more of these out there?
by jedimastert on 8/24/21, 2:12 PM
Egdelord opinion time: I think HN has way too much of a hivemind to really think outside the box enough to catch trends.
by throwthere on 8/24/21, 2:22 PM
This is a good start. It does not do much to answer the question of "predicting" trends. You would need to quantify false positives somehow. That is, tech that is big on HN but not elsewhere. I'm thinking of all the JS frameworks of the day, for one. For instance, Meteor JS was the bees knees for awhile.
by bdcravens on 8/24/21, 4:41 PM
It's barely an indicator of what's currently popular, skewing heavily to subsections of the tech world
by mateo1 on 8/24/21, 3:47 PM
As with all online communities HN is suffering from it's increased popularity. If you remember how Reddit was a good 10 years back, on every thread there would be 1-2 actual experts posting substantiated and well-sourced commentary on news relevant to their field. Usually undergraduates and young professionals passionate about their field. At some point this was discovered by journalists and for a brief period of time major news organizations would rely on reddit users for sources. Many if not most discussions would be done in good faith or at least with solid arguments. The userbase was usually 18+. As it got more popular it got worse, mainly because of the "eternal september", altering userbase due to smartphone proliferation and changes in ranking algorithms and business model.
HN is not the same as reddit and isn't doomed to the same faults, but as as a forum gets more popular, experts reduce in percentage, they're more reluctant and careful on what and how they comment, the "populism" of the voting system becomes harmful and all the attention creates financial incentives for bad actors.
As for using it as some kind of blackbox betting algorithm for startups and trends, that's just a bad idea. I'm certain that you will get the same graph structure for all things that never became popular.
Obviously any forum focused on leading edge technologies will discuss advancements long before they become a subject of interest in the general populace. But they'll also discuss everything that will fail or never reach the mainstream sphere.
by brundolf on 8/24/21, 3:28 PM
I would say it's more a good radar for what (software, mostly web) technologies are entering general awareness. If something might be successful one day, you'll probably find out about it. And then lots of those things will sputter out, but not all of them.
by chubot on 8/24/21, 2:48 PM
Hm never saw this thread about "fixing RAZR" as a pre-approved YC idea (pre-Android and pre-iPhone), but it's pretty interesting:
https://news.ycombinator.com/item?id=13465
by dougSF70 on 8/24/21, 2:07 PM
Basically, no one knows what is going to catch in until it has. As one poster mentioned, false positives could be plentiful. Also, for companies or brands such as Tesla, their PR machine will be working harder than HN to spread the word, so not a good data point.
by Ensorceled on 8/24/21, 2:26 PM
I think a lot of people on HN are already working remote (I've been 100% remote since before the pandemic). It's a business process and business management issue that doesn't really come up on Hacker News except in productivity discussions.
by enriquto on 8/24/21, 2:45 PM
if it was, everybody would be coding in rust using neovim, which does not really seem to happen
by salynchnew on 8/24/21, 6:53 PM
This is a great example of survivorship bias! A+ work. :)
https://en.wikipedia.org/wiki/Survivorship_bias#As_a_general...
by ArtWomb on 8/24/21, 2:19 PM
If something looks useful, I'll sign up day one. Even if it doesn't pan out. That's how you get early adopter status. Found out about: Figma, Airtable, Stripe, Coinbase, Robinhood, Alpaca. All through HN posts ;)
by sbussard on 8/24/21, 2:23 PM
Remote work is so obvious to the hacker news crowd that there’s no need for it to trend here. What’s artificial is going to an office to use a computer there. There better be food to make up for the traffic.
by creaghpatr on 8/24/21, 2:00 PM
Love the visualizations, I would also want to compare against false positives though. From my time here, I feel like there's been a lot of flops that never broke into the public conscious.
by chaostheory on 8/24/21, 4:30 PM
I would say no. I made the first post on Bitcoin here.
https://news.ycombinator.com/item?id=599852
by vmception on 8/24/21, 3:58 PM
Just find whatever is being ridiculed on HN and that’s the alpha bet
by drdrey on 8/24/21, 5:57 PM
The serverless post on the Scoble blog is not even about what we think of as Serverless now, just good old cloud. As in "you don't have any physical servers?"
by chansiky on 8/27/21, 9:16 PM
Is no one going to mention Rust? If HN is good at predicting trends, then we're about to see a lot more Rust in a few years.
by esjeon on 8/24/21, 4:44 PM
As backed by this post, HN isn’t precisely good at prediction. It simply has far less noise than the general internet, making it easier to notice the upcoming trends.
by artembugara on 8/24/21, 5:36 PM
https://en.wikipedia.org/wiki/Survivorship_bias
by heywherelogingo on 8/24/21, 2:19 PM
Skimming the front page there's nothing on future tech.
by justinzollars on 8/24/21, 5:20 PM
It looks like interest in Bitcoin has peaked on HN and in decline; We could extrapolate the same will happen on google in two years.
by yodsanklai on 8/24/21, 2:08 PM
HN is certainly a source to know what is trendy. I'd be interested to see a list of trends based on HN top posts and comment.
by tomcooks on 8/24/21, 4:48 PM
Cue in joke about that Dropbox thread comment
by peter303 on 8/24/21, 6:54 PM
A class project would be to use machine learning on these graphs to spot trends. HN has lots of data examples.
by doo_daa on 8/24/21, 4:25 PM
(How) did you disambiguate Tesla as in Tesla Cars from Tesla as in Nikola Tesla from the either source?
by mFixman on 8/24/21, 2:11 PM
> For each topic I counted how many times it had been mentioned in a post title, then calculated a 0-100 index, where 100 is the month with the most mentions.
This is not really useful data then. A time series with a big spike will "lose" against an identical one without the spike.
Maybe the amount of days when a certain word was in the front page would be a better comparison metric on the HN side?
by Razengan on 8/24/21, 7:33 PM
On the contrary. The very nature of vote-based forums like Reddit and HN means that you must agree with the masses and uphold the status quo to even be seen at all let alone heard, so it’s very rare that you get a glimpse of upcoming disruption or anything new at all.
TL;DR: No, most people here are sticks in the mud. If you listened to them you’d think that future laptops would still have parallel ports and floppy drives and communicate via IRC.
by dqpb on 8/24/21, 3:32 PM
Now do the other study - where on the Internet were these things first talked about.
by xyzzy21 on 8/24/21, 2:43 PM
No really. Unless you think the ONLY future tech is certain niches in software.
by devops000 on 8/24/21, 3:34 PM
You forgot the "coronavirus" or "pandemic" in the analysis
by littleweep on 8/24/21, 2:22 PM
If so, we might have (actually) seen the year of the Linux desktop.
by jbverschoor on 8/24/21, 9:25 PM
For cloud storage and collaboration (Dropbox) -> Nope
Blockchain -> Nope
by lvl100 on 8/24/21, 10:56 PM
This place is pretty right leaning and mostly front-end.
by theonlybutlet on 8/24/21, 1:58 PM
Would be interesting to see the prefix "meta"
by lamontcg on 8/24/21, 7:04 PM
this suffers from survivorship bias though and isn't looking at stuff that hacker news yaps about but never happens.
by m1117 on 8/24/21, 4:30 PM
Hacker News's a great source of rage.
by jpswade on 8/24/21, 4:13 PM
No, it predicted Dropbox would be dead.
by tommek4077 on 8/24/21, 2:09 PM
Yes, if JavaScript is a future trend ;)
by wetpaws on 8/24/21, 6:08 PM
Dropbox folks would smile at this post
by privatdozent on 8/24/21, 4:34 PM
Cool analysis!
by timbaboon on 8/24/21, 2:11 PM
Hacker News causes success of future tech trends :P
by MontyCarloHall on 8/24/21, 3:20 PM
Relevant XKCD: https://xkcd.com/1497/
by resheku on 8/24/21, 7:57 PM
no, it isn't
by nix23 on 8/24/21, 2:28 PM
No, pretty much the opposite.
by black_13 on 8/24/21, 1:59 PM
A lot of self involved bay area types
by bpodgursky on 8/24/21, 2:09 PM
LOL
Hacker News is where you to to learn that self-hosted rack-mount servers in your own basement running a LAMP stack provisioned with self-generated bash scripts is the future.