by kertoip_1 on 5/26/22, 2:57 PM with 170 comments
Is it only that XMPP was overcomplicated and if managed better it would survive? Or is it something else?
by tetraca on 5/26/22, 4:59 PM
The environment outside of major providers was (it definitely is) kind of janky. If you can set up a server application it's not hard but it's not exactly easy to get started, or get people on it without handholding (from my own experience).
by jcranmer on 5/26/22, 3:50 PM
SMTP is an old protocol; it really predates much of what we'd think of as the Internet (e.g., it predates DNS or even IPv4). This means that during the big initial growth phase of the Internet, SMTP was already an established standard that could be used to transfer email. Indeed, for email routing on the internet, you kind of had to support SMTP anyways, which makes it difficult for any other protocol to gain traction.
That's not to say that there weren't alternative protocols. The biggest of these was X.400, which many in the 90s saw as the eventual replacement of SMTP. But this was hampered by the already existing install base of SMTP. Other failures leading to the failure of X.400 was its reliance on the OSI stack, which lagged behind in implementation compared to the TCP/IP stack, and the rather cumbersome addressing model of X.400 compared to SMTP. The value added in supporting X.400 in addition to SMTP wasn't worth the cost of implementing X.400, and it wasn't really feasible to support only X.400 given the already widespread use of SMTP.
In this model, XMPP actually works closer to X.400 than it does to SMTP. It came about after the protocols it aimed to replace were already in existence--and wide use. (See also IRC, which is still alive and active). So implementing XMPP means you need to justify the value-add of XMPP over existing legacy protocols. If you were implementing your own, new chat service, it might make sense to build it on top of XMPP instead of a custom protocol. But replacing existing chat protocols with XMPP was again a costly move with benefits rarely justifying the move. The further federation goal of XMPP would have been an anti-goal to many of the chat implementations.
by tapoxi on 5/26/22, 3:24 PM
* Onboarding was difficult. There was no obvious choice of server or client to use.
* Adding friends was difficult. You needed to send a subscription request to a contact, and they needed to send one to you. If anything happened during this process, you couldn't chat.
* Popular XMPP clients, like Pidgin, also supported the other chat services (AIM, ICQ, MSN, Yahoo, etc) so people just continued using those.
* Network effect. You need to convince a mass of people its better, otherwise nobody's using it because no-one uses it.
* No obvious benefit to the user. It's decentralized sure, but there weren't many improvements over AIM that people actually used.
* A lack of good iPhone XMPP clients.
In 2005 Google added XMPP support to Google Talk/GMail Chat and they were federated, but nobody federated back and they closed off its successor (Hangouts).
by taway2022-05-26 on 5/26/22, 3:27 PM
At work (operations for a space mission), we use an XMPP-based chat system for tactical communications along with several other systems, but Slack was introduced and more and more communcations are being handled over Slack. I'm advocating for replacing several systems with Jitsi, which is XMPP-based, because it allows us better control and customizability. It's an uphill battle.
by jasode on 5/26/22, 3:47 PM
There are probably multiple reasons and we can't replay history to know which reason contributes the most but one key difference is that email accounts (SMTP) were pushed onto the regular non-techie consumers whereas XMPP was more of a geek tool that users had to pull and opt into.
E.g. New students at a university automatically got a ".edu" email address. Residents got a free email account (ISP) with their cable service. Employees got a corporate email address. In other words, millions effortlessly got the utility of email/SMTP even without installing AOL CDROMS.
To further reinforce email/STMP, if consumers want to order something from Amazon, they needed an email address to create an account instead of an XMPP address.
Other forms of communication like XMPP/IRC don't have that widely disseminated self-reinforcing utility cycle so they stay a niche tool. What critical service in normal life requires an XMPP address? I can't think of any.
by dragontamer on 5/26/22, 4:52 PM
All of these federated protocols: XMPP and even Mastodon to an extent, are smaller groups of humble nerds running instances. They aren't massively advertised, large-scale billion dollars of investment companies like Discord or Facebook, trying to build network effects and/or advertising revenue.
-----------
When competing against profitable companies, its not sufficient to merely exist. You're competing against advertising and eyeballs. Facebook is big because its big, because it advertises, because it is constantly pushing for more-and-more users. Same with Twitter, same with Discord.
This leads to little advantages: a $10-million+ UI overhaul every few years. Designers to simplify the interface and onboard faster. Paying Google/Apple absurd amounts of money to access the push-notification APIs on phones. Etc. etc.
Smaller XMPP instances already fail at the push-notification thing. Who will pay for that? And without push-notifications, do you really have a modern chat platform? Other companies can afford the costs.
by api on 5/26/22, 3:23 PM
One is that XMPP is an overcomplicated nightmare. The protocol was verbose, flabby, and hard to implement, and the server software was very hard to set up and run especially for novices. A person could not just install a server and start chatting, even if the server was just stand-alone let alone linked to anyone else.
The second and IMHO more fundamental reason is that the Internet is a dark forest. Any open system that becomes sufficiently popular will be destroyed by abuse.
If you offer a chance to make any amount of money whatsoever online millions of hustlers will rush into the void like gas molecules invading a cracked vacuum bell and will scramble all over each other to suck every last fraction of a penny out until nothing of value remains. So far only centralized managed systems have been (somewhat) successful at keeping the barbarians at bay. I'm not saying a decentralized system could never succeed here, but I don't think it's been done yet. (Cryptocurrency isn't an example as it's already been throughly destroyed for its original vision and use case by scammers. It's a great case study in exactly what I mean.)
(Edit: the goal is not always money either. Read money as "value." Political propaganda, cult recruitment, weaponized disinformation, or just trolling for lulz all count as extracting value of some kind at the expense of the commons.)
SMTP along with Usenet was one of the first casualties of this phenomenon, and I would argue that it did fail as an open system. You can run your own SMTP server but it's not for novices or people without time on their hands. You'll have to fight constantly to keep your IP out of blacklists and to keep spam away from your users. Spammers are in an arms race against huge companies with massive training data sets and entire teams dedicated to spam filtering, so this only gets harder over time. An independent SMTP server is easy prey.
99% or more of users use one of several large mail hosts. These are mostly Google, Microsoft, and Apple in the USA. E-mail hosting is cheap to free and the vast majority of people would rather someone else deal with the pain of defending them from spam.
by runjake on 5/26/22, 6:02 PM
But, like commenter tetraca mentioned, XMPP has been embraced and extended, and it's not very interoperable, as implemented.
SMTP has been and will always be around forever, albeit eventually viewed like FAX is today. I personally view it like that today.
by timbre1234 on 5/26/22, 4:54 PM
by shireboy on 5/26/22, 3:43 PM
by dangerface on 5/26/22, 3:59 PM
As developers we look too often for technical reasons for a limitation or failure but usually the social reasons are more significant.
Getting a client installed on a computer was socially difficult like convincing some one to install an app on their phone. Having a browser installed on every computer and phone made it trivial to get a client on a computer the only down side is being forced to use http and websockets as the protocol.
DNS and WHOIS used to be protocols in their own right but both are being replaced by http, I wouldn't be surprised if http came for smtp next.
by GekkePrutser on 5/26/22, 5:26 PM
Of all the original internet protocols, SMTP is the ice that needs replacement the most IMO.
by kazinator on 5/26/22, 4:13 PM
I tried XMPP early on, and found the software to have a user experience that was utter garbage, so after that I didn't give it a second thought.
At the enterprise level, though, garbage UX is the norm. SMTP is hard to deploy, but you have paid full-time sysadmins whose job it is to figure out the necessary parts of sendmail.cf or whatever else. You tell them "make it so" and they do.
I suspect there was no enterprise level push for XMPP anywhere.
by Zash on 5/26/22, 3:28 PM
by eternityforest on 5/26/22, 3:21 PM
As far as any individual user is concerned, loss of the server(Which their identity is tied to) would be a hassle. Just like email.
SMTP got really entrenched really early. It's what gmail uses. They can't really do a proprietary mail protocol, SMTP was around way before them and nobody wanted to switch.
Instant messaging for the masses was proprietary from the start, as far as I can tell.
There was IRC, but it seems like by the time the internet was everywhere, people were already on AOL and ICQ and a bazillion other proprietary ones offering various extras and integrations.
None of them seem that interested in open protocols.
by blihp on 5/26/22, 4:16 PM
SMTP was early enough and good enough to get widespread adoption. IRC in its original form was never good enough (esp. re: scaling), so services like AIM were able to step in and were good enough and open enough[1] solution while the open source world stuck with IRC for too long because it 'works for us'. So open standards like IRC never got entrenched the way SMTP did in proprietary solutions and by the late 90's the open source world had become pretty weak on developing new Internet standards that would even get widespread adoption in the open source world.
It's worth noting that there's nothing etched in stone to say that SMTP has won indefinitely. Try setting up a purely open source SMTP server on the Internet these days and see who you can talk to. Google has been making it harder and harder[2] to use non-Google clients with their servers. Slowly but surely, business interests seem to be enveloping SMTP with their own walled gardens... it's just taking longer.
[1] Well, sort of. They actually seemed to want to be proprietary about it. But every time they changed the protocol, the apps that worked with it were able to reverse engineer their changes quickly and AOL didn't fight too hard to prevent them from doing so.
[2] Or at least enough of a chore so that most people won't bother trying.
by emteycz on 5/26/22, 3:03 PM
Then we tried Slack but most people didn't come there too much. Today we have a Telegram group and it's so much better UX/UI wise...
by webmaven on 5/26/22, 3:37 PM
I'm not entirely certain why this was, but I suspect that valuing extensibility over interoperability (or the perception of it) was part of it. Optional extensions and interop with hosts that used a different set simply gave hosts/operators too much to think about in terms of deploying the service for users.
by yobbo on 5/26/22, 6:19 PM
In the 90s email addresses along with domain names became part of professional fashion. Everyone wanted @mycompany.com addresses, and that created one of the obstacles for walled garden email. And ISPs often offered email-addresses to subscribers.
"The Microsoft Network" was an attempt at a walled garden that failed.
Hotmail etc had no choice but to support the existing open protocols. Hotmail was just one of many free webmails.
No instant messaging network used "global identities" (or hierarchical identities/addresses like email). The early-mover ICQ was a walled garden, and it died quickly when it became outdated. ICQ logins/identities had no relevance to anything else.
If instant messaging platforms that wished to form a network had existed, then adopting some protocol would have been trivial. Creating new a network of services/operators competing with already an established platform is not trivial.
by tupolef on 5/27/22, 5:00 AM
It's 2022, XMPP is 23 years old, and I still can't use it as a primary solution for my community. There is no client/server environment available that meets the most demanded purposes and runs on all common devices.
In 5 years I have tried to set up an infrastructure 3 times to have a look, I always come back to Matrix (just because I have more hope that it will turn out better in the future).
XMPP is ready to do too many things, that's a good point, but the most common practices that everyone finds on centralized solutions are not implemented on all platforms, clients, and servers.
The server documentation doesn't give any advice on which clients to use, and the clients doesn't advise which server to use. Everyone has their head in the sand.
It's been at least 15 years since it's easy to set up an email infrastructure by yourself, that everyone can use on all their devices.
by bfrog on 5/26/22, 8:26 PM
That might be the biggest contribution matrix has to offer.
The json encoding is large and seemingly under defined. Many things only seem to work with element and the python server.
by LinuxBender on 5/26/22, 3:10 PM
If there were an alternative to SMTP I believe it would be NNTP given it is similar in concept and just introduces threaded messaging which SMTP clients try to mimic now. There are several potential rabbit hole discussions of why moving away from SMTP are highly unlikely to ever occur.
by pvg on 5/26/22, 3:35 PM
by MithrilTuxedo on 5/26/22, 4:22 PM
I thought it's what everyone used for messaging and push notifications. I know Google extended it beyond compatibility around when Hangouts came out and they started doing browser notifications. Apple stopped providing sources for ejabberd in the last few years, so I assume they moved off that for their iChat/messaging backend, but isn't iChat still an XMPP client?
Aren't most SMS messages routed via XMPP?
by PaulHoule on 5/26/22, 3:00 PM
by zokier on 5/26/22, 4:32 PM
by falcolas on 5/26/22, 3:18 PM
by r00fus on 5/26/22, 6:19 PM
Essentially, XMPP's federation goal was antithetical to the vested interests of messaging providers so it was ignored. But the protocol itself was relatively well designed and so formed the basis of a lot of what we use today.
by akrymski on 5/26/22, 7:07 PM
by mro_name on 5/26/22, 6:22 PM
They never managed to promote pleasant onboarding and finally shut it down ('nobody wants such') months before WhatsApp rocketed - with ejabberd.
by jum1p on 5/30/22, 10:30 AM
SMTP was first so one registered accounts with email and not XMPP. Companies send stuff like bills via email.
There were various popular providers -- today Gmail is the most popular one but others like AOL were popular before. It's hard to change one's address.
Sometimes, your ISP gives you an email address.
Email is formal, so job application happens also over email (at least in not very technical jobs).
Probably because email was so important, there were and are still more people who host themselves, including companies (companies with their own xmpp setup are rare).
The rest of email is today (mostly) at Facebook, Twitter or LinkedIn.
by JoelMcCracken on 5/26/22, 4:11 PM
Eventually, for various reasons, Google decided that an open xmpp based system didn't work for them, and they closed it and extended it. This was effectively the end of xmpp as a distributed tech that many people use today.
Interesting, in a lot of ways SMTP is no longer an open system. It is extremely hard to operate a mail server in such a way that your valid emails get delivered correctly and not marked as spam. So for example in the name of fighting spam, it is extremely difficult to participate in this "open" system.
by olliej on 5/26/22, 8:02 PM
I vaguely recall there being something else as well as ICQ but we’re talking back to high school for me so well out of the range of good/clear memory.
I think messenger also came into existence at around the same time as jabber so concurrent development presumably meant completely independent protocol, so it wouldn’t even be subject to embrace/extend/extinguish - this was the height of MS monopoly so presumably it meant essentially “everyone” instantly had messenger?
by smm11 on 5/26/22, 3:50 PM
Same issue with Mastodon, in my opinion. People want to grab something from the Store or Play, sign in, and be where everyone else is.
by hdjjhhvvhga on 5/26/22, 6:44 PM
by 0xbadcafebee on 5/26/22, 3:29 PM
XMPP:
- is designed for chatting and P2P and VoIP over persistent HTTP connections
- requires everyone to be online
- servers have to implement 50 different specs
- works at the frontend and backend
- is designed to send one large message in bulk over a regular short-lived TCP session
- is a store-and-forward messaging system, highly resilient to slow or inconsistent networks and server issues
- is simple and the ecosystem is layered, so the core server doesn't have to implement a dozen specs
- is practically backend-only
by wruza on 5/26/22, 6:10 PM
by usrn on 5/26/22, 3:48 PM
Aren't open protocols and interoperability great?
by badrabbit on 5/26/22, 4:48 PM
by ipaddr on 5/27/22, 3:26 AM
by nikolay on 5/26/22, 5:19 PM
by mike_hearn on 5/26/22, 7:40 PM
SMTP dates from the earliest days of the internet and you don't really see new decentralized/federated protocols after about 1995, which is when IM started to be developed. But that isn't by itself an answer, it just raises the question of why open protocol development mostly stopped in that era.
I think a big reason why was the advancement of networks and hardware made it feasible to actually build and run large internet connected datacenters for the first time. In turn that meant you could run a server farm to which everyone could connect to, without running massive modem banks and the like.
In the early days of the internet that wasn't the case. Servers were machines running in people's offices and the ecosystem was extremely fragmented. There were lots of different kinds of network and email system, SMTP was really a sort of hack to bridge different systems together. The internet was congealing, being patched together out of lots of existing stuff, and big companies mostly didn't care. Networks were often only intermittently connected, the internet was for hobbyists and researchers, not 'serious' companies like AOL or Microsoft, who didn't "get" the internet at all until some famous memo by Bill Gates (they were pushing their own closed network). SMTP reflects this world with its handling of offline servers, lowest-common-denominator simplicity etc.
Once big companies realized the internet mattered and datacenter / networking technology got far enough along, the era of decentralized federated infrastructure died out pretty quick. This is in many ways very sad, but it's also in some ways inevitable. The decentralized cooperative standards-based approach sounds romantic, but it can be described in other much less noble ways e.g. as a form of committee-controlled pure communism in which nobody owns anything.
Phrased that way it's maybe obvious why that model of development has failed: there's no incentive to do anything good. SMTP isn't good. It's like an MVP email protocol. XMPP isn't that good either in many ways; you certainly wouldn't design a protocol that way today. And of course the issue here is not just the protocol but the entire stack - the servers, and most especially the clients. Every federated protocol system always suffers badly from a fragmented and min-viable client landscape, which certainly held back XMPP a lot.
Who is incentivized to make something good? Well, people who have an ownership stake in the results. The prospect of a big payout is what lets you attract teams of the most skilled people who will work crazy hours to build a service people love. It's the capitalist model - wall off the garden and charge an entrance fee. Suddenly finding gardeners isn't a problem anymore, because you can just pay them instead of asking them to volunteer.
The lack of ownership causes all sorts of subtle problems in the federated protocol landscape, beyond the obvious ones like a proliferation of low quality clients written in people's spare time. Probably the worst is that there's little innovation. People become motivated to build a federated/open protocol mostly when they see a successful proprietary product and they'd like to have the benefits of that product without the loss of freedom, dependence on a single vendor, having to pay for it etc. These communities are always going where the puck was, not where it's going now. By the time they finish cloning one cool feature their proprietary competition has, it's too late, that competition already moved ahead and added something else. This really nailed XMPP because networks like MSN were constantly adding features that XMPP couldn't easily respond to, and you see it again with stuff like stickers in Telegram. No open source dev is going to spontaneously decide to add stickers if they haven't seen that feature before, because it'd require recruiting a ton of artists and would seem trivial. Proprietary software companies are much more likely to come up with popular stuff like that. They find it much easier to build multi-disciplinary teams.