from Hacker News

Ask HN: Recommendations for Peer to Peer Frameworks?

by zilchers on 4/8/18, 12:49 PM with 75 comments

I'm looking for peer to peer framework recommendations to build a distributed app. In particular, a framework that supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.

I'm not particular about language (Java, Go, Javascript, etc), since it's for a new project. I found this thread from 2016 (https://news.ycombinator.com/item?id=11145691), it had some interesting recommendations, but quite a few were out of date, and with the big push to distributed apps nowadays, I thought it might be a good time to readdress the topic.

Some of the solutions form the other thread or that I've found:

- Smokesignal (Javascript / Node, not updated in >10 months) https://github.com/marcelklehr/smokesignal

- ZeroTier: https://www.zerotier.com/product-netcon.shtml

- Maki: https://maki.io

- Telehash: http://telehash.org/

- A whole host of blockchain projects, Bitcoin, Ethereum, HyperLedger, Scorex (Scala blockchain)

Bitcoin is an interesting option, it certainly has a large user base and is well tested and secure code, but forking it has it's own set of issues.

  • by 0XAFFE on 4/8/18, 1:55 PM

    Maybe Scuttlebutt? https://www.scuttlebutt.nz/

    Scuttlebutt is advertised as a social network, but in core it is an eternal distributed log. You can pass either public or private messages and assets around.

  • by rauljordan on 4/8/18, 1:52 PM

    Look into libp2p created by the Protocol Labs team (created IPFS). We're thinking of moving to this framework internally at Ethereum.

  • by indescions_2018 on 4/8/18, 2:39 PM

    Awesome List of P2P resources:

    https://github.com/kgryte/awesome-peer-to-peer

    Personal bias tilts to web tech. WebRTC will require you to centrally implement all your peer broker and logic. But take a look at something like Peer5 Video CDN to appreciate scalability:

    https://www.peer5.com/

  • by shpx on 4/8/18, 10:41 PM

    https://urbit.org/docs/

    The idea with urbit (amongst others, it's a big system) is for everyone to have a personal server that's easier to administrate than a linux server. It's been posted a bunch of times https://hn.algolia.com/?query=urbit&sort=byPopularity&prefix...

    I've also heard about https://zeronet.io/ and https://beakerbrowser.com/

  • by petegordon on 4/8/18, 1:57 PM

    I’m curious what your goal is. Do you want the complete immutability of transactions like bitcoin/blockchain; or do you want the peer/seed capability like BitTorrent and IPFS with a DHT (Kademlia)? I’d be interested in seeing a good list of the technology choices that can make up a P2P project if anyone knows of one, please share.

    But you definitely should checkout IPFS and Kademlia.

  • by mcjiggerlog on 4/8/18, 2:30 PM

    Maybe check out Dat[0] - especially useful if you need some kind of distributed data store. The main implementation is in Javascript.

    [0] https://datproject.org/

  • by no_identd on 4/8/18, 3:06 PM

    The 181 pages long GNUnet dissertation very recently got released: https://grothoff.org/christian/habil.pdf

    I'd strongly recommend going for GNUnet, nobody takes me serious when I say this, but it WILL surplant IP eventually.

  • by JamesLeonis on 4/8/18, 7:21 PM

    The WebTorrent Project [1] has a whole suite of tools for decentralized applications based around BitTorrent. It's not a framework per-se, but it breaks up the protocol into libraries for your use.

    If you're interested in Ethereum, it has other non-blockchain technologies that you can use for your apps. It has the Swarm network for storing bits of data, the Whisper network for chat and messaging, as well as Public-Key signing to verify messages. I'm using the latter functionality for a demo to replace password as the security mechanism.

    [1]: https://github.com/webtorrent

  • by jeswin on 4/9/18, 2:03 AM

    Secure Scuttlebutt. It's probably the only one seeing everyday use to get things done. Patchwork, the flagship app is as good as Facebook/Twitter. Scuttlers often use git over SSB, even.

    What SSB gets right and the others don't, is the incentive to mirror or replicate. Virtual currency is a weak incentive in an early stage network - but friendship isn't; you mirror your friends' data.

    The basic API to program the dapp can be seen here - http://scuttlebot.io/ But for more advanced uses see https://youtu.be/f_baWUW4R8Y

  • by alexmorley on 4/8/18, 6:25 PM

    Maybe checkout the dat project https://datproject.org/ which is whats used by BeakerBroswer etc.

  • by DyslexicAtheist on 4/8/18, 1:56 PM

  • by Dowwie on 4/8/18, 1:17 PM

    How about IPFS? OpenBazaar is an IPFS case study. They recently released a revamped v2 built with Go.

  • by povilasb on 4/9/18, 1:12 PM

    Another project you might find useful, that we at https://www.maidsafe.net are actively working on, is https://github.com/maidsafe/crust/.

    It's a generic peer-to-peer networking library written in Rust.

      * it connects two peers together
  * various NAT traversal techniques are implemented: hole punching, UPnP/IGD, etc.
  * it supports both TCP and UDP. In case of UDP it uses uTP: http://www.bittorrent.org/beps/bep_0029.html
  * all messages are encrypted
  * it implements automatic peer discovery on LAN
  * it caches it's previous connections and is able to reuse them in the future
  * etc.

  • by monocasa on 4/8/18, 7:23 PM

    The virtual synchrony protocols don't get enough love. Conceptually, they're sort of like if you performed PAXOS to determine ordering for a distributed finite state machine. Let's you build all sorts of distributed applications where you trust the nodes.

    https://en.wikipedia.org/wiki/Virtual_synchrony

    http://www.jgroups.org/

    https://archive.codeplex.com/?p=vsync

  • by billylo on 4/8/18, 1:26 PM

    I am using https://hypelabs.io/

    Pretty good for small payloads.

  • by wslh on 4/8/18, 2:47 PM

    Re blockchain technologies: while there is an obvious intersection between the world of blockchain and P2P, blockchain technologies are slow because BFT adds many requirements to the game. Basically blockchain technologies use P2P to reach a common state and this could not be necessary in your use case.

    You can take a look at OpenBazaar and Particl for P2P implementations.

  • by sova on 4/9/18, 12:57 AM

    ZeroMQ http://zguide.zeromq.org/py:all ... "In One Hundred Words: ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source."

  • by doomjunky on 4/9/18, 8:51 PM

    Peer Name Resolution Protocol (PNRP)

    - Developed by Microsoft

    - Open Specification

    - DHT-based

    - IPv6 only

    - Windows support since Windows XP SP 3

    - .NET support since .NET Framework 3.5 (System.Net.PeerToPeer)

    [1]: https://en.wikipedia.org/wiki/Peer_Name_Resolution_Protocol

    [2]: https://msdn.microsoft.com/en-us/library/aa371699.aspx "PNRP Namespace Provider API"

    [3]: https://msdn.microsoft.com/en-us/library/bb726971.aspx "Peer Name Resolution Protocol"

  • by hapnin on 4/8/18, 3:25 PM

  • by smaddock on 4/8/18, 4:59 PM

    If you plan on having any collaborative editing features, Yjs [1] is a p2p framework utilizing CRDTs with support for IPFS.

    [1] https://github.com/y-js/yjs

  • by devxpy on 4/9/18, 10:26 AM

    I recently discovered zyre. Its from the same guy that brought us the wonderful zeromq library

    While it does a lot of the connection and communication part, it lacks a way to bypass nat routers.

    Here is a book on zyre http://zguide.zeromq.org/php:chapter8

    I also found this thing called pwnat which bypasses nat routers without the need for UDP hole punching !

    https://github.com/samyk/pwnat

    I feel like if someone made a high level wrapper combining these two, we can have a very good framework.

  • by tribler on 4/8/18, 4:26 PM

    > supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.

    Our audacious 'IPv8' work is now deployed live. Might fit you needs. Includes above features in a low-complexity approach. https://github.com/Tribler/py-ipv8 Our fresh IETF Internet Standard draft of this work: https://tools.ietf.org/html/draft-pouwelse-trustchain-00

  • by tomp2p on 4/9/18, 12:05 AM

    TomP2P (https://tomp2p.net/) is a library handling firewall / UPNP support, however, the focus is academic/experimental.

  • by mayamatrix on 4/9/18, 1:12 AM

    Worth having a look into the Holochain framework:

    https://holochain.org/

    Depending on the application(s) you have in mind.

  • by mempko on 4/8/18, 5:43 PM

    How about something [completely different](http://firestr.com)

  • by martindale on 4/8/18, 3:36 PM

    Maki 0.3 will include components from Fabric: https://fabric.fm — automatically available to anyone upgrading from 0.2. :)

  • by dnomad on 4/8/18, 2:04 PM

    I2P works well enough.

  • by w3news on 4/9/18, 4:42 AM

    Check this great talk: https://youtu.be/izQFMADw70w

  • by cjbprime on 4/8/18, 2:42 PM

    WebTorrent.

  • by adamnemecek on 4/8/18, 1:21 PM

    Look into crdt it’s the shit for collaborative editing (idk if that’s what you ares looking for tho).