From f7109aa0f896a35b460dbfd4b68cf568dcb15a35 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 11:29:11 +0530 Subject: [PATCH 01/35] ccreated new folder hp-docs. --- hp-docs/index.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 hp-docs/index.md diff --git a/hp-docs/index.md b/hp-docs/index.md new file mode 100644 index 0000000..e69de29 From a1e9a3cb4196185b3164317c98a25990a82da7c4 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 11:39:41 +0530 Subject: [PATCH 02/35] Add QuickStart & Overview --- hp-docs/README.md | 119 +++++++ hp-docs/index.md | 0 hp-docs/quick-start.md | 729 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 848 insertions(+) create mode 100644 hp-docs/README.md delete mode 100644 hp-docs/index.md create mode 100644 hp-docs/quick-start.md diff --git a/hp-docs/README.md b/hp-docs/README.md new file mode 100644 index 0000000..ca834ed --- /dev/null +++ b/hp-docs/README.md @@ -0,0 +1,119 @@ +--- +description: Welcome to the Holepunch docs! 👋 +--- + +# Pear by Holepunch + +> Pear loads applications remotely from peers and allows anyone to create and share applications with peers. + +Pear by Holepunch is a combined Runtime, Software Development Kit (SDK) and Peer-to-Peer (P2P) Deployment tool. +Pear makes it possible to build, share and extend P2P applications using common web and mobile technology stacks. +Herein is everything needed to create unstoppable, zero-infrastructure P2P applications for desktop, terminal and mobile. + +Welcome to the Internet of Peers +Holepunch, the P2P Company + +## Platform +* [Pear Command line](pear/01-command-line.md) + +## Building blocks + +The Holepunch Ecosystem is constructed utilizing the following structural components. + +1. [hypercore.md](building-blocks/hypercore.md "mention"): A distributed, secure append-only log is a useful tool for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. +2. [hyperbee.md](building-blocks/hyperbee.md "mention"): An append-only B-tree running on a Hypercore. It provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. +3. [hyperdrive.md](building-blocks/hyperdrive.md "mention"): A secure, real-time distributed file system that simplifies peer-to-peer (P2P) file sharing. It provides an efficient way to store and access data across multiple connected devices in a decentralized manner. +4. [autobase.md](building-blocks/autobase.md "mention"): An experimental module that's used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. +5. [hyperdht.md](building-blocks/hyperdht.md "mention"): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. +6. [hyperswarm.md](building-blocks/hyperswarm.md "mention"): A high-level API for finding and connecting to peers who are interested in a "topic." + +## Helpers + +Helper modules can be used together with the building blocks to create cutting-edge peer-to-peer tools and applications. + +1. [corestore.md](helpers/corestore.md "mention"): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. +2. [localdrive.md](helpers/localdrive.md "mention"): A file system interoperable with Hyperdrive. +3. [mirrordrive.md](helpers/mirrordrive.md "mention"): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md "mention") or a [localdrive.md](helpers/localdrive.md "mention") into another one. +4. [secretstream.md](helpers/secretstream.md "mention"): SecretStream is used to securely create connections between two peers in Hyperswarm. +5. [compact-encoding.md](helpers/compact-encoding.md "mention"): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. +6. [protomux.md](helpers/protomux.md "mention"): Multiplex multiple message oriented protocols over a stream. + +## Tools built using Holepunch + +
hypershell.mdA CLI to create and connect to P2P E2E encrypted shells.hypershell.md
hypertele.mdA swiss-knife proxy powered by hyperdht.md.hypertele.md
hyperbeam.mdA one-to-one and end-to-end encrypted internet pipe.hyperbeam.md
hyperssh.mdA CLI to run SSH over the hyperdht.md.hyperssh.md
drives.mdCLI to download, seed, and mirror a hyperdrive.md or a localdrive.md.drives.md
+ +{% hint style="success" %} +These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). +{% endhint %} + +## Applications built using Holepunch + +* [keet.io.md](apps/keet.io.md "mention")**:** A peer-to-peer chat and video-conferencing application with end-to-end encryption. + +## What's new? + +If you're already familiar with [hypercore.md](building-blocks/hypercore.md "mention"), [hyperswarm.md](building-blocks/hyperswarm.md "mention"), and the rest of the modules here, you might be wondering what's changed recently. In short: a lot! Here's a brief breakdown of some of the highlights. + +### Better Building Blocks + +We've decided to focus entirely on making our core building blocks as easy to use, as fast, and as reliable as possible. The goal is to give developers all the essential pieces needed to make killer P2P apps, without being unnecessarily opinionated and without introducing operational complexity. + +We'd previously introduced the Hyperspace daemon and `hyp` CLI tool as core modules, but in practice found that they don't fit cleanly into this vision. We feel it's better to keep a tight focus on the fundamental building blocks so that others can build similar higher-level tools with zero friction. Given that, they are both now considered deprecated. + +### Hypercore v10 + +* The [`session`](building-blocks/hypercore.md#core.session-options) and [`snapshot`](building-blocks/hypercore.md#core.snapshot-options) methods for providing multiple views over the same underlying Hypercore, which simplifies resource management. +* A [`truncate`](building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](building-blocks/autobase.md "mention"), as described below. +* An improved fork detection in the replication protocol, to improve resilience. +* Optional on-disk encryption for blocks (in addition to the existing transport encryption). +* The storage layer now uses a write-ahead log to better ensure that power loss or unexpected shutdown cannot lead to data corruption. + +### Hyperswarm v4 + +* An improved UDP holepunching algorithm that uses arbitrary DHT nodes (optionally selected by the connecting peers) to proxy necessary metadata while being maximally privacy-preserving. +* A custom-built transport protocol, [UDX](https://github.com/hyperswarm/libudx), that takes advantage of the holepunching algorithm to avoid unnecessary overhead (it doesn't include handshaking since holepunching takes care of that, for example). It's blazing fast. +* A simplified DHT API that closely resembles NodeJS's `net` module, but using public keys instead of IPs. + +### Hyperdrive + +* Uses Hyperbee internally for storing file metadata +* Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. +* Auxiliary tools, [localdrive.md](helpers/localdrive.md "mention") and [mirrordrive.md](helpers/mirrordrive.md "mention"), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. + +### Autobase (experimental) + +Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows you to turn many Hypercores, owned by many different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view you see on screen. + +Since Autobase's output shares the familiar Hypercore API, you can plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. + +{% hint style="warning" %} +Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! +{% endhint %} + +## Stability indexing + +Throughout the documentation, indications of a module's stability are provided. Some modules are well-established and used widely, making them highly unlikely to ever change. Other modules may be new, experimental, or known to have risks associated with their use. + +The following stability indices have been used: + +| Stability | Description | +| :----------------------------------------------------------: | :---------------------------------------------------------: | +| **stable** | Unlikely to change or be removed in the foreseeable future. | +| **experimental** | New, untested, or have known issues. | +| **deprecated** | Being removed or replaced in the future. | +| **unstable** | May change or be removed without warning. | + +#### Stability overview + +| Module | Stability | +| -------------------------------------------------------- | :----------------------------------------------------------: | +| [hypercore.md](building-blocks/hypercore.md "mention") | **stable** | +| [hyperbee.md](building-blocks/hyperbee.md "mention") | **stable** | +| [hyperdrive.md](building-blocks/hyperdrive.md "mention") | **stable** | +| [autobase.md](building-blocks/autobase.md "mention") | **experimental** | +| [hyperswarm.md](building-blocks/hyperswarm.md "mention") | **stable** | +| [hyperdht.md](building-blocks/hyperdht.md "mention") | **stable** | + +{% hint style="warning" %} +Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. +{% endhint %} diff --git a/hp-docs/index.md b/hp-docs/index.md deleted file mode 100644 index e69de29..0000000 diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md new file mode 100644 index 0000000..2b7c71b --- /dev/null +++ b/hp-docs/quick-start.md @@ -0,0 +1,729 @@ +--- +description: A primer on Holepunch's modular building blocks +--- + +# Quick start + +* [Setup](quick-start.md#setup) +* [Hyperswarm's DHT: Connecting Two Peers by Key](quick-start.md#hyperswarms-dht-connecting-two-peers-by-key) +* [Hyperswarm: Connecting to Many Peers by Topic](quick-start.md#hyperswarm-connecting-to-many-peers-by-topic) +* [Hypercore: The Basics](quick-start.md#hypercore-the-basics) +* [Corestore: Working with Many Hypercores](quick-start.md#corestore-working-with-many-hypercores) +* [Hyperbee: Sharing Append-Only Databases](quick-start.md#hyperbee-sharing-append-only-databases) +* [Hyperdrive: A Full P2P Filesystem](quick-start.md#hyperdrive-a-full-p2p-filesystem) + +### Setup + +{% hint style="warning" %} +Before we begin, make sure you are running Node v16 or greater. +{% endhint %} + +To follow along with this quick-start guide, download the [Holepunch examples repo](https://github.com/holepunchto/examples) + +```bash +git clone https://github.com/holepunchto/examples +``` + +Install the required dependencies + +```bash +cd examples +npm install +``` + +Install the required dependencies + +{% code overflow="wrap" %} + +```bash +npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a debounceify graceful-goodbye --save +``` + +{% endcode %} + +{% hint style="info" %} +Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. +{% endhint %} + +### Hyperswarm's DHT: Connecting Two Peers by Key + +[Hyperswarm](building-blocks/hyperswarm.md) allows you to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. + +In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. + +{% hint style="info" %} +Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. + +For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. +{% endhint %} + +Let's use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. + +`server.mjs` will create a key pair and then start a server that will listen on the generated key pair. The public key is logged into the console. Copy it for instantiating the client. + +{% code title="server.mjs" %} + +```javascript +import DHT from 'hyperdht' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +const dht = new DHT() + +// This keypair is your peer identifier in the DHT +const keyPair = DHT.keyPair() + +const server = dht.createServer(conn => { + console.log('got connection!') + process.stdin.pipe(conn).pipe(process.stdout) +}) + +server.listen(keyPair).then(() => { + console.log('listening on:', b4a.toString(keyPair.publicKey, 'hex')) +}) + +// Unnannounce the public key before exiting the process +// (This is not a requirement, but it helps avoid DHT pollution) +goodbye(() => server.close()) +``` + +{% endcode %} + +`client.mjs` will spin up a client, and the public key copied earlier must be supplied as a command line argument for connecting to the server. The client process will log `got connection` into the console when it connects to the server. + +Once it's connected, try typing in both terminals! + +{% code title="client.mjs" %} + +``` javascript +import DHT from 'hyperdht' +import b4a from 'b4a' + +console.log('Connecting to:', process.argv[2]) +const publicKey = b4a.from(process.argv[2], 'hex') + +const dht = new DHT() +const conn = dht.connect(publicKey) +conn.once('open', () => console.log('got connection!')) + +process.stdin.pipe(conn).pipe(process.stdout) +``` + +{% endcode %} + +### Hyperswarm: Connecting to Many Peers by Topic + +In the above example, we connected two peers directly using the first peer's public key. Often, you'll want to discover peers swarming a common topic, and connect to as many of them as you can. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. + +The [Hyperswarm](building-blocks/hyperswarm.md) module provides a higher-level interface over the underlying DHT, abstracting away the mechanics of establishing and maintaining connections. Instead, you 'join' topics, and the swarm discovers peers automatically. It also handles reconnections in the event of failures. + +In the previous example, we needed to explicitly indicate which peer was the server and which was the client. Here, we create two peers, have them join a common topic, and let the swarm deal with connections. + +This example consists of a single file, `peer.mjs`. In one terminal, type `node peer.mjs`, it will display the topic. Copy/paste that topic into N additional terminals with `node peer.mjs (topic)`. Each peer will log information about the other connected peers. + +Start typing into any terminal you like, and it will be broadcast to all connected peers! + +{% code title="peer.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import goodbye from 'graceful-goodbye' +import crypto from 'hypercore-crypto' +import b4a from 'b4a' + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// Keep track of all connections and console.log incoming data +const conns = [] +swarm.on('connection', conn => { + const name = b4a.toString(conn.remotePublicKey, 'hex') + console.log('* got a connection from:', name, '*') + conns.push(conn) + conn.once('close', () => conns.splice(conns.indexOf(conn), 1)) + conn.on('data', data => console.log(`${name}: ${data}`)) +}) + +// Broadcast stdin to all connections +process.stdin.on('data', d => { + for (const conn of conns) { + conn.write(d) + } +}) + +// Join a common topic +const topic = process.argv[2] ? b4a.from(process.argv[2], 'hex') : crypto.randomBytes(32) +const discovery = swarm.join(topic, { client: true, server: true }) + +// The flushed promise will resolve when the topic has been fully announced to the DHT +discovery.flushed().then(() => { + console.log('joined topic:', b4a.toString(topic, 'hex')) +}) +``` + +{% endcode %} + +### Hypercore: The Basics + +In the Hyperswarm examples, peers can exchange chat messages so long as both are online at the same time and directly connected, and those messages are not persistent (they will be lost if the recipient is offline). Hypercore fixes all of these problems. + +[hypercore.md](building-blocks/hypercore.md "mention") is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications. + +Let's extend the ephemeral chat example above but using Hypercore to add many significant new features: + +1. **Persistence**: The owner of the Hypercore can add messages at any time, and they'll be persisted to disk. Whenever they come online, readers can replicate these messages over Hyperswarm. +2. **Many Readers:** New messages added to the Hypercore will be broadcast to interested readers. The owner gives each reader a reading capability (`core.key`) and a corresponding discovery key (`core.discoveryKey`). The former is used to authorize the reader, ensuring that they have permission to read messages, and the latter is used to discover the owner (and other readers) on the swarm. + +The following example consists of two files: `reader.mjs` and `writer.mjs`. When these two files are executed (run using node), two peers are created and connected. A Hypercore is used to store the data entered into the command line. + +`writer.mjs` stores the data entered into the command line to the Hypercore instance. The Hypercore instance is replicated with other peers using Hyperswarm. + +{% code title="writer.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import Hypercore from 'hypercore' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +const core = new Hypercore('./writer-storage') + +// core.key and core.discoveryKey will only be set after core.ready resolves +await core.ready() +console.log('hypercore key:', b4a.toString(core.key, 'hex')) + +// Append all stdin data as separate blocks to the core +process.stdin.on('data', data => core.append(data)) + +// core.discoveryKey is *not* a read capability for the core +// It's only used to discover other peers who *might* have the core +swarm.join(core.discoveryKey) +swarm.on('connection', conn => core.replicate(conn)) +``` + +{% endcode %} + +`reader.mjs` uses Hyperswarm to connect to the previously initiated peer and synchronize the local Hypercore instance with the Hypercore instance of the writer. + +{% code title="reader.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import Hypercore from 'hypercore' +import goodbye from 'graceful-goodbye' + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +const core = new Hypercore('./reader-storage', process.argv[2]) +await core.ready() + +const foundPeers = core.findingPeers() +swarm.join(core.discoveryKey) +swarm.on('connection', conn => core.replicate(conn)) + +// swarm.flush() will wait until *all* discoverable peers have been connected to +// It might take a while, so don't await it +// Instead, use core.findingPeers() to mark when the discovery process is completed +swarm.flush().then(() => foundPeers()) + +// This won't resolve until either +// a) the first peer is found +// or b) no peers could be found +await core.update() + +let position = core.length +console.log(`Skipping ${core.length} earlier blocks...`) +for await (const block of core.createReadStream({ start: core.length, live: true })) { + console.log(`Block ${position++}: ${block}`) +} +``` + +{% endcode %} + +### Corestore: Working with Many Hypercores + +An append-only log is powerful on its own, but it's most useful as a building block for constructing larger data structures, such as databases or filesystems. When building these data structures, you'll often need many cores, each with different responsibilities. For example, Hyperdrive uses one core to store file metadata and another to store file contents. + +[corestore.md](helpers/corestore.md "mention") is a Hypercore factory that makes it easier to manage large collections of named Hypercores. Below you'll find a simple example that demonstrates a pattern we often use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. + +This example consists of two files: `writer.mjs` and `reader.mjs`. In the previous example, we replicated only a single Hypercore instance. But in this example, we will replicate a single Corestore instance, which will internally manage the replication of a collection of Hypercores. + +The file `writer.mjs` uses a Corestore instance to create three Hypercores, which are then replicated with other peers using Hyperswarm. The keys for the second and third cores are stored in the first core (the first core 'bootstraps' the system). Messages entered into the command line are written into the second and third cores, depending on the length of the message. To execute `reader.mjs`, you should copy the main core key logged into the command line. + +{% code title="writer.mjs" %} + +```javascript +import Corestore from 'corestore' +import Hyperswarm from 'hyperswarm' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +const store = new Corestore('./writer-storage') +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// A name is a purely-local, and maps to a key pair. It's not visible to readers. +// Since a name always corresponds to a key pair, these are all writable +const core1 = store.get({ name: 'core-1', valueEncoding: 'json' }) +const core2 = store.get({ name: 'core-2' }) +const core3 = store.get({ name: 'core-3' }) +await Promise.all([core1.ready(), core2.ready(), core3.ready()]) + +console.log('main core key:', b4a.toString(core1.key, 'hex')) + +// Here we'll only join the swarm with the core1's discovery key +// We don't need to announce core2 and core3, because they'll replicated with core1 +swarm.join(core1.discoveryKey) + +// Corestore replication internally manages to replicate every loaded core +// Corestore *does not* exchange keys (read capabilities) during replication. +swarm.on('connection', conn => store.replicate(conn)) + +// Since Corestore does not exchange keys, they need to be exchanged elsewhere. +// Here, we'll record the other keys in the first block of core1. +if (core1.length === 0) { + await core1.append({ + otherKeys: [core2, core3].map(core => b4a.toString(core.key, 'hex')) + }) +} + +// Record all short messages in core2, and all long ones in core3 +process.stdin.on('data', data => { + if (data.length < 5) { + console.log('appending short data to core2') + core2.append(data) + } else { + console.log('appending long data to core3') + core3.append(data) + } +}) +``` + +{% endcode %} + +`reader.mjs` connects to the previous peer with Hyperswarm and replicates the local Corestore instance to receive the data from it. This requires the copied key to be supplied as an argument when executing the file, which will then be used to create a core with the same public key as the other peer (i.e., the same discovery key for both the reader and writer peers). + +{% code title="reader.mjs" %} + +```javascript +import Corestore from 'corestore' +import Hyperswarm from 'hyperswarm' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +// pass the key as a command line argument +const key = b4a.from(process.argv[2], 'hex') + +// creation of a Corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of corestore instance on every connection +swarm.on('connection', conn => store.replicate(conn)) + +// creation/getting of a hypercore instance using the key passed +const core = store.get({ key, valueEncoding: 'json' }) +// wait till all the properties of the hypercore instance are initialized +await core.ready() + +const foundPeers = store.findingPeers() +// join a topic +swarm.join(core.discoveryKey) +swarm.flush().then(() => foundPeers()) + +// update the meta-data of the hypercore instance +await core.update() + +if (core.length === 0) { + console.log('Could not connect to the writer peer') + process.exit(1) +} + +// getting cores using the keys stored in the first block of main core +const { otherKeys } = await core.get(0) +for (const key of otherKeys) { + const core = store.get({ key: b4a.from(key, 'hex') }) + // on every append to the hypercore, + // download the latest block of the core and log it to the console + core.on('append', () => { + const seq = core.length - 1 + core.get(seq).then(block => { + console.log(`Block ${seq} in Core ${key}: ${block}`) + }) + }) +} +``` + +{% endcode %} + +### Hyperbee: Sharing Append-Only Databases + +[hyperbee.md](building-blocks/hyperbee.md "mention") is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. + +The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-reader.mjs`. + +`writer.mjs` stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and download the dictionary data into their local Hyperbee instances. + +{% hint style="info" %} +Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. +{% endhint %} + +{% code title="writer.mjs" %} + +```javascript +import fs from 'fs' +import zlib from 'zlib' + +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import Hyperbee from 'hyperbee' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +// create a corestore instance with the given location +const store = new Corestore('./writer-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of corestore instance +swarm.on('connection', conn => store.replicate(conn)) + +// creation of Hypercore instance (if not already created) +const core = store.get({ name: 'my-bee-core' }) + +// creation of Hyperbee instance using the core instance +const bee = new Hyperbee(core, { + keyEncoding: 'utf-8', + valueEncoding: 'utf-8' +}) + +// wait till all the properties of the hypercore are initialized +await core.ready() + +// join a topic +const discovery = swarm.join(core.discoveryKey) + +// Only display the key once the Hyperbee has been announced to the DHT +discovery.flushed().then(() => { + console.log('bee key:', b4a.toString(core.key, 'hex')) +}) + +// Only import the dictionary the first time this script is executed +// The first block will always be the Hyperbee header block +if (core.length <= 1) { + console.log('importing dictionary...') + const dict = await loadDictionary() + const batch = bee.batch() + for (const { key, value } of dict) { + await batch.put(key, value) + } + await batch.flush() +} else { + // Otherwise just seed the previously-imported dictionary + console.log('seeding dictionary...') +} + +async function loadDictionary() { + const compressed = await fs.promises.readFile('./dict.json.gz') + return new Promise((resolve, reject) => { + // unzip the compressed file and return the content + zlib.unzip(compressed, (err, dict) => { + if (err) return reject(err) + return resolve(JSON.parse(b4a.toString(dict))) + }) + }) +} +``` + +{% endcode %} + +`bee-reader.mjs` creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as the above file. On every word entered in the command line, it will download the respective data to the local Hyperbee instance. + +Try looking at how much disk space the `reader-storage` directory is using after each query. You'll notice that it's significantly smaller than `writer-storage`! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call **sparse downloading.** + +{% code title="bee-reader.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import Hyperbee from 'hyperbee' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +// creation of a corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of the corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create or get the hypercore using the public key supplied as command-line argument +const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) + +// create a hyperbee instance using the hypercore instance +const bee = new Hyperbee(core, { + keyEncoding: 'utf-8', + valueEncoding: 'utf-8' +}) + +// wait till the hypercore properties to be intialized +await core.ready() + +// logging the public key of the hypercore instance +console.log('core key here is:', core.key.toString('hex')) + +// Attempt to connect to peers +swarm.join(core.discoveryKey) + +// Do a single Hyperbee.get for every line of stdin data +// Each `get` will only download the blocks necessary to satisfy the query +process.stdin.setEncoding('utf-8') +process.stdin.on('data', data => { + const word = data.trim() + if (!word.length) return + bee.get(word).then(node => { + if (!node || !node.value) console.log(`No dictionary entry for ${data}`) + else console.log(`${data} -> ${node.value}`) + }, err => console.error(err)) +}) +``` + +{% endcode %} + +Importantly, a Hyperbee is **just** a Hypercore, where the tree nodes are stored as Hypercore blocks. Let's examine the Hyperbee as if it were just a Hypercore and log out a few blocks. + +`core-reader.mjs` will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which we can easily import and use. + +{% code title="core-reader.mjs" %} + +```javascript +import Hypercore from 'hypercore' +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +import { Node } from 'hyperbee/lib/messages.js' + +// creation of a corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of the corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create or get the hypercore using the public key supplied as command-line argument +const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) +// wait till the properties of the hypercore instance are initialized +await core.ready() + +const foundPeers = store.findingPeers() +// join a topic +swarm.join(core.discoveryKey) +swarm.flush().then(() => foundPeers()) + +// update the meta-data information of the hypercore instance +await core.update() + +const seq = core.length - 1 +const lastBlock = await core.get(core.length - 1) + +// print the information about the last block or the latest block of the hypercore instance +console.log(`Raw Block ${seq}:`, lastBlock) +console.log(`Decoded Block ${seq}`, Node.decode(lastBlock)) +``` + +{% endcode %} + +### Hyperdrive: A Full P2P Filesystem + +[hyperdrive.md](building-blocks/hyperdrive.md "mention") is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. + +Let's mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, we'll use two additional tools: [mirrordrive.md](helpers/mirrordrive.md "mention") and [localdrive.md](helpers/localdrive.md "mention"), which handle all interactions between Hyperdrives and the local filesystem. + +This example consists of three files: `writer.mjs`, `drive-reader.mjs` and `bee-reader.mjs`. + +`writer.mjs` creates a local drive instance for a local directory and then mirrors the local drive into the Hyperdrive instance. The store used to create the Hyperdrive instance is replicated using Hyperswarm to make the data of Hyperdrive accessible to other peers. Copy the drive key logged into the command line for the `reader.mjs` execution. + +{% code title="writer.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import Hyperdrive from 'hyperdrive' +import Localdrive from 'localdrive' +import Corestore from 'corestore' +import goodbye from 'graceful-goodbye' +import debounce from 'debounceify' +import b4a from 'b4a' + +// create a Corestore instance +const store = new Corestore('./writer-storage') +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of the corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// A local drive provides a Hyperdrive interface to a local directory +const local = new Localdrive('./writer-dir') + +// A Hyperdrive takes a Corestore because it needs to create many cores +// One for a file metadata Hyperbee, and one for a content Hypercore +const drive = new Hyperdrive(store) + +// wait till the properties of the hyperdrive instance are initialized +await drive.ready() + +// Import changes from the local drive into the Hyperdrive +const mirror = debounce(mirrorDrive) + +const discovery = swarm.join(drive.discoveryKey) +await discovery.flushed() + +console.log('drive key:', b4a.toString(drive.key, 'hex')) + +// start the mirroring process (i.e copying) of content from writer-dir to the drive +// whenever something is entered (other than '/n' or Enter )in the command-line +process.stdin.setEncoding('utf-8') +process.stdin.on('data', (d) => { + if (!d.match('\n')) return + mirror() +}) + +// this function copies the contents from writer-dir directory to the drive +async function mirrorDrive () { + console.log('started mirroring changes from \'./writer-dir\' into the drive...') + const mirror = local.mirror(drive) + await mirror.done() + console.log('finished mirroring:', mirror.count) +} +``` + +{% endcode %} + +`drive-reader.mjs` creates a local drive instance for a local directory and then mirrors the contents of the local Hyperdrive instance into the local drive instance (which will write the contents to the local directory). + +Try running `node drive-reader.mjs (key-from-above)`, then add/remove/modify files inside `writer-dir` then press `Enter` in the writer's terminal (to import the local changes into the writer's drive). You'll see all new changes mirror into `reader-dir`. + +{% code title="drive-reader.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import Hyperdrive from 'hyperdrive' +import Localdrive from 'localdrive' +import Corestore from 'corestore' +import goodbye from 'graceful-goodbye' +import debounce from 'debounceify' +import b4a from 'b4a' + +// create a Corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of store on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create a local copy of the remote drive +const local = new Localdrive('./reader-dir') + +// create a hyperdrive using the public key passed as a command-line argument +const drive = new Hyperdrive(store, b4a.from(process.argv[2], 'hex')) + +// wait till all the properties of the drive are initialized +await drive.ready() + +const mirror = debounce(mirrorDrive) + +// call the mirror function whenever content gets appended +// to the Hypercore instance of the hyperdrive +drive.core.on('append', mirror) + +const foundPeers = store.findingPeers() + +// join a topic +swarm.join(drive.discoveryKey, { client: true, server: false }) +swarm.flush().then(() => foundPeers()) + +// start the mirroring process (i.e copying the contents from remote drive to local dir) +mirror() + +async function mirrorDrive () { + console.log('started mirroring remote drive into \'./reader-dir\'...') + const mirror = drive.mirror(local) + await mirror.done() + console.log('finished mirroring:', mirror.count) +} +``` + +{% endcode %} + +Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee (which is **just** a Hypercore). Let's inspect the Hyperdrive as though it were a Hyperbee, and log out some file metadata. + +`bee-reader.mjs` creates a Hyperbee instance using the Hypercore instance created with the copied public key. Every time the Hyperbee is updated (an `append` event is emitted on the underlying Hypercore), all file metadata nodes will be logged out. + +Try adding or removing a few files from the writer's data directory, then pressing `Enter` in the writer's terminal to mirror the changes. + +{% code title="bee-reader.mjs" %} + +```javascript +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import Hyperbee from 'hyperbee' +import goodbye from 'graceful-goodbye' +import debounce from 'debounceify' +import b4a from 'b4a' + +// create a Corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replicate corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create/get the hypercore instance using the public key supplied as command-line arg +const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) + +// create a hyperbee instance using the hypercore instance +const bee = new Hyperbee(core, { + keyEncoding: 'utf-8', + valueEncoding: 'json' +}) + +// wait till the properties of the hypercore instance are initialized +await core.ready() + +const foundPeers = store.findingPeers() +swarm.join(core.discoveryKey) +swarm.flush().then(() => foundPeers()) + +// execute the listBee function whenever the data is appended to the underlying hypercore +core.on('append', listBee) + +listBee() + +// listBee function will list the key-value pairs present in the hyperbee instance +async function listBee () { + console.log('\n***************') + console.log('hyperbee contents are now:') + for await (const node of bee.createReadStream()) { + console.log(' ', node.key, '->', node.value) + } +} +``` + +{% endcode %} From 48ec9e221297e5ad5af70515ad70f8bf9b4f8e4c Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 12:35:40 +0530 Subject: [PATCH 03/35] Update ReadMe --- hp-docs/README.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index ca834ed..7cc4eb9 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -1,5 +1,5 @@ --- -description: Welcome to the Holepunch docs! 👋 +description: Welcome to the Pear docs! 👋 --- # Pear by Holepunch @@ -42,9 +42,8 @@ Helper modules can be used together with the building blocks to create cutting-e
hypershell.mdA CLI to create and connect to P2P E2E encrypted shells.hypershell.md
hypertele.mdA swiss-knife proxy powered by hyperdht.md.hypertele.md
hyperbeam.mdA one-to-one and end-to-end encrypted internet pipe.hyperbeam.md
hyperssh.mdA CLI to run SSH over the hyperdht.md.hyperssh.md
drives.mdCLI to download, seed, and mirror a hyperdrive.md or a localdrive.md.drives.md
-{% hint style="success" %} -These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). -{% endhint %} + These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). + ## Applications built using Holepunch @@ -86,9 +85,8 @@ Hypercores are single-writer data structures, but collaboration is crucial. [aut Since Autobase's output shares the familiar Hypercore API, you can plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. -{% hint style="warning" %} -Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! -{% endhint %} + Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! + ## Stability indexing @@ -114,6 +112,5 @@ The following stability indices have been used: | [hyperswarm.md](building-blocks/hyperswarm.md "mention") | **stable** | | [hyperdht.md](building-blocks/hyperdht.md "mention") | **stable** | -{% hint style="warning" %} -Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. -{% endhint %} + + Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. From e26e1c675d96fef11d7a026ce737394db9968120 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 13:24:26 +0530 Subject: [PATCH 04/35] improved grammar as discussed in the call. --- hp-docs/README.md | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 7cc4eb9..f701d7e 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -6,12 +6,11 @@ description: Welcome to the Pear docs! 👋 > Pear loads applications remotely from peers and allows anyone to create and share applications with peers. -Pear by Holepunch is a combined Runtime, Software Development Kit (SDK) and Peer-to-Peer (P2P) Deployment tool. -Pear makes it possible to build, share and extend P2P applications using common web and mobile technology stacks. -Herein is everything needed to create unstoppable, zero-infrastructure P2P applications for desktop, terminal and mobile. +Pear by Holepunch is a combined runtime, Software Development Kit (SDK) and Peer-to-Peer (P2P) Deployment tool. +Pear makes it possible to build, share, and extend P2P applications using common web and mobile technology stacks. +Pears enables to create unstoppable, zero-infrastructure P2P applications for desktop, terminal and mobile. -Welcome to the Internet of Peers -Holepunch, the P2P Company +Welcome to the Internet of Pears-Holepunch, the P2P Company! ## Platform * [Pear Command line](pear/01-command-line.md) @@ -20,10 +19,10 @@ Holepunch, the P2P Company The Holepunch Ecosystem is constructed utilizing the following structural components. -1. [hypercore.md](building-blocks/hypercore.md "mention"): A distributed, secure append-only log is a useful tool for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. -2. [hyperbee.md](building-blocks/hyperbee.md "mention"): An append-only B-tree running on a Hypercore. It provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. -3. [hyperdrive.md](building-blocks/hyperdrive.md "mention"): A secure, real-time distributed file system that simplifies peer-to-peer (P2P) file sharing. It provides an efficient way to store and access data across multiple connected devices in a decentralized manner. -4. [autobase.md](building-blocks/autobase.md "mention"): An experimental module that's used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. +1. [hypercore.md](building-blocks/hypercore.md "mention"): A distributed, secure append-only log is a tool for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. +2. [hyperbee.md](building-blocks/hyperbee.md "mention"): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. +3. [hyperdrive.md](building-blocks/hyperdrive.md "mention"): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. +4. [autobase.md](building-blocks/autobase.md "mention"): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. 5. [hyperdht.md](building-blocks/hyperdht.md "mention"): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. 6. [hyperswarm.md](building-blocks/hyperswarm.md "mention"): A high-level API for finding and connecting to peers who are interested in a "topic." @@ -35,7 +34,7 @@ Helper modules can be used together with the building blocks to create cutting-e 2. [localdrive.md](helpers/localdrive.md "mention"): A file system interoperable with Hyperdrive. 3. [mirrordrive.md](helpers/mirrordrive.md "mention"): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md "mention") or a [localdrive.md](helpers/localdrive.md "mention") into another one. 4. [secretstream.md](helpers/secretstream.md "mention"): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md "mention"): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. +5. [compact-encoding.md](helpers/compact-encoding.md "mention"): A series of binary encoding schemes for building fast and small parsers and serializers. Pears use this in Keet to store chat messages and in Hypercore's replication protocol. 6. [protomux.md](helpers/protomux.md "mention"): Multiplex multiple message oriented protocols over a stream. ## Tools built using Holepunch @@ -51,13 +50,13 @@ Helper modules can be used together with the building blocks to create cutting-e ## What's new? -If you're already familiar with [hypercore.md](building-blocks/hypercore.md "mention"), [hyperswarm.md](building-blocks/hyperswarm.md "mention"), and the rest of the modules here, you might be wondering what's changed recently. In short: a lot! Here's a brief breakdown of some of the highlights. +Pears is continuously introucing new enhancements and following are the few highlights. ### Better Building Blocks -We've decided to focus entirely on making our core building blocks as easy to use, as fast, and as reliable as possible. The goal is to give developers all the essential pieces needed to make killer P2P apps, without being unnecessarily opinionated and without introducing operational complexity. +Pears focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being unnecessarily opinionated and without any operational complexity. -We'd previously introduced the Hyperspace daemon and `hyp` CLI tool as core modules, but in practice found that they don't fit cleanly into this vision. We feel it's better to keep a tight focus on the fundamental building blocks so that others can build similar higher-level tools with zero friction. Given that, they are both now considered deprecated. +The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. ### Hypercore v10 @@ -65,7 +64,7 @@ We'd previously introduced the Hyperspace daemon and `hyp` CLI tool as core modu * A [`truncate`](building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](building-blocks/autobase.md "mention"), as described below. * An improved fork detection in the replication protocol, to improve resilience. * Optional on-disk encryption for blocks (in addition to the existing transport encryption). -* The storage layer now uses a write-ahead log to better ensure that power loss or unexpected shutdown cannot lead to data corruption. +* The storage layer now uses a write-ahead log to ensure that power loss or unexpected shutdown cannot lead to data corruption. ### Hyperswarm v4 @@ -77,15 +76,21 @@ We'd previously introduced the Hyperspace daemon and `hyp` CLI tool as core modu * Uses Hyperbee internally for storing file metadata * Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. -* Auxiliary tools, [localdrive.md](helpers/localdrive.md "mention") and [mirrordrive.md](helpers/mirrordrive.md "mention"), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. +* Auxiliary tools, [localdrive.md](helpers/localdrive.md "mention") and [mirrordrive.md](helpers/mirrordrive.md "mention"), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem when deploying Keet. ### Autobase (experimental) -Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows you to turn many Hypercores, owned by many different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view you see on screen. +Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view to see on screen. -Since Autobase's output shares the familiar Hypercore API, you can plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. +As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. +<<<<<<< HEAD +{% hint style="warning" %} +Autobase is still experimental and is likely to change significantly in the near future. Give it a shot! +{% endhint %} +======= Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! +>>>>>>> ea20f22852e518505a9bc5035bfc5ee1c67373ee ## Stability indexing From 9d336ae95520b46d427b67e2f5ddfb3fe5e117e7 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 13:50:39 +0530 Subject: [PATCH 05/35] Add emoji --- hp-docs/quick-start.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index 2b7c71b..366e73c 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -14,9 +14,8 @@ description: A primer on Holepunch's modular building blocks ### Setup -{% hint style="warning" %} -Before we begin, make sure you are running Node v16 or greater. -{% endhint %} +> ⚠️ Before we begin, make sure you are running Node v16 or greater. + To follow along with this quick-start guide, download the [Holepunch examples repo](https://github.com/holepunchto/examples) From ef9e042c4c8735f26fbcc5c40470653bd29f7346 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 13:59:13 +0530 Subject: [PATCH 06/35] Improve text --- hp-docs/README.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index f701d7e..9c877f6 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -41,7 +41,7 @@ Helper modules can be used together with the building blocks to create cutting-e
hypershell.mdA CLI to create and connect to P2P E2E encrypted shells.hypershell.md
hypertele.mdA swiss-knife proxy powered by hyperdht.md.hypertele.md
hyperbeam.mdA one-to-one and end-to-end encrypted internet pipe.hyperbeam.md
hyperssh.mdA CLI to run SSH over the hyperdht.md.hyperssh.md
drives.mdCLI to download, seed, and mirror a hyperdrive.md or a localdrive.md.drives.md
- These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). + ✔️These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). ## Applications built using Holepunch @@ -84,13 +84,7 @@ Hypercores are single-writer data structures, but collaboration is crucial. [aut As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. -<<<<<<< HEAD -{% hint style="warning" %} -Autobase is still experimental and is likely to change significantly in the near future. Give it a shot! -{% endhint %} -======= - Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! ->>>>>>> ea20f22852e518505a9bc5035bfc5ee1c67373ee + ⚠️ Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! ## Stability indexing @@ -118,4 +112,4 @@ The following stability indices have been used: | [hyperdht.md](building-blocks/hyperdht.md "mention") | **stable** | - Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. + ⚠️ Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. From 5da25bc0c8c36e2f2071fb04c7f861b2a4b4ac7f Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 14:09:26 +0530 Subject: [PATCH 07/35] Add block quotes in quickstart --- hp-docs/README.md | 2 +- hp-docs/quick-start.md | 19 ++++++++----------- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 9c877f6..4a94385 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -41,7 +41,7 @@ Helper modules can be used together with the building blocks to create cutting-e
hypershell.mdA CLI to create and connect to P2P E2E encrypted shells.hypershell.md
hypertele.mdA swiss-knife proxy powered by hyperdht.md.hypertele.md
hyperbeam.mdA one-to-one and end-to-end encrypted internet pipe.hyperbeam.md
hyperssh.mdA CLI to run SSH over the hyperdht.md.hyperssh.md
drives.mdCLI to download, seed, and mirror a hyperdrive.md or a localdrive.md.drives.md
- ✔️These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). + ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). ## Applications built using Holepunch diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index 366e73c..f7777d3 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -38,11 +38,8 @@ Install the required dependencies npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a debounceify graceful-goodbye --save ``` -{% endcode %} + ℹ️ Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. -{% hint style="info" %} -Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. -{% endhint %} ### Hyperswarm's DHT: Connecting Two Peers by Key @@ -50,11 +47,11 @@ Every code example in this page is meant to be run standalone, so you can copy/p In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. -{% hint style="info" %} -Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. -For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. -{% endhint %} + ℹ️ Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. + + For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. + Let's use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. @@ -369,9 +366,9 @@ The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-r `writer.mjs` stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and download the dictionary data into their local Hyperbee instances. -{% hint style="info" %} -Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. -{% endhint %} + + ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. + {% code title="writer.mjs" %} From 7c4c5c26438eb4de3d5d9bbabf82f70c610bbbae Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 14:30:40 +0530 Subject: [PATCH 08/35] Update README.md --- hp-docs/README.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 4a94385..af2fc5c 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -1,19 +1,10 @@ --- -description: Welcome to the Pear docs! 👋 +description: Welcome to the Holepunch docs! 👋 --- -# Pear by Holepunch +# Overview -> Pear loads applications remotely from peers and allows anyone to create and share applications with peers. - -Pear by Holepunch is a combined runtime, Software Development Kit (SDK) and Peer-to-Peer (P2P) Deployment tool. -Pear makes it possible to build, share, and extend P2P applications using common web and mobile technology stacks. -Pears enables to create unstoppable, zero-infrastructure P2P applications for desktop, terminal and mobile. - -Welcome to the Internet of Pears-Holepunch, the P2P Company! - -## Platform -* [Pear Command line](pear/01-command-line.md) +Holepunch equips developers with a powerful suite of independent components to effortlessly construct peer-to-peer applications. ## Building blocks From 1a08f0b28e9c9dd775beacc13b163f5e9bc5bb22 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 14:56:25 +0530 Subject: [PATCH 09/35] Update quick-start.md --- hp-docs/quick-start.md | 59 ++++++++++++++++-------------------------- 1 file changed, 22 insertions(+), 37 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index f7777d3..e4cf2b1 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -32,14 +32,14 @@ npm install Install the required dependencies -{% code overflow="wrap" %} ```bash npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a debounceify graceful-goodbye --save ``` - ℹ️ Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. - +``` + ℹ️: Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. +``` ### Hyperswarm's DHT: Connecting Two Peers by Key @@ -48,7 +48,7 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. - ℹ️ Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. + ℹ️: Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. @@ -57,9 +57,9 @@ Let's use the HyperDHT to create a basic CLI chat app where a client peer connec `server.mjs` will create a key pair and then start a server that will listen on the generated key pair. The public key is logged into the console. Copy it for instantiating the client. -{% code title="server.mjs" %} ```javascript +server.mjs import DHT from 'hyperdht' import goodbye from 'graceful-goodbye' import b4a from 'b4a' @@ -83,15 +83,16 @@ server.listen(keyPair).then(() => { goodbye(() => server.close()) ``` -{% endcode %} + `client.mjs` will spin up a client, and the public key copied earlier must be supplied as a command line argument for connecting to the server. The client process will log `got connection` into the console when it connects to the server. Once it's connected, try typing in both terminals! -{% code title="client.mjs" %} + ``` javascript +client.mjs import DHT from 'hyperdht' import b4a from 'b4a' @@ -105,7 +106,6 @@ conn.once('open', () => console.log('got connection!')) process.stdin.pipe(conn).pipe(process.stdout) ``` -{% endcode %} ### Hyperswarm: Connecting to Many Peers by Topic @@ -119,9 +119,10 @@ This example consists of a single file, `peer.mjs`. In one terminal, type `node Start typing into any terminal you like, and it will be broadcast to all connected peers! -{% code title="peer.mjs" %} + ```javascript +peer.mjs import Hyperswarm from 'hyperswarm' import goodbye from 'graceful-goodbye' import crypto from 'hypercore-crypto' @@ -157,8 +158,6 @@ discovery.flushed().then(() => { }) ``` -{% endcode %} - ### Hypercore: The Basics In the Hyperswarm examples, peers can exchange chat messages so long as both are online at the same time and directly connected, and those messages are not persistent (they will be lost if the recipient is offline). Hypercore fixes all of these problems. @@ -174,9 +173,10 @@ The following example consists of two files: `reader.mjs` and `writer.mjs`. When `writer.mjs` stores the data entered into the command line to the Hypercore instance. The Hypercore instance is replicated with other peers using Hyperswarm. -{% code title="writer.mjs" %} + ```javascript +writer.mjs import Hyperswarm from 'hyperswarm' import Hypercore from 'hypercore' import goodbye from 'graceful-goodbye' @@ -200,13 +200,13 @@ swarm.join(core.discoveryKey) swarm.on('connection', conn => core.replicate(conn)) ``` -{% endcode %} `reader.mjs` uses Hyperswarm to connect to the previously initiated peer and synchronize the local Hypercore instance with the Hypercore instance of the writer. -{% code title="reader.mjs" %} + ```javascript +reader.mjs import Hyperswarm from 'hyperswarm' import Hypercore from 'hypercore' import goodbye from 'graceful-goodbye' @@ -238,7 +238,6 @@ for await (const block of core.createReadStream({ start: core.length, live: true } ``` -{% endcode %} ### Corestore: Working with Many Hypercores @@ -250,9 +249,9 @@ This example consists of two files: `writer.mjs` and `reader.mjs`. In the previo The file `writer.mjs` uses a Corestore instance to create three Hypercores, which are then replicated with other peers using Hyperswarm. The keys for the second and third cores are stored in the first core (the first core 'bootstraps' the system). Messages entered into the command line are written into the second and third cores, depending on the length of the message. To execute `reader.mjs`, you should copy the main core key logged into the command line. -{% code title="writer.mjs" %} ```javascript +writer.mjs import Corestore from 'corestore' import Hyperswarm from 'hyperswarm' import goodbye from 'graceful-goodbye' @@ -299,13 +298,12 @@ process.stdin.on('data', data => { }) ``` -{% endcode %} `reader.mjs` connects to the previous peer with Hyperswarm and replicates the local Corestore instance to receive the data from it. This requires the copied key to be supplied as an argument when executing the file, which will then be used to create a core with the same public key as the other peer (i.e., the same discovery key for both the reader and writer peers). -{% code title="reader.mjs" %} ```javascript +reader.mjs import Corestore from 'corestore' import Hyperswarm from 'hyperswarm' import goodbye from 'graceful-goodbye' @@ -356,8 +354,6 @@ for (const key of otherKeys) { } ``` -{% endcode %} - ### Hyperbee: Sharing Append-Only Databases [hyperbee.md](building-blocks/hyperbee.md "mention") is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. @@ -370,9 +366,8 @@ The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-r ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. -{% code title="writer.mjs" %} - ```javascript +writer.mjs import fs from 'fs' import zlib from 'zlib' @@ -438,15 +433,13 @@ async function loadDictionary() { } ``` -{% endcode %} `bee-reader.mjs` creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as the above file. On every word entered in the command line, it will download the respective data to the local Hyperbee instance. Try looking at how much disk space the `reader-storage` directory is using after each query. You'll notice that it's significantly smaller than `writer-storage`! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call **sparse downloading.** -{% code title="bee-reader.mjs" %} - ```javascript +bee-reader.mjs import Hyperswarm from 'hyperswarm' import Corestore from 'corestore' import Hyperbee from 'hyperbee' @@ -493,15 +486,13 @@ process.stdin.on('data', data => { }) ``` -{% endcode %} - Importantly, a Hyperbee is **just** a Hypercore, where the tree nodes are stored as Hypercore blocks. Let's examine the Hyperbee as if it were just a Hypercore and log out a few blocks. `core-reader.mjs` will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which we can easily import and use. -{% code title="core-reader.mjs" %} ```javascript +core-reader.mjs import Hypercore from 'hypercore' import Hyperswarm from 'hyperswarm' import Corestore from 'corestore' @@ -540,8 +531,6 @@ console.log(`Raw Block ${seq}:`, lastBlock) console.log(`Decoded Block ${seq}`, Node.decode(lastBlock)) ``` -{% endcode %} - ### Hyperdrive: A Full P2P Filesystem [hyperdrive.md](building-blocks/hyperdrive.md "mention") is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. @@ -552,9 +541,9 @@ This example consists of three files: `writer.mjs`, `drive-reader.mjs` and `bee- `writer.mjs` creates a local drive instance for a local directory and then mirrors the local drive into the Hyperdrive instance. The store used to create the Hyperdrive instance is replicated using Hyperswarm to make the data of Hyperdrive accessible to other peers. Copy the drive key logged into the command line for the `reader.mjs` execution. -{% code title="writer.mjs" %} ```javascript +writer.js import Hyperswarm from 'hyperswarm' import Hyperdrive from 'hyperdrive' import Localdrive from 'localdrive' @@ -606,15 +595,13 @@ async function mirrorDrive () { } ``` -{% endcode %} - `drive-reader.mjs` creates a local drive instance for a local directory and then mirrors the contents of the local Hyperdrive instance into the local drive instance (which will write the contents to the local directory). Try running `node drive-reader.mjs (key-from-above)`, then add/remove/modify files inside `writer-dir` then press `Enter` in the writer's terminal (to import the local changes into the writer's drive). You'll see all new changes mirror into `reader-dir`. -{% code title="drive-reader.mjs" %} ```javascript +drive-reader.mjs import Hyperswarm from 'hyperswarm' import Hyperdrive from 'hyperdrive' import Localdrive from 'localdrive' @@ -664,7 +651,6 @@ async function mirrorDrive () { } ``` -{% endcode %} Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee (which is **just** a Hypercore). Let's inspect the Hyperdrive as though it were a Hyperbee, and log out some file metadata. @@ -672,9 +658,9 @@ Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee Try adding or removing a few files from the writer's data directory, then pressing `Enter` in the writer's terminal to mirror the changes. -{% code title="bee-reader.mjs" %} ```javascript +bee-reader.mjs import Hyperswarm from 'hyperswarm' import Corestore from 'corestore' import Hyperbee from 'hyperbee' @@ -722,4 +708,3 @@ async function listBee () { } ``` -{% endcode %} From 0e66acea5c4f670ace1d6adf53b53d6c12caf048 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Wed, 17 Jan 2024 14:58:37 +0530 Subject: [PATCH 10/35] Update quick-start.md --- hp-docs/quick-start.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index e4cf2b1..14d2c8b 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -38,7 +38,7 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de ``` ``` - ℹ️: Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. +ℹ️: Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. ``` ### Hyperswarm's DHT: Connecting Two Peers by Key @@ -47,11 +47,11 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. +``` +ℹ️: Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. - ℹ️: Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. - - For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. - +For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. +``` Let's use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. @@ -362,9 +362,9 @@ The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-r `writer.mjs` stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and download the dictionary data into their local Hyperbee instances. - - ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. - +``` +ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. +``` ```javascript writer.mjs From 58932b8458d5ed79f7c84243aa21b2bba29f4b26 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 15:42:29 +0530 Subject: [PATCH 11/35] Updated for grammar and language --- hp-docs/quick-start.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index 14d2c8b..0980f9e 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -14,23 +14,23 @@ description: A primer on Holepunch's modular building blocks ### Setup -> ⚠️ Before we begin, make sure you are running Node v16 or greater. +> ⚠️ Before begining the setup, ensure sure the system is running on Node v16 or greater. -To follow along with this quick-start guide, download the [Holepunch examples repo](https://github.com/holepunchto/examples) +Download the [Holepunch examples repo](https://github.com/holepunchto/examples) ```bash git clone https://github.com/holepunchto/examples ``` -Install the required dependencies +Install the required dependencies: ```bash cd examples npm install ``` -Install the required dependencies +Install the required dependencies: ```bash @@ -38,12 +38,12 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de ``` ``` -ℹ️: Every code example in this page is meant to be run standalone, so you can copy/paste each example into a JS file, and run it with NodeJS. +ℹ️: Every code example in this page is meant to be run standalone, so copy/paste each example into a JS file, and run it with NodeJS. ``` ### Hyperswarm's DHT: Connecting Two Peers by Key -[Hyperswarm](building-blocks/hyperswarm.md) allows you to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. +[Hyperswarm](building-blocks/hyperswarm.md) used to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. @@ -53,7 +53,7 @@ In the HyperDHT, peers are identified by a public key, not by an IP address. If For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. ``` -Let's use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. +For example, use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. `server.mjs` will create a key pair and then start a server that will listen on the generated key pair. The public key is logged into the console. Copy it for instantiating the client. @@ -109,15 +109,15 @@ process.stdin.pipe(conn).pipe(process.stdout) ### Hyperswarm: Connecting to Many Peers by Topic -In the above example, we connected two peers directly using the first peer's public key. Often, you'll want to discover peers swarming a common topic, and connect to as many of them as you can. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. +In the above example, two peers connected directly using the first peer's public key. The frequent use of Hyperswarm is to discover peers swarming a common topic, and connect to as many of them as possible. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. -The [Hyperswarm](building-blocks/hyperswarm.md) module provides a higher-level interface over the underlying DHT, abstracting away the mechanics of establishing and maintaining connections. Instead, you 'join' topics, and the swarm discovers peers automatically. It also handles reconnections in the event of failures. +The [Hyperswarm](building-blocks/hyperswarm.md) module provides a higher-level interface over the underlying DHT, abstracting away the mechanics of establishing and maintaining connections. Instead, 'join' topics, and the swarm discovers peers automatically. It also handles reconnections in the event of failures. -In the previous example, we needed to explicitly indicate which peer was the server and which was the client. Here, we create two peers, have them join a common topic, and let the swarm deal with connections. +In the previous example, to explicitly indicate which peer was the server and which was the client, create two peers, have them join a common topic, and let the swarm deal with connections. This example consists of a single file, `peer.mjs`. In one terminal, type `node peer.mjs`, it will display the topic. Copy/paste that topic into N additional terminals with `node peer.mjs (topic)`. Each peer will log information about the other connected peers. -Start typing into any terminal you like, and it will be broadcast to all connected peers! +Start typing into any terminal anything, and it will be broadcast to all connected peers! @@ -164,7 +164,7 @@ In the Hyperswarm examples, peers can exchange chat messages so long as both are [hypercore.md](building-blocks/hypercore.md "mention") is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications. -Let's extend the ephemeral chat example above but using Hypercore to add many significant new features: +Now extend the ephemeral chat example above but using Hypercore to add many significant new features: 1. **Persistence**: The owner of the Hypercore can add messages at any time, and they'll be persisted to disk. Whenever they come online, readers can replicate these messages over Hyperswarm. 2. **Many Readers:** New messages added to the Hypercore will be broadcast to interested readers. The owner gives each reader a reading capability (`core.key`) and a corresponding discovery key (`core.discoveryKey`). The former is used to authorize the reader, ensuring that they have permission to read messages, and the latter is used to discover the owner (and other readers) on the swarm. @@ -241,13 +241,13 @@ for await (const block of core.createReadStream({ start: core.length, live: true ### Corestore: Working with Many Hypercores -An append-only log is powerful on its own, but it's most useful as a building block for constructing larger data structures, such as databases or filesystems. When building these data structures, you'll often need many cores, each with different responsibilities. For example, Hyperdrive uses one core to store file metadata and another to store file contents. +An append-only log is powerful on its own, but it's most useful as a building block for constructing larger data structures, such as databases or filesystems. Building these data structures often requires many cores, each with different responsibilities. For example, Hyperdrive uses one core to store file metadata and another to store file contents. -[corestore.md](helpers/corestore.md "mention") is a Hypercore factory that makes it easier to manage large collections of named Hypercores. Below you'll find a simple example that demonstrates a pattern we often use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. +[corestore.md](helpers/corestore.md "mention") is a Hypercore factory that makes it easier to manage large collections of named Hypercores. A simple example below demonstrates a pattern often in use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. This example consists of two files: `writer.mjs` and `reader.mjs`. In the previous example, we replicated only a single Hypercore instance. But in this example, we will replicate a single Corestore instance, which will internally manage the replication of a collection of Hypercores. -The file `writer.mjs` uses a Corestore instance to create three Hypercores, which are then replicated with other peers using Hyperswarm. The keys for the second and third cores are stored in the first core (the first core 'bootstraps' the system). Messages entered into the command line are written into the second and third cores, depending on the length of the message. To execute `reader.mjs`, you should copy the main core key logged into the command line. +The file `writer.mjs` uses a Corestore instance to create three Hypercores, which are then replicated with other peers using Hyperswarm. The keys for the second and third cores are stored in the first core (the first core 'bootstraps' the system). Messages entered into the command line are written into the second and third cores, depending on the length of the message. To execute `reader.mjs`, copy the main core key logged into the command line. ```javascript @@ -436,7 +436,7 @@ async function loadDictionary() { `bee-reader.mjs` creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as the above file. On every word entered in the command line, it will download the respective data to the local Hyperbee instance. -Try looking at how much disk space the `reader-storage` directory is using after each query. You'll notice that it's significantly smaller than `writer-storage`! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call **sparse downloading.** +Try looking at disk space the `reader-storage` directory is using after each query. notice that it's significantly smaller than `writer-storage`! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call **sparse downloading.** ```javascript bee-reader.mjs @@ -486,7 +486,7 @@ process.stdin.on('data', data => { }) ``` -Importantly, a Hyperbee is **just** a Hypercore, where the tree nodes are stored as Hypercore blocks. Let's examine the Hyperbee as if it were just a Hypercore and log out a few blocks. +Importantly, a Hyperbee is **just** a Hypercore, where the tree nodes are stored as Hypercore blocks. Now examine the Hyperbee as if it were just a Hypercore and log out a few blocks. `core-reader.mjs` will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which we can easily import and use. @@ -597,7 +597,7 @@ async function mirrorDrive () { `drive-reader.mjs` creates a local drive instance for a local directory and then mirrors the contents of the local Hyperdrive instance into the local drive instance (which will write the contents to the local directory). -Try running `node drive-reader.mjs (key-from-above)`, then add/remove/modify files inside `writer-dir` then press `Enter` in the writer's terminal (to import the local changes into the writer's drive). You'll see all new changes mirror into `reader-dir`. +Try running `node drive-reader.mjs (key-from-above)`, then add/remove/modify files inside `writer-dir` then press `Enter` in the writer's terminal (to import the local changes into the writer's drive). Observe that all new changes mirror into `reader-dir`. ```javascript @@ -652,7 +652,7 @@ async function mirrorDrive () { ``` -Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee (which is **just** a Hypercore). Let's inspect the Hyperdrive as though it were a Hyperbee, and log out some file metadata. +Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee (which is **just** a Hypercore). Now inspect the Hyperdrive as though it were a Hyperbee, and log out some file metadata. `bee-reader.mjs` creates a Hyperbee instance using the Hypercore instance created with the copied public key. Every time the Hyperbee is updated (an `append` event is emitted on the underlying Hypercore), all file metadata nodes will be logged out. From 7a37902f73ab3e77bcb9661fa81558a35e200f63 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 15:46:51 +0530 Subject: [PATCH 12/35] Update quick-start.md removed duplicate sentence. --- hp-docs/quick-start.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index 0980f9e..a507f0a 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -30,8 +30,6 @@ cd examples npm install ``` -Install the required dependencies: - ```bash npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a debounceify graceful-goodbye --save From e278d5458487dc98d03b8f7368abc81066a56208 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 16:01:08 +0530 Subject: [PATCH 13/35] Update quick-start.md Updated code blocks and grammar. --- hp-docs/quick-start.md | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index a507f0a..c3207be 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -35,9 +35,7 @@ npm install npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a debounceify graceful-goodbye --save ``` -``` -ℹ️: Every code example in this page is meant to be run standalone, so copy/paste each example into a JS file, and run it with NodeJS. -``` +> ℹ️ Every code example in this page is meant to be run standalone, so copy/paste each example into a JS file, and run it with NodeJS. ### Hyperswarm's DHT: Connecting Two Peers by Key @@ -45,11 +43,9 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. -``` -ℹ️: Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. - +> ℹ️ Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. -``` + For example, use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. @@ -360,9 +356,7 @@ The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-r `writer.mjs` stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and download the dictionary data into their local Hyperbee instances. -``` -ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. -``` +> ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. ```javascript writer.mjs @@ -533,7 +527,7 @@ console.log(`Decoded Block ${seq}`, Node.decode(lastBlock)) [hyperdrive.md](building-blocks/hyperdrive.md "mention") is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. -Let's mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, we'll use two additional tools: [mirrordrive.md](helpers/mirrordrive.md "mention") and [localdrive.md](helpers/localdrive.md "mention"), which handle all interactions between Hyperdrives and the local filesystem. +Now mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, use two additional tools: [mirrordrive.md](helpers/mirrordrive.md "mention") and [localdrive.md](helpers/localdrive.md "mention"), which handle all interactions between Hyperdrives and the local filesystem. This example consists of three files: `writer.mjs`, `drive-reader.mjs` and `bee-reader.mjs`. From bdbe88661714ce50cd29c0b35821435c9e128a8f Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 16:04:54 +0530 Subject: [PATCH 14/35] Update README.md updated grammar! --- hp-docs/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index af2fc5c..b8d5247 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -19,13 +19,13 @@ The Holepunch Ecosystem is constructed utilizing the following structural compon ## Helpers -Helper modules can be used together with the building blocks to create cutting-edge peer-to-peer tools and applications. +Helper modules can be used together with the building blocks to create cutting-edge P2P tools and applications. 1. [corestore.md](helpers/corestore.md "mention"): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. 2. [localdrive.md](helpers/localdrive.md "mention"): A file system interoperable with Hyperdrive. 3. [mirrordrive.md](helpers/mirrordrive.md "mention"): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md "mention") or a [localdrive.md](helpers/localdrive.md "mention") into another one. 4. [secretstream.md](helpers/secretstream.md "mention"): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md "mention"): A series of binary encoding schemes for building fast and small parsers and serializers. Pears use this in Keet to store chat messages and in Hypercore's replication protocol. +5. [compact-encoding.md](helpers/compact-encoding.md "mention"): A series of binary encoding schemes for building fast and small parsers and serializers. Holepunch use this in Keet to store chat messages and in Hypercore's replication protocol. 6. [protomux.md](helpers/protomux.md "mention"): Multiplex multiple message oriented protocols over a stream. ## Tools built using Holepunch @@ -41,11 +41,11 @@ Helper modules can be used together with the building blocks to create cutting-e ## What's new? -Pears is continuously introucing new enhancements and following are the few highlights. +Holepunch is continuously introucing new enhancements and following are the few highlights. ### Better Building Blocks -Pears focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being unnecessarily opinionated and without any operational complexity. +Holepunch focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being unnecessarily opinionated and without any operational complexity. The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. From 57847b47a4e4f4bd6eb445decc23cbf2e772e3ae Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 17:54:41 +0530 Subject: [PATCH 15/35] Update README.md Added Pear Overview --- hp-docs/README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index b8d5247..8983c21 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -1,8 +1,17 @@ --- -description: Welcome to the Holepunch docs! 👋 +Welcome to the Holepunch docs! 👋 --- # Overview +Pear loads applications remotely from peers and allows anyone to create and share applications with peers. + +Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool. + +Pear makes it possible to build, share and extend P2P applications using common Web and Mobile technology. + +Herein is everything needed to create unstoppable, zero-infrastructure P2P applications for Desktop, Terminal & Mobile (soon). + +Welcome to the Internet of Pears–Holepunch, the P2P Company! Holepunch equips developers with a powerful suite of independent components to effortlessly construct peer-to-peer applications. From 574269ce0723345f0f74fa1856ddf82a83c435cf Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Wed, 17 Jan 2024 18:00:01 +0530 Subject: [PATCH 16/35] Update README.md updated for content arrangement in overview section! --- hp-docs/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 8983c21..c2b362a 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -5,16 +5,14 @@ Welcome to the Holepunch docs! 👋 # Overview Pear loads applications remotely from peers and allows anyone to create and share applications with peers. -Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool. +Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool and makes it possible to build, share and extend P2P applications using common Web and Mobile technology. -Pear makes it possible to build, share and extend P2P applications using common Web and Mobile technology. +Holepunch equips developers with a powerful suite of independent components to effortlessly construct peer-to-peer applications. Herein is everything needed to create unstoppable, zero-infrastructure P2P applications for Desktop, Terminal & Mobile (soon). Welcome to the Internet of Pears–Holepunch, the P2P Company! -Holepunch equips developers with a powerful suite of independent components to effortlessly construct peer-to-peer applications. - ## Building blocks The Holepunch Ecosystem is constructed utilizing the following structural components. From 650268154729a0209248f3456d5e9dee4a2cc65a Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:00:59 +0530 Subject: [PATCH 17/35] Update README.md updated the description. --- hp-docs/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index c2b362a..0818f72 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -5,11 +5,11 @@ Welcome to the Holepunch docs! 👋 # Overview Pear loads applications remotely from peers and allows anyone to create and share applications with peers. -Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool and makes it possible to build, share and extend P2P applications using common Web and Mobile technology. +Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool, and makes it possible to build, share and extend P2P applications using common Web and Mobile technology. -Holepunch equips developers with a powerful suite of independent components to effortlessly construct peer-to-peer applications. +Holepunch equips developers with a powerful suite of independent components to effortlessly construct P2P applications. -Herein is everything needed to create unstoppable, zero-infrastructure P2P applications for Desktop, Terminal & Mobile (soon). +Holepunch has everything needed to create unstoppable, zero-infrastructure P2P applications for Desktop, Terminal & Mobile (soon). Welcome to the Internet of Pears–Holepunch, the P2P Company! From 59814b145416ff743c4d5ba84143b5e4b9973f41 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:18:16 +0530 Subject: [PATCH 18/35] Update README.md applied code block for note. --- hp-docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 0818f72..c65b202 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -39,7 +39,7 @@ Helper modules can be used together with the building blocks to create cutting-e
hypershell.mdA CLI to create and connect to P2P E2E encrypted shells.hypershell.md
hypertele.mdA swiss-knife proxy powered by hyperdht.md.hypertele.md
hyperbeam.mdA one-to-one and end-to-end encrypted internet pipe.hyperbeam.md
hyperssh.mdA CLI to run SSH over the hyperdht.md.hyperssh.md
drives.mdCLI to download, seed, and mirror a hyperdrive.md or a localdrive.md.drives.md
- ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). +> ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). ## Applications built using Holepunch From e15d80b49c26289efe9d8f6836b9b1700798239e Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:19:32 +0530 Subject: [PATCH 19/35] Update README.md updated code block styles --- hp-docs/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index c65b202..fdb020e 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -82,7 +82,7 @@ Hypercores are single-writer data structures, but collaboration is crucial. [aut As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. - ⚠️ Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! +> ⚠️ Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! ## Stability indexing @@ -110,4 +110,4 @@ The following stability indices have been used: | [hyperdht.md](building-blocks/hyperdht.md "mention") | **stable** | - ⚠️ Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. + >⚠️ Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. From f8053502bb135fca3eb3eea5aa09b237fa61131d Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:23:13 +0530 Subject: [PATCH 20/35] Update README.md Removed Pear reference content --- hp-docs/README.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index fdb020e..8498ca3 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -3,16 +3,9 @@ Welcome to the Holepunch docs! 👋 --- # Overview -Pear loads applications remotely from peers and allows anyone to create and share applications with peers. - -Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool, and makes it possible to build, share and extend P2P applications using common Web and Mobile technology. Holepunch equips developers with a powerful suite of independent components to effortlessly construct P2P applications. -Holepunch has everything needed to create unstoppable, zero-infrastructure P2P applications for Desktop, Terminal & Mobile (soon). - -Welcome to the Internet of Pears–Holepunch, the P2P Company! - ## Building blocks The Holepunch Ecosystem is constructed utilizing the following structural components. From 9eb36818edc9104dc4032a4dcd7c387992f22bd4 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:01:05 +0530 Subject: [PATCH 21/35] Update README.md --- hp-docs/README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 8498ca3..cef9f4a 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -30,7 +30,13 @@ Helper modules can be used together with the building blocks to create cutting-e ## Tools built using Holepunch -
hypershell.mdA CLI to create and connect to P2P E2E encrypted shells.hypershell.md
hypertele.mdA swiss-knife proxy powered by hyperdht.md.hypertele.md
hyperbeam.mdA one-to-one and end-to-end encrypted internet pipe.hyperbeam.md
hyperssh.mdA CLI to run SSH over the hyperdht.md.hyperssh.md
drives.mdCLI to download, seed, and mirror a hyperdrive.md or a localdrive.md.drives.md
+| Tools | Description | +| :----------------------------------------------------------: | :---------------------------------------------------------: | +| **[Hypershell](https://docs.holepunch.to/tools/hypershell)** | A CLI to create and connect to P2P E2E encrypted shells.. | +| **[Hypertele](https://docs.holepunch.to/tools/hypertele)** | A swiss-knife proxy powered by [HyperDHT](https://docs.holepunch.to/building-blocks/hyperdht). | +| **[Hyperbeam](https://docs.holepunch.to/tools/hyperbeam)** | A one-to-one and end-to-end encrypted internet pipe. | +| **[Hyperssh](https://docs.holepunch.to/tools/hyperssh)** | A CLI to run SSH over the [HyperDHT](https://docs.holepunch.to/building-blocks/hyperdht). | +| **[Drives](https://docs.holepunch.to/tools/drives)** | CLI to download, seed, and mirror a [hyperdrive](https://docs.holepunch.to/building-blocks/hyperdrive) or a [localdrive](https://docs.holepunch.to/helpers/localdrive). | > ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). From da67f767fcdceb4740bb46c29d234786e405ad34 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:13:32 +0530 Subject: [PATCH 22/35] Update README.md --- hp-docs/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index cef9f4a..07c062f 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -21,12 +21,12 @@ The Holepunch Ecosystem is constructed utilizing the following structural compon Helper modules can be used together with the building blocks to create cutting-edge P2P tools and applications. -1. [corestore.md](helpers/corestore.md "mention"): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. -2. [localdrive.md](helpers/localdrive.md "mention"): A file system interoperable with Hyperdrive. -3. [mirrordrive.md](helpers/mirrordrive.md "mention"): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md "mention") or a [localdrive.md](helpers/localdrive.md "mention") into another one. -4. [secretstream.md](helpers/secretstream.md "mention"): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md "mention"): A series of binary encoding schemes for building fast and small parsers and serializers. Holepunch use this in Keet to store chat messages and in Hypercore's replication protocol. -6. [protomux.md](helpers/protomux.md "mention"): Multiplex multiple message oriented protocols over a stream. +1. [corestore.md](helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. +2. [localdrive.md](helpers/localdrive.md): A file system interoperable with Hyperdrive. +3. [mirrordrive.md](helpers/mirrordrive.md): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md) or a [localdrive.md](helpers/localdrive.md) into another one. +4. [secretstream.md](helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. +5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. Holepunch use this in Keet to store chat messages and in Hypercore's replication protocol. +6. [protomux.md](helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. ## Tools built using Holepunch From 9be08866e96250c617f697bf56630ea7c8dcfe68 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:17:54 +0530 Subject: [PATCH 23/35] Update README.md --- hp-docs/README.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 07c062f..c48cf53 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -10,12 +10,12 @@ Holepunch equips developers with a powerful suite of independent components to e The Holepunch Ecosystem is constructed utilizing the following structural components. -1. [hypercore.md](building-blocks/hypercore.md "mention"): A distributed, secure append-only log is a tool for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. -2. [hyperbee.md](building-blocks/hyperbee.md "mention"): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. -3. [hyperdrive.md](building-blocks/hyperdrive.md "mention"): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. -4. [autobase.md](building-blocks/autobase.md "mention"): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. -5. [hyperdht.md](building-blocks/hyperdht.md "mention"): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. -6. [hyperswarm.md](building-blocks/hyperswarm.md "mention"): A high-level API for finding and connecting to peers who are interested in a "topic." +1. [hypercore.md](building-blocks/hypercore.md): A distributed, secure append-only log is a tool for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. +2. [hyperbee.md](building-blocks/hyperbee.md): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. +3. [hyperdrive.md](building-blocks/hyperdrive.md): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. +4. [autobase.md](building-blocks/autobase.md): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. +5. [hyperdht.md](building-blocks/hyperdht.md): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. +6. [hyperswarm.md](building-blocks/hyperswarm.md): A high-level API for finding and connecting to peers who are interested in a "topic." ## Helpers @@ -43,11 +43,11 @@ Helper modules can be used together with the building blocks to create cutting-e ## Applications built using Holepunch -* [keet.io.md](apps/keet.io.md "mention")**:** A peer-to-peer chat and video-conferencing application with end-to-end encryption. +* [keet.io.md](apps/keet.io.md)**:** A peer-to-peer chat and video-conferencing application with end-to-end encryption. ## What's new? -Holepunch is continuously introucing new enhancements and following are the few highlights. +Holepunch is continuously introducing new enhancements and the following are a few highlights. ### Better Building Blocks @@ -55,15 +55,15 @@ Holepunch focus is on making the core building blocks easy to use, fast, and rel The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. -### Hypercore v10 +### Hypercore * The [`session`](building-blocks/hypercore.md#core.session-options) and [`snapshot`](building-blocks/hypercore.md#core.snapshot-options) methods for providing multiple views over the same underlying Hypercore, which simplifies resource management. -* A [`truncate`](building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](building-blocks/autobase.md "mention"), as described below. +* A [`truncate`](building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](building-blocks/autobase.md), as described below. * An improved fork detection in the replication protocol, to improve resilience. * Optional on-disk encryption for blocks (in addition to the existing transport encryption). * The storage layer now uses a write-ahead log to ensure that power loss or unexpected shutdown cannot lead to data corruption. -### Hyperswarm v4 +### Hyperswarm * An improved UDP holepunching algorithm that uses arbitrary DHT nodes (optionally selected by the connecting peers) to proxy necessary metadata while being maximally privacy-preserving. * A custom-built transport protocol, [UDX](https://github.com/hyperswarm/libudx), that takes advantage of the holepunching algorithm to avoid unnecessary overhead (it doesn't include handshaking since holepunching takes care of that, for example). It's blazing fast. @@ -73,7 +73,7 @@ The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. * Uses Hyperbee internally for storing file metadata * Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. -* Auxiliary tools, [localdrive.md](helpers/localdrive.md "mention") and [mirrordrive.md](helpers/mirrordrive.md "mention"), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem when deploying Keet. +* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem when deploying Keet. ### Autobase (experimental) @@ -101,12 +101,12 @@ The following stability indices have been used: | Module | Stability | | -------------------------------------------------------- | :----------------------------------------------------------: | -| [hypercore.md](building-blocks/hypercore.md "mention") | **stable** | -| [hyperbee.md](building-blocks/hyperbee.md "mention") | **stable** | -| [hyperdrive.md](building-blocks/hyperdrive.md "mention") | **stable** | -| [autobase.md](building-blocks/autobase.md "mention") | **experimental** | -| [hyperswarm.md](building-blocks/hyperswarm.md "mention") | **stable** | -| [hyperdht.md](building-blocks/hyperdht.md "mention") | **stable** | +| [hypercore.md](building-blocks/hypercore.md) | **stable** | +| [hyperbee.md](building-blocks/hyperbee.md) | **stable** | +| [hyperdrive.md](building-blocks/hyperdrive.md) | **stable** | +| [autobase.md](building-blocks/autobase.md) | **experimental** | +| [hyperswarm.md](building-blocks/hyperswarm.md) | **stable** | +| [hyperdht.md](building-blocks/hyperdht.md) | **stable** | >⚠️ Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. From 16b726b3d9578408abcef705add96224eef9f789 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:20:48 +0530 Subject: [PATCH 24/35] Update quick-start.md --- hp-docs/quick-start.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index c3207be..69635a8 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -14,7 +14,7 @@ description: A primer on Holepunch's modular building blocks ### Setup -> ⚠️ Before begining the setup, ensure sure the system is running on Node v16 or greater. +> ⚠️ Before beginning the setup, ensure sure the system is running on Node v16 or greater. Download the [Holepunch examples repo](https://github.com/holepunchto/examples) @@ -44,7 +44,7 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. > ℹ️ Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. -For example, Keet implements its own relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. +For example, Keet implements its relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. For example, use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. @@ -156,7 +156,7 @@ discovery.flushed().then(() => { In the Hyperswarm examples, peers can exchange chat messages so long as both are online at the same time and directly connected, and those messages are not persistent (they will be lost if the recipient is offline). Hypercore fixes all of these problems. -[hypercore.md](building-blocks/hypercore.md "mention") is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications. +[hypercore.md](building-blocks/hypercore.md) is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications. Now extend the ephemeral chat example above but using Hypercore to add many significant new features: @@ -237,7 +237,7 @@ for await (const block of core.createReadStream({ start: core.length, live: true An append-only log is powerful on its own, but it's most useful as a building block for constructing larger data structures, such as databases or filesystems. Building these data structures often requires many cores, each with different responsibilities. For example, Hyperdrive uses one core to store file metadata and another to store file contents. -[corestore.md](helpers/corestore.md "mention") is a Hypercore factory that makes it easier to manage large collections of named Hypercores. A simple example below demonstrates a pattern often in use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. +[corestore.md](helpers/corestore.md) is a Hypercore factory that makes it easier to manage large collections of named Hypercores. A simple example below demonstrates a pattern often in use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. This example consists of two files: `writer.mjs` and `reader.mjs`. In the previous example, we replicated only a single Hypercore instance. But in this example, we will replicate a single Corestore instance, which will internally manage the replication of a collection of Hypercores. @@ -350,7 +350,7 @@ for (const key of otherKeys) { ### Hyperbee: Sharing Append-Only Databases -[hyperbee.md](building-blocks/hyperbee.md "mention") is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. +[hyperbee.md](building-blocks/hyperbee.md) is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-reader.mjs`. @@ -525,9 +525,9 @@ console.log(`Decoded Block ${seq}`, Node.decode(lastBlock)) ### Hyperdrive: A Full P2P Filesystem -[hyperdrive.md](building-blocks/hyperdrive.md "mention") is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. +[hyperdrive.md](building-blocks/hyperdrive.md) is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. -Now mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, use two additional tools: [mirrordrive.md](helpers/mirrordrive.md "mention") and [localdrive.md](helpers/localdrive.md "mention"), which handle all interactions between Hyperdrives and the local filesystem. +Now mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, use two additional tools: [mirrordrive.md](helpers/mirrordrive.md) and [localdrive.md](helpers/localdrive.md), which handle all interactions between Hyperdrives and the local filesystem. This example consists of three files: `writer.mjs`, `drive-reader.mjs` and `bee-reader.mjs`. From a9b04ff349ddcaec4c01bb8e12916faced005980 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:32:10 +0530 Subject: [PATCH 25/35] Update README.md Made few text edits based on the review. --- hp-docs/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index c48cf53..88e0478 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -25,7 +25,7 @@ Helper modules can be used together with the building blocks to create cutting-e 2. [localdrive.md](helpers/localdrive.md): A file system interoperable with Hyperdrive. 3. [mirrordrive.md](helpers/mirrordrive.md): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md) or a [localdrive.md](helpers/localdrive.md) into another one. 4. [secretstream.md](helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. Holepunch use this in Keet to store chat messages and in Hypercore's replication protocol. +5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. Holepunch uses this in Keet to store chat messages and in Hypercore's replication protocol. 6. [protomux.md](helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. ## Tools built using Holepunch @@ -51,7 +51,7 @@ Holepunch is continuously introducing new enhancements and the following are a f ### Better Building Blocks -Holepunch focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being unnecessarily opinionated and without any operational complexity. +Holepunch focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being opinionated and introducing any operational complexity. The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. @@ -73,11 +73,11 @@ The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. * Uses Hyperbee internally for storing file metadata * Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. -* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem when deploying Keet. +* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use this every day when deploying Keet. ### Autobase (experimental) -Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view to see on screen. +Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view the member see on the screen. As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. From da58b34230d18bd7769357d77edb5ea37d2da9e3 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:45:02 +0530 Subject: [PATCH 26/35] Update README.md language changes --- hp-docs/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 88e0478..25f99aa 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -73,11 +73,11 @@ The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. * Uses Hyperbee internally for storing file metadata * Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. -* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use this every day when deploying Keet. +* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. ### Autobase (experimental) -Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view the member see on the screen. +Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view members see on the screen. As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. From d581dd0275ced12dc456b807b99aa783ec540b32 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 14:09:41 +0530 Subject: [PATCH 27/35] Update README.md Made corrections as per review. --- hp-docs/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 25f99aa..50a015a 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -4,7 +4,7 @@ Welcome to the Holepunch docs! 👋 # Overview -Holepunch equips developers with a powerful suite of independent components to effortlessly construct P2P applications. +Holepunch equips developers with a powerful suite of independent components to effortlessly construct P2P applications. ## Building blocks @@ -25,7 +25,7 @@ Helper modules can be used together with the building blocks to create cutting-e 2. [localdrive.md](helpers/localdrive.md): A file system interoperable with Hyperdrive. 3. [mirrordrive.md](helpers/mirrordrive.md): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md) or a [localdrive.md](helpers/localdrive.md) into another one. 4. [secretstream.md](helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. Holepunch uses this in Keet to store chat messages and in Hypercore's replication protocol. +5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. 6. [protomux.md](helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. ## Tools built using Holepunch From 78b531703546f77061db48126f03f167c7befbd9 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 14:38:16 +0530 Subject: [PATCH 28/35] Update quick-start.md resolved conservations. --- hp-docs/quick-start.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index 69635a8..895cd28 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -39,15 +39,16 @@ npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a de ### Hyperswarm's DHT: Connecting Two Peers by Key -[Hyperswarm](building-blocks/hyperswarm.md) used to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. +[Hyperswarm](building-blocks/hyperswarm.md) helps to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. -In the HyperDHT, peers are identified by a public key, not by an IP address. If you know someone's public key, you can connect to them regardless of where they're located, even if they move between different networks. +In the HyperDHT, peers are identified by a public key, not by an IP address. With the public key, users can connect to each other irrespective of their location, even if they move between different networks. > ℹ️ Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. -For example, Keet implements its relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. +> +> For example, Keet implements its relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. -For example, use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. +Use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. `server.mjs` will create a key pair and then start a server that will listen on the generated key pair. The public key is logged into the console. Copy it for instantiating the client. @@ -103,15 +104,15 @@ process.stdin.pipe(conn).pipe(process.stdout) ### Hyperswarm: Connecting to Many Peers by Topic -In the above example, two peers connected directly using the first peer's public key. The frequent use of Hyperswarm is to discover peers swarming a common topic, and connect to as many of them as possible. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. +In the above example, two peers connected directly using the first peer's public key. Hyperswarm helps to discover peers swarming a common topic, and connect to as many of them as possible. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. The [Hyperswarm](building-blocks/hyperswarm.md) module provides a higher-level interface over the underlying DHT, abstracting away the mechanics of establishing and maintaining connections. Instead, 'join' topics, and the swarm discovers peers automatically. It also handles reconnections in the event of failures. -In the previous example, to explicitly indicate which peer was the server and which was the client, create two peers, have them join a common topic, and let the swarm deal with connections. +In the previous example, we needed to explicitly indicate which peer was the server and which was the client. By using Hyperswarm, we create two peers, have them join a common topic, and let the swarm deal with connections. This example consists of a single file, `peer.mjs`. In one terminal, type `node peer.mjs`, it will display the topic. Copy/paste that topic into N additional terminals with `node peer.mjs (topic)`. Each peer will log information about the other connected peers. -Start typing into any terminal anything, and it will be broadcast to all connected peers! +Start typing into any terminal, and it will be broadcast to all connected peers! From 867fc1c10570290392ff747e2942ea155e827768 Mon Sep 17 00:00:00 2001 From: ss-9984 <155604356+ss-9984@users.noreply.github.com> Date: Thu, 18 Jan 2024 14:53:14 +0530 Subject: [PATCH 29/35] Update README.md Fixed review comments. --- hp-docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 50a015a..05056d3 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -51,7 +51,7 @@ Holepunch is continuously introducing new enhancements and the following are a f ### Better Building Blocks -Holepunch focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being opinionated and introducing any operational complexity. +Our focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being opinionated and introducing any operational complexity. The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. From c3e30cee39b1a5d756b3f2287d27e8b542bff138 Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Thu, 18 Jan 2024 15:02:21 +0530 Subject: [PATCH 30/35] Update README.md --- hp-docs/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/hp-docs/README.md b/hp-docs/README.md index 05056d3..eacd10b 100644 --- a/hp-docs/README.md +++ b/hp-docs/README.md @@ -2,8 +2,6 @@ Welcome to the Holepunch docs! 👋 --- -# Overview - Holepunch equips developers with a powerful suite of independent components to effortlessly construct P2P applications. ## Building blocks From aaaff1e245c06a4702bebb15be397522d253499c Mon Sep 17 00:00:00 2001 From: Ribhav28 <153540042+Ribhav28@users.noreply.github.com> Date: Thu, 18 Jan 2024 15:03:56 +0530 Subject: [PATCH 31/35] Update quick-start.md --- hp-docs/quick-start.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md index 895cd28..f892cc2 100644 --- a/hp-docs/quick-start.md +++ b/hp-docs/quick-start.md @@ -1,9 +1,7 @@ ---- -description: A primer on Holepunch's modular building blocks ---- - # Quick start +A primer on Holepunch's modular building blocks + * [Setup](quick-start.md#setup) * [Hyperswarm's DHT: Connecting Two Peers by Key](quick-start.md#hyperswarms-dht-connecting-two-peers-by-key) * [Hyperswarm: Connecting to Many Peers by Topic](quick-start.md#hyperswarm-connecting-to-many-peers-by-topic) From 26ca839e4d4b527f2ab557de879d609128331695 Mon Sep 17 00:00:00 2001 From: dmc Date: Fri, 19 Jan 2024 19:32:09 +0100 Subject: [PATCH 32/35] fixup --- guide/connecting-many-peers.md | 57 +++ guide/connecting-two-peers.md | 69 ++++ guide/corestore.md | 123 ++++++ guide/hyperbee.md | 183 +++++++++ guide/hypercore.md | 86 ++++ guide/hyperdrive.md | 188 +++++++++ hp-docs/README.md | 110 ------ hp-docs/quick-start.md | 701 --------------------------------- readme.md | 96 ++++- 9 files changed, 801 insertions(+), 812 deletions(-) create mode 100644 guide/connecting-many-peers.md create mode 100644 guide/connecting-two-peers.md create mode 100644 guide/corestore.md create mode 100644 guide/hyperbee.md create mode 100644 guide/hypercore.md create mode 100644 guide/hyperdrive.md delete mode 100644 hp-docs/README.md delete mode 100644 hp-docs/quick-start.md diff --git a/guide/connecting-many-peers.md b/guide/connecting-many-peers.md new file mode 100644 index 0000000..11087e5 --- /dev/null +++ b/guide/connecting-many-peers.md @@ -0,0 +1,57 @@ +### Hyperswarm: Connecting to Many Peers by Topic + +Get setup by creating a project folder and installing dependencies: + +```bash +mkdir connect-two-peers +cd connect-two-peers +pear init -y -t terminal +npm install hyperswarm b4a graceful-goodbye hypercore-crypto +``` + +In the former example, two peers connected directly using the first peer's public key. Hyperswarm helps to discover peers swarming a common topic, and connect to as many of them as possible. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. + +The [Hyperswarm](../building-blocks/hyperswarm.md) module provides a higher-level interface over the underlying DHT, abstracting away the mechanics of establishing and maintaining connections. Instead, 'join' topics, and the swarm discovers peers automatically. It also handles reconnections in the event of failures. + +In the previous example, we needed to explicitly indicate which peer was the server and which was the client. By using Hyperswarm, we create two peers, have them join a common topic, and let the swarm deal with connections. + +This example consists of a single file, `peer.mjs`. In one terminal, type `node peer.mjs`, it will display the topic. Copy/paste that topic into N additional terminals with `node peer.mjs (topic)`. Each peer will log information about the other connected peers. + +Start typing into any terminal, and it will be broadcast to all connected peers. + +```javascript +//peer.mjs +import Hyperswarm from 'hyperswarm' +import goodbye from 'graceful-goodbye' +import crypto from 'hypercore-crypto' +import b4a from 'b4a' + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// Keep track of all connections and console.log incoming data +const conns = [] +swarm.on('connection', conn => { + const name = b4a.toString(conn.remotePublicKey, 'hex') + console.log('* got a connection from:', name, '*') + conns.push(conn) + conn.once('close', () => conns.splice(conns.indexOf(conn), 1)) + conn.on('data', data => console.log(`${name}: ${data}`)) +}) + +// Broadcast stdin to all connections +process.stdin.on('data', d => { + for (const conn of conns) { + conn.write(d) + } +}) + +// Join a common topic +const topic = process.argv[2] ? b4a.from(process.argv[2], 'hex') : crypto.randomBytes(32) +const discovery = swarm.join(topic, { client: true, server: true }) + +// The flushed promise will resolve when the topic has been fully announced to the DHT +discovery.flushed().then(() => { + console.log('joined topic:', b4a.toString(topic, 'hex')) +}) +``` diff --git a/guide/connecting-two-peers.md b/guide/connecting-two-peers.md new file mode 100644 index 0000000..37d1503 --- /dev/null +++ b/guide/connecting-two-peers.md @@ -0,0 +1,69 @@ + +### Hyperswarm's DHT: Connecting Two Peers by Key + +Get setup by creating a project folder and installing dependencies: + +```bash +mkdir connect-two-peers +cd connect-two-peers +pear init -y -t terminal +npm install hyperdht b4a graceful-goodbye +``` + +[Hyperswarm](../building-blocks/hyperswarm.md) helps to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. + +In the HyperDHT, peers are identified by a public key, not by an IP address. With the public key, users can connect to each other irrespective of their location, even if they move between different networks. + +> Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing [NATs](https://en.wikipedia.org/wiki/Network_address_translation), in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. + +> For example, Keet implements its relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. + +Use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. + +`server.mjs` will create a key pair and then start a server that will listen on the generated key pair. The public key is logged into the console. Copy it for instantiating the client. + + +```javascript +//server.mjs +import DHT from 'hyperdht' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +const dht = new DHT() + +// This keypair is your peer identifier in the DHT +const keyPair = DHT.keyPair() + +const server = dht.createServer(conn => { + console.log('got connection!') + process.stdin.pipe(conn).pipe(process.stdout) +}) + +server.listen(keyPair).then(() => { + console.log('listening on:', b4a.toString(keyPair.publicKey, 'hex')) +}) + +// Unnannounce the public key before exiting the process +// (This is not a requirement, but it helps avoid DHT pollution) +goodbye(() => server.close()) +``` + +`client.mjs` will spin up a client, and the public key copied earlier must be supplied as a command line argument for connecting to the server. The client process will log `got connection` into the console when it connects to the server. + +Once it's connected, try typing in both terminals! + +``` javascript +//client.mjs +import DHT from 'hyperdht' +import b4a from 'b4a' + +console.log('Connecting to:', process.argv[2]) +const publicKey = b4a.from(process.argv[2], 'hex') + +const dht = new DHT() +const conn = dht.connect(publicKey) +conn.once('open', () => console.log('got connection!')) + +process.stdin.pipe(conn).pipe(process.stdout) +``` + diff --git a/guide/corestore.md b/guide/corestore.md new file mode 100644 index 0000000..b89a56b --- /dev/null +++ b/guide/corestore.md @@ -0,0 +1,123 @@ + +### Corestore: Working with Many Hypercores + +Get setup by creating a project folder and installing dependencies: + +```bash +mkdir many-cores +cd many-cores +pear init -y -t terminal +npm install corestore hyperswarm b4a graceful-goodbye +``` + +An append-only log is powerful on its own, but it's most useful as a building block for constructing larger data structures, such as databases or filesystems. Building these data structures often requires many cores, each with different responsibilities. For example, Hyperdrive uses one core to store file metadata and another to store file contents. + +[corestore.md](../helpers/corestore.md) is a Hypercore factory that makes it easier to manage large collections of named Hypercores. A simple example below demonstrates a pattern often in use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. + +This example consists of two files: `writer.js` and `reader.js`. In the previous example, we replicated only a single Hypercore instance. But in this example, we will replicate a single Corestore instance, which will internally manage the replication of a collection of Hypercores. + +The file `writer.js` uses a Corestore instance to create three Hypercores, which are then replicated with other peers using Hyperswarm. The keys for the second and third cores are stored in the first core (the first core 'bootstraps' the system). Messages entered into the command line are written into the second and third cores, depending on the length of the message. To execute `reader.js`, copy the main core key logged into the command line. + + +```javascript +//writer.js +import Corestore from 'corestore' +import Hyperswarm from 'hyperswarm' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +const store = new Corestore('./writer-storage') +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// A name is a purely-local, and maps to a key pair. It's not visible to readers. +// Since a name always corresponds to a key pair, these are all writable +const core1 = store.get({ name: 'core-1', valueEncoding: 'json' }) +const core2 = store.get({ name: 'core-2' }) +const core3 = store.get({ name: 'core-3' }) +await Promise.all([core1.ready(), core2.ready(), core3.ready()]) + +console.log('main core key:', b4a.toString(core1.key, 'hex')) + +// Here we'll only join the swarm with the core1's discovery key +// We don't need to announce core2 and core3, because they'll replicated with core1 +swarm.join(core1.discoveryKey) + +// Corestore replication internally manages to replicate every loaded core +// Corestore *does not* exchange keys (read capabilities) during replication. +swarm.on('connection', conn => store.replicate(conn)) + +// Since Corestore does not exchange keys, they need to be exchanged elsewhere. +// Here, we'll record the other keys in the first block of core1. +if (core1.length === 0) { + await core1.append({ + otherKeys: [core2, core3].map(core => b4a.toString(core.key, 'hex')) + }) +} + +// Record all short messages in core2, and all long ones in core3 +process.stdin.on('data', data => { + if (data.length < 5) { + console.log('appending short data to core2') + core2.append(data) + } else { + console.log('appending long data to core3') + core3.append(data) + } +}) +``` + +`reader.js` connects to the previous peer with Hyperswarm and replicates the local Corestore instance to receive the data from it. This requires the copied key to be supplied as an argument when executing the file, which will then be used to create a core with the same public key as the other peer (i.e., the same discovery key for both the reader and writer peers). + + +```javascript +//reader.js +import Corestore from 'corestore' +import Hyperswarm from 'hyperswarm' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +// pass the key as a command line argument +const key = b4a.from(process.argv[2], 'hex') + +// creation of a Corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of corestore instance on every connection +swarm.on('connection', conn => store.replicate(conn)) + +// creation/getting of a hypercore instance using the key passed +const core = store.get({ key, valueEncoding: 'json' }) +// wait till all the properties of the hypercore instance are initialized +await core.ready() + +const foundPeers = store.findingPeers() +// join a topic +swarm.join(core.discoveryKey) +swarm.flush().then(() => foundPeers()) + +// update the meta-data of the hypercore instance +await core.update() + +if (core.length === 0) { + console.log('Could not connect to the writer peer') + process.exit(1) +} + +// getting cores using the keys stored in the first block of main core +const { otherKeys } = await core.get(0) +for (const key of otherKeys) { + const core = store.get({ key: b4a.from(key, 'hex') }) + // on every append to the hypercore, + // download the latest block of the core and log it to the console + core.on('append', () => { + const seq = core.length - 1 + core.get(seq).then(block => { + console.log(`Block ${seq} in Core ${key}: ${block}`) + }) + }) +} +``` diff --git a/guide/hyperbee.md b/guide/hyperbee.md new file mode 100644 index 0000000..b01333a --- /dev/null +++ b/guide/hyperbee.md @@ -0,0 +1,183 @@ + +### Hyperbee: Sharing Append-Only Databases +Get setup by creating a project folder and installing dependencies: + +```bash +mkdir connect-two-peers +cd connect-two-peers +pear init -y -t terminal +npm install hyperdht b4a graceful-goodbye +``` + +[hyperbee.md](../building-blocks/hyperbee.md) is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. + +The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-reader.mjs`. + +`writer.mjs` stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and download the dictionary data into their local Hyperbee instances. + +> Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. + +```javascript +//writer.mjs +import fs from 'fs' +import zlib from 'zlib' + +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import Hyperbee from 'hyperbee' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +// create a corestore instance with the given location +const store = new Corestore('./writer-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of corestore instance +swarm.on('connection', conn => store.replicate(conn)) + +// creation of Hypercore instance (if not already created) +const core = store.get({ name: 'my-bee-core' }) + +// creation of Hyperbee instance using the core instance +const bee = new Hyperbee(core, { + keyEncoding: 'utf-8', + valueEncoding: 'utf-8' +}) + +// wait till all the properties of the hypercore are initialized +await core.ready() + +// join a topic +const discovery = swarm.join(core.discoveryKey) + +// Only display the key once the Hyperbee has been announced to the DHT +discovery.flushed().then(() => { + console.log('bee key:', b4a.toString(core.key, 'hex')) +}) + +// Only import the dictionary the first time this script is executed +// The first block will always be the Hyperbee header block +if (core.length <= 1) { + console.log('importing dictionary...') + const dict = await loadDictionary() + const batch = bee.batch() + for (const { key, value } of dict) { + await batch.put(key, value) + } + await batch.flush() +} else { + // Otherwise just seed the previously-imported dictionary + console.log('seeding dictionary...') +} + +async function loadDictionary() { + const compressed = await fs.promises.readFile('./dict.json.gz') + return new Promise((resolve, reject) => { + // unzip the compressed file and return the content + zlib.unzip(compressed, (err, dict) => { + if (err) return reject(err) + return resolve(JSON.parse(b4a.toString(dict))) + }) + }) +} +``` + + +`bee-reader.mjs` creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as the above file. On every word entered in the command line, it will download the respective data to the local Hyperbee instance. + +Try looking at disk space the `reader-storage` directory is using after each query. notice that it's significantly smaller than `writer-storage`! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call **sparse downloading.** + +```javascript +bee-reader.mjs +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import Hyperbee from 'hyperbee' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +// creation of a corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of the corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create or get the hypercore using the public key supplied as command-line argument +const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) + +// create a hyperbee instance using the hypercore instance +const bee = new Hyperbee(core, { + keyEncoding: 'utf-8', + valueEncoding: 'utf-8' +}) + +// wait till the hypercore properties to be intialized +await core.ready() + +// logging the public key of the hypercore instance +console.log('core key here is:', core.key.toString('hex')) + +// Attempt to connect to peers +swarm.join(core.discoveryKey) + +// Do a single Hyperbee.get for every line of stdin data +// Each `get` will only download the blocks necessary to satisfy the query +process.stdin.setEncoding('utf-8') +process.stdin.on('data', data => { + const word = data.trim() + if (!word.length) return + bee.get(word).then(node => { + if (!node || !node.value) console.log(`No dictionary entry for ${data}`) + else console.log(`${data} -> ${node.value}`) + }, err => console.error(err)) +}) +``` + +Importantly, a Hyperbee is **just** a Hypercore, where the tree nodes are stored as Hypercore blocks. Now examine the Hyperbee as if it were just a Hypercore and log out a few blocks. + +`core-reader.mjs` will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which we can easily import and use. + + +```javascript +core-reader.mjs +import Hypercore from 'hypercore' +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +import { Node } from 'hyperbee/lib/messages.js' + +// creation of a corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of the corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create or get the hypercore using the public key supplied as command-line argument +const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) +// wait till the properties of the hypercore instance are initialized +await core.ready() + +const foundPeers = store.findingPeers() +// join a topic +swarm.join(core.discoveryKey) +swarm.flush().then(() => foundPeers()) + +// update the meta-data information of the hypercore instance +await core.update() + +const seq = core.length - 1 +const lastBlock = await core.get(core.length - 1) + +// print the information about the last block or the latest block of the hypercore instance +console.log(`Raw Block ${seq}:`, lastBlock) +console.log(`Decoded Block ${seq}`, Node.decode(lastBlock)) +``` \ No newline at end of file diff --git a/guide/hypercore.md b/guide/hypercore.md new file mode 100644 index 0000000..ad0335d --- /dev/null +++ b/guide/hypercore.md @@ -0,0 +1,86 @@ +### Hypercore: The Basics + +Get setup by creating a project folder and installing dependencies: + +```bash +mkdir hypercore-basics +cd hypercore-basics +pear init -y -t terminal +npm install hyperswarm hypercore b4a graceful-goodbye +``` + +In the Hyperswarm examples, peers can exchange chat messages so long as both are online at the same time and directly connected, and those messages are not persistent (they will be lost if the recipient is offline). Hypercore fixes all of these problems. + +[hypercore.md](../building-blocks/hypercore.md) is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications. + +Now extend the ephemeral chat example above but using Hypercore to add many significant new features: + +1. **Persistence**: The owner of the Hypercore can add messages at any time, and they'll be persisted to disk. Whenever they come online, readers can replicate these messages over Hyperswarm. +2. **Many Readers:** New messages added to the Hypercore will be broadcast to interested readers. The owner gives each reader a reading capability (`core.key`) and a corresponding discovery key (`core.discoveryKey`). The former is used to authorize the reader, ensuring that they have permission to read messages, and the latter is used to discover the owner (and other readers) on the swarm. + +The following example consists of two files: `reader.js` and `writer.js`. When these two files are executed (run using node), two peers are created and connected. A Hypercore is used to store the data entered into the command line. + +The `writer.js` code stores the data entered into the command line to the Hypercore instance. The Hypercore instance is replicated with other peers using Hyperswarm. + + +```javascript +//writer.js +import Hyperswarm from 'hyperswarm' +import Hypercore from 'hypercore' +import goodbye from 'graceful-goodbye' +import b4a from 'b4a' + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +const core = new Hypercore('./writer-storage') + +// core.key and core.discoveryKey will only be set after core.ready resolves +await core.ready() +console.log('hypercore key:', b4a.toString(core.key, 'hex')) + +// Append all stdin data as separate blocks to the core +process.stdin.on('data', data => core.append(data)) + +// core.discoveryKey is *not* a read capability for the core +// It's only used to discover other peers who *might* have the core +swarm.join(core.discoveryKey) +swarm.on('connection', conn => core.replicate(conn)) +``` + + +`reader.js` uses Hyperswarm to connect to the previously initiated peer and synchronize the local Hypercore instance with the Hypercore instance of the writer. + +```javascript +//reader.js +import Hyperswarm from 'hyperswarm' +import Hypercore from 'hypercore' +import goodbye from 'graceful-goodbye' + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +const core = new Hypercore('./reader-storage', process.argv[2]) +await core.ready() + +const foundPeers = core.findingPeers() +swarm.join(core.discoveryKey) +swarm.on('connection', conn => core.replicate(conn)) + +// swarm.flush() will wait until *all* discoverable peers have been connected to +// It might take a while, so don't await it +// Instead, use core.findingPeers() to mark when the discovery process is completed +swarm.flush().then(() => foundPeers()) + +// This won't resolve until either +// a) the first peer is found +// or b) no peers could be found +await core.update() + +let position = core.length +console.log(`Skipping ${core.length} earlier blocks...`) +for await (const block of core.createReadStream({ start: core.length, live: true })) { + console.log(`Block ${position++}: ${block}`) +} +``` + diff --git a/guide/hyperdrive.md b/guide/hyperdrive.md new file mode 100644 index 0000000..05162f1 --- /dev/null +++ b/guide/hyperdrive.md @@ -0,0 +1,188 @@ + + +### Hyperdrive: A Full P2P Filesystem + +Get setup by creating a project folder and installing dependencies: + +```bash +mkdir p2p-filesystem +cd p2p-filesystem +pear init -y -t terminal +npm install hyperswarm hyperdrive localdrive corestore debounceify b4a graceful-goodbye +``` + +[hyperdrive.md](../building-blocks/hyperdrive.md) is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. + +Now mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, use two additional tools: [mirrordrive.md](../helpers/mirrordrive.md) and [localdrive.md](../helpers/localdrive.md), which handle all interactions between Hyperdrives and the local filesystem. + +This example consists of three files: `writer.mjs`, `drive-reader.mjs` and `bee-reader.mjs`. + +`writer.mjs` creates a local drive instance for a local directory and then mirrors the local drive into the Hyperdrive instance. The store used to create the Hyperdrive instance is replicated using Hyperswarm to make the data of Hyperdrive accessible to other peers. Copy the drive key logged into the command line for the `reader.mjs` execution. + + +```javascript +writer.js +import Hyperswarm from 'hyperswarm' +import Hyperdrive from 'hyperdrive' +import Localdrive from 'localdrive' +import Corestore from 'corestore' +import goodbye from 'graceful-goodbye' +import debounce from 'debounceify' +import b4a from 'b4a' + +// create a Corestore instance +const store = new Corestore('./writer-storage') +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of the corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// A local drive provides a Hyperdrive interface to a local directory +const local = new Localdrive('./writer-dir') + +// A Hyperdrive takes a Corestore because it needs to create many cores +// One for a file metadata Hyperbee, and one for a content Hypercore +const drive = new Hyperdrive(store) + +// wait till the properties of the hyperdrive instance are initialized +await drive.ready() + +// Import changes from the local drive into the Hyperdrive +const mirror = debounce(mirrorDrive) + +const discovery = swarm.join(drive.discoveryKey) +await discovery.flushed() + +console.log('drive key:', b4a.toString(drive.key, 'hex')) + +// start the mirroring process (i.e copying) of content from writer-dir to the drive +// whenever something is entered (other than '/n' or Enter )in the command-line +process.stdin.setEncoding('utf-8') +process.stdin.on('data', (d) => { + if (!d.match('\n')) return + mirror() +}) + +// this function copies the contents from writer-dir directory to the drive +async function mirrorDrive () { + console.log('started mirroring changes from \'./writer-dir\' into the drive...') + const mirror = local.mirror(drive) + await mirror.done() + console.log('finished mirroring:', mirror.count) +} +``` + +`drive-reader.mjs` creates a local drive instance for a local directory and then mirrors the contents of the local Hyperdrive instance into the local drive instance (which will write the contents to the local directory). + +Try running `node drive-reader.mjs (key-from-above)`, then add/remove/modify files inside `writer-dir` then press `Enter` in the writer's terminal (to import the local changes into the writer's drive). Observe that all new changes mirror into `reader-dir`. + + +```javascript +drive-reader.mjs +import Hyperswarm from 'hyperswarm' +import Hyperdrive from 'hyperdrive' +import Localdrive from 'localdrive' +import Corestore from 'corestore' +import goodbye from 'graceful-goodbye' +import debounce from 'debounceify' +import b4a from 'b4a' + +// create a Corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replication of store on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create a local copy of the remote drive +const local = new Localdrive('./reader-dir') + +// create a hyperdrive using the public key passed as a command-line argument +const drive = new Hyperdrive(store, b4a.from(process.argv[2], 'hex')) + +// wait till all the properties of the drive are initialized +await drive.ready() + +const mirror = debounce(mirrorDrive) + +// call the mirror function whenever content gets appended +// to the Hypercore instance of the hyperdrive +drive.core.on('append', mirror) + +const foundPeers = store.findingPeers() + +// join a topic +swarm.join(drive.discoveryKey, { client: true, server: false }) +swarm.flush().then(() => foundPeers()) + +// start the mirroring process (i.e copying the contents from remote drive to local dir) +mirror() + +async function mirrorDrive () { + console.log('started mirroring remote drive into \'./reader-dir\'...') + const mirror = drive.mirror(local) + await mirror.done() + console.log('finished mirroring:', mirror.count) +} +``` + + +Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee (which is **just** a Hypercore). Now inspect the Hyperdrive as though it were a Hyperbee, and log out some file metadata. + +`bee-reader.mjs` creates a Hyperbee instance using the Hypercore instance created with the copied public key. Every time the Hyperbee is updated (an `append` event is emitted on the underlying Hypercore), all file metadata nodes will be logged out. + +Try adding or removing a few files from the writer's data directory, then pressing `Enter` in the writer's terminal to mirror the changes. + + +```javascript +bee-reader.mjs +import Hyperswarm from 'hyperswarm' +import Corestore from 'corestore' +import Hyperbee from 'hyperbee' +import goodbye from 'graceful-goodbye' +import debounce from 'debounceify' +import b4a from 'b4a' + +// create a Corestore instance +const store = new Corestore('./reader-storage') + +const swarm = new Hyperswarm() +goodbye(() => swarm.destroy()) + +// replicate corestore instance on connection with other peers +swarm.on('connection', conn => store.replicate(conn)) + +// create/get the hypercore instance using the public key supplied as command-line arg +const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) + +// create a hyperbee instance using the hypercore instance +const bee = new Hyperbee(core, { + keyEncoding: 'utf-8', + valueEncoding: 'json' +}) + +// wait till the properties of the hypercore instance are initialized +await core.ready() + +const foundPeers = store.findingPeers() +swarm.join(core.discoveryKey) +swarm.flush().then(() => foundPeers()) + +// execute the listBee function whenever the data is appended to the underlying hypercore +core.on('append', listBee) + +listBee() + +// listBee function will list the key-value pairs present in the hyperbee instance +async function listBee () { + console.log('\n***************') + console.log('hyperbee contents are now:') + for await (const node of bee.createReadStream()) { + console.log(' ', node.key, '->', node.value) + } +} +``` + diff --git a/hp-docs/README.md b/hp-docs/README.md deleted file mode 100644 index eacd10b..0000000 --- a/hp-docs/README.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -Welcome to the Holepunch docs! 👋 ---- - -Holepunch equips developers with a powerful suite of independent components to effortlessly construct P2P applications. - -## Building blocks - -The Holepunch Ecosystem is constructed utilizing the following structural components. - -1. [hypercore.md](building-blocks/hypercore.md): A distributed, secure append-only log is a tool for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. -2. [hyperbee.md](building-blocks/hyperbee.md): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. -3. [hyperdrive.md](building-blocks/hyperdrive.md): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. -4. [autobase.md](building-blocks/autobase.md): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. -5. [hyperdht.md](building-blocks/hyperdht.md): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. -6. [hyperswarm.md](building-blocks/hyperswarm.md): A high-level API for finding and connecting to peers who are interested in a "topic." - -## Helpers - -Helper modules can be used together with the building blocks to create cutting-edge P2P tools and applications. - -1. [corestore.md](helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. -2. [localdrive.md](helpers/localdrive.md): A file system interoperable with Hyperdrive. -3. [mirrordrive.md](helpers/mirrordrive.md): Mirror a [hyperdrive.md](building-blocks/hyperdrive.md) or a [localdrive.md](helpers/localdrive.md) into another one. -4. [secretstream.md](helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. -6. [protomux.md](helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. - -## Tools built using Holepunch - -| Tools | Description | -| :----------------------------------------------------------: | :---------------------------------------------------------: | -| **[Hypershell](https://docs.holepunch.to/tools/hypershell)** | A CLI to create and connect to P2P E2E encrypted shells.. | -| **[Hypertele](https://docs.holepunch.to/tools/hypertele)** | A swiss-knife proxy powered by [HyperDHT](https://docs.holepunch.to/building-blocks/hyperdht). | -| **[Hyperbeam](https://docs.holepunch.to/tools/hyperbeam)** | A one-to-one and end-to-end encrypted internet pipe. | -| **[Hyperssh](https://docs.holepunch.to/tools/hyperssh)** | A CLI to run SSH over the [HyperDHT](https://docs.holepunch.to/building-blocks/hyperdht). | -| **[Drives](https://docs.holepunch.to/tools/drives)** | CLI to download, seed, and mirror a [hyperdrive](https://docs.holepunch.to/building-blocks/hyperdrive) or a [localdrive](https://docs.holepunch.to/helpers/localdrive). | - -> ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). - - -## Applications built using Holepunch - -* [keet.io.md](apps/keet.io.md)**:** A peer-to-peer chat and video-conferencing application with end-to-end encryption. - -## What's new? - -Holepunch is continuously introducing new enhancements and the following are a few highlights. - -### Better Building Blocks - -Our focus is on making the core building blocks easy to use, fast, and reliable. The goal is to give developers all the essential pieces to make powerful P2P apps without being opinionated and introducing any operational complexity. - -The core modules Hyperspace daemon and `hyp` CLI tools are now deprecated. - -### Hypercore - -* The [`session`](building-blocks/hypercore.md#core.session-options) and [`snapshot`](building-blocks/hypercore.md#core.snapshot-options) methods for providing multiple views over the same underlying Hypercore, which simplifies resource management. -* A [`truncate`](building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](building-blocks/autobase.md), as described below. -* An improved fork detection in the replication protocol, to improve resilience. -* Optional on-disk encryption for blocks (in addition to the existing transport encryption). -* The storage layer now uses a write-ahead log to ensure that power loss or unexpected shutdown cannot lead to data corruption. - -### Hyperswarm - -* An improved UDP holepunching algorithm that uses arbitrary DHT nodes (optionally selected by the connecting peers) to proxy necessary metadata while being maximally privacy-preserving. -* A custom-built transport protocol, [UDX](https://github.com/hyperswarm/libudx), that takes advantage of the holepunching algorithm to avoid unnecessary overhead (it doesn't include handshaking since holepunching takes care of that, for example). It's blazing fast. -* A simplified DHT API that closely resembles NodeJS's `net` module, but using public keys instead of IPs. - -### Hyperdrive - -* Uses Hyperbee internally for storing file metadata -* Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. -* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. - -### Autobase (experimental) - -Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view members see on the screen. - -As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. - -> ⚠️ Autobase is still experimental and is likely to change significantly in the near future. If you're feeling adventurous, give it a shot! - - -## Stability indexing - -Throughout the documentation, indications of a module's stability are provided. Some modules are well-established and used widely, making them highly unlikely to ever change. Other modules may be new, experimental, or known to have risks associated with their use. - -The following stability indices have been used: - -| Stability | Description | -| :----------------------------------------------------------: | :---------------------------------------------------------: | -| **stable** | Unlikely to change or be removed in the foreseeable future. | -| **experimental** | New, untested, or have known issues. | -| **deprecated** | Being removed or replaced in the future. | -| **unstable** | May change or be removed without warning. | - -#### Stability overview - -| Module | Stability | -| -------------------------------------------------------- | :----------------------------------------------------------: | -| [hypercore.md](building-blocks/hypercore.md) | **stable** | -| [hyperbee.md](building-blocks/hyperbee.md) | **stable** | -| [hyperdrive.md](building-blocks/hyperdrive.md) | **stable** | -| [autobase.md](building-blocks/autobase.md) | **experimental** | -| [hyperswarm.md](building-blocks/hyperswarm.md) | **stable** | -| [hyperdht.md](building-blocks/hyperdht.md) | **stable** | - - - >⚠️ Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. diff --git a/hp-docs/quick-start.md b/hp-docs/quick-start.md deleted file mode 100644 index f892cc2..0000000 --- a/hp-docs/quick-start.md +++ /dev/null @@ -1,701 +0,0 @@ -# Quick start - -A primer on Holepunch's modular building blocks - -* [Setup](quick-start.md#setup) -* [Hyperswarm's DHT: Connecting Two Peers by Key](quick-start.md#hyperswarms-dht-connecting-two-peers-by-key) -* [Hyperswarm: Connecting to Many Peers by Topic](quick-start.md#hyperswarm-connecting-to-many-peers-by-topic) -* [Hypercore: The Basics](quick-start.md#hypercore-the-basics) -* [Corestore: Working with Many Hypercores](quick-start.md#corestore-working-with-many-hypercores) -* [Hyperbee: Sharing Append-Only Databases](quick-start.md#hyperbee-sharing-append-only-databases) -* [Hyperdrive: A Full P2P Filesystem](quick-start.md#hyperdrive-a-full-p2p-filesystem) - -### Setup - -> ⚠️ Before beginning the setup, ensure sure the system is running on Node v16 or greater. - - -Download the [Holepunch examples repo](https://github.com/holepunchto/examples) - -```bash -git clone https://github.com/holepunchto/examples -``` - -Install the required dependencies: - -```bash -cd examples -npm install -``` - - -```bash -npm install hyperswarm hypercore corestore hyperbee hyperdrive localdrive b4a debounceify graceful-goodbye --save -``` - -> ℹ️ Every code example in this page is meant to be run standalone, so copy/paste each example into a JS file, and run it with NodeJS. - -### Hyperswarm's DHT: Connecting Two Peers by Key - -[Hyperswarm](building-blocks/hyperswarm.md) helps to find and connect to peers who are announcing a common 'topic'. The swarm topic can be anything. The HyperDHT uses a series of holepunching techniques to establish direct connections between peers, even if they're located on home networks with tricky NATs. - -In the HyperDHT, peers are identified by a public key, not by an IP address. With the public key, users can connect to each other irrespective of their location, even if they move between different networks. - -> ℹ️ Hyperswarm's holepunching will fail if both the client peer and the server peer are on randomizing NATs, in which case the connection must be relayed through a third peer. Hyperswarm does not do any relaying by default. -> -> For example, Keet implements its relaying system wherein other call participants can serve as relays -- the more participants in the call, the stronger overall connectivity becomes. - - -Use the HyperDHT to create a basic CLI chat app where a client peer connects to a server peer by public key. This example consists of two files: `client.mjs` and `server.mjs`. - -`server.mjs` will create a key pair and then start a server that will listen on the generated key pair. The public key is logged into the console. Copy it for instantiating the client. - - -```javascript -server.mjs -import DHT from 'hyperdht' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -const dht = new DHT() - -// This keypair is your peer identifier in the DHT -const keyPair = DHT.keyPair() - -const server = dht.createServer(conn => { - console.log('got connection!') - process.stdin.pipe(conn).pipe(process.stdout) -}) - -server.listen(keyPair).then(() => { - console.log('listening on:', b4a.toString(keyPair.publicKey, 'hex')) -}) - -// Unnannounce the public key before exiting the process -// (This is not a requirement, but it helps avoid DHT pollution) -goodbye(() => server.close()) -``` - - - -`client.mjs` will spin up a client, and the public key copied earlier must be supplied as a command line argument for connecting to the server. The client process will log `got connection` into the console when it connects to the server. - -Once it's connected, try typing in both terminals! - - - -``` javascript -client.mjs -import DHT from 'hyperdht' -import b4a from 'b4a' - -console.log('Connecting to:', process.argv[2]) -const publicKey = b4a.from(process.argv[2], 'hex') - -const dht = new DHT() -const conn = dht.connect(publicKey) -conn.once('open', () => console.log('got connection!')) - -process.stdin.pipe(conn).pipe(process.stdout) -``` - - -### Hyperswarm: Connecting to Many Peers by Topic - -In the above example, two peers connected directly using the first peer's public key. Hyperswarm helps to discover peers swarming a common topic, and connect to as many of them as possible. This will become clearer in the Hypercore example, but it's the best way to distribute peer-to-peer data structures. - -The [Hyperswarm](building-blocks/hyperswarm.md) module provides a higher-level interface over the underlying DHT, abstracting away the mechanics of establishing and maintaining connections. Instead, 'join' topics, and the swarm discovers peers automatically. It also handles reconnections in the event of failures. - -In the previous example, we needed to explicitly indicate which peer was the server and which was the client. By using Hyperswarm, we create two peers, have them join a common topic, and let the swarm deal with connections. - -This example consists of a single file, `peer.mjs`. In one terminal, type `node peer.mjs`, it will display the topic. Copy/paste that topic into N additional terminals with `node peer.mjs (topic)`. Each peer will log information about the other connected peers. - -Start typing into any terminal, and it will be broadcast to all connected peers! - - - -```javascript -peer.mjs -import Hyperswarm from 'hyperswarm' -import goodbye from 'graceful-goodbye' -import crypto from 'hypercore-crypto' -import b4a from 'b4a' - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// Keep track of all connections and console.log incoming data -const conns = [] -swarm.on('connection', conn => { - const name = b4a.toString(conn.remotePublicKey, 'hex') - console.log('* got a connection from:', name, '*') - conns.push(conn) - conn.once('close', () => conns.splice(conns.indexOf(conn), 1)) - conn.on('data', data => console.log(`${name}: ${data}`)) -}) - -// Broadcast stdin to all connections -process.stdin.on('data', d => { - for (const conn of conns) { - conn.write(d) - } -}) - -// Join a common topic -const topic = process.argv[2] ? b4a.from(process.argv[2], 'hex') : crypto.randomBytes(32) -const discovery = swarm.join(topic, { client: true, server: true }) - -// The flushed promise will resolve when the topic has been fully announced to the DHT -discovery.flushed().then(() => { - console.log('joined topic:', b4a.toString(topic, 'hex')) -}) -``` - -### Hypercore: The Basics - -In the Hyperswarm examples, peers can exchange chat messages so long as both are online at the same time and directly connected, and those messages are not persistent (they will be lost if the recipient is offline). Hypercore fixes all of these problems. - -[hypercore.md](building-blocks/hypercore.md) is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications. - -Now extend the ephemeral chat example above but using Hypercore to add many significant new features: - -1. **Persistence**: The owner of the Hypercore can add messages at any time, and they'll be persisted to disk. Whenever they come online, readers can replicate these messages over Hyperswarm. -2. **Many Readers:** New messages added to the Hypercore will be broadcast to interested readers. The owner gives each reader a reading capability (`core.key`) and a corresponding discovery key (`core.discoveryKey`). The former is used to authorize the reader, ensuring that they have permission to read messages, and the latter is used to discover the owner (and other readers) on the swarm. - -The following example consists of two files: `reader.mjs` and `writer.mjs`. When these two files are executed (run using node), two peers are created and connected. A Hypercore is used to store the data entered into the command line. - -`writer.mjs` stores the data entered into the command line to the Hypercore instance. The Hypercore instance is replicated with other peers using Hyperswarm. - - - -```javascript -writer.mjs -import Hyperswarm from 'hyperswarm' -import Hypercore from 'hypercore' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -const core = new Hypercore('./writer-storage') - -// core.key and core.discoveryKey will only be set after core.ready resolves -await core.ready() -console.log('hypercore key:', b4a.toString(core.key, 'hex')) - -// Append all stdin data as separate blocks to the core -process.stdin.on('data', data => core.append(data)) - -// core.discoveryKey is *not* a read capability for the core -// It's only used to discover other peers who *might* have the core -swarm.join(core.discoveryKey) -swarm.on('connection', conn => core.replicate(conn)) -``` - - -`reader.mjs` uses Hyperswarm to connect to the previously initiated peer and synchronize the local Hypercore instance with the Hypercore instance of the writer. - - - -```javascript -reader.mjs -import Hyperswarm from 'hyperswarm' -import Hypercore from 'hypercore' -import goodbye from 'graceful-goodbye' - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -const core = new Hypercore('./reader-storage', process.argv[2]) -await core.ready() - -const foundPeers = core.findingPeers() -swarm.join(core.discoveryKey) -swarm.on('connection', conn => core.replicate(conn)) - -// swarm.flush() will wait until *all* discoverable peers have been connected to -// It might take a while, so don't await it -// Instead, use core.findingPeers() to mark when the discovery process is completed -swarm.flush().then(() => foundPeers()) - -// This won't resolve until either -// a) the first peer is found -// or b) no peers could be found -await core.update() - -let position = core.length -console.log(`Skipping ${core.length} earlier blocks...`) -for await (const block of core.createReadStream({ start: core.length, live: true })) { - console.log(`Block ${position++}: ${block}`) -} -``` - - -### Corestore: Working with Many Hypercores - -An append-only log is powerful on its own, but it's most useful as a building block for constructing larger data structures, such as databases or filesystems. Building these data structures often requires many cores, each with different responsibilities. For example, Hyperdrive uses one core to store file metadata and another to store file contents. - -[corestore.md](helpers/corestore.md) is a Hypercore factory that makes it easier to manage large collections of named Hypercores. A simple example below demonstrates a pattern often in use: co-replicating many cores using Corestore, where several 'internal cores' are linked to from a primary core. Only the primary core is announced on the swarm -- the keys for the others are recorded inside of that core. - -This example consists of two files: `writer.mjs` and `reader.mjs`. In the previous example, we replicated only a single Hypercore instance. But in this example, we will replicate a single Corestore instance, which will internally manage the replication of a collection of Hypercores. - -The file `writer.mjs` uses a Corestore instance to create three Hypercores, which are then replicated with other peers using Hyperswarm. The keys for the second and third cores are stored in the first core (the first core 'bootstraps' the system). Messages entered into the command line are written into the second and third cores, depending on the length of the message. To execute `reader.mjs`, copy the main core key logged into the command line. - - -```javascript -writer.mjs -import Corestore from 'corestore' -import Hyperswarm from 'hyperswarm' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -const store = new Corestore('./writer-storage') -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// A name is a purely-local, and maps to a key pair. It's not visible to readers. -// Since a name always corresponds to a key pair, these are all writable -const core1 = store.get({ name: 'core-1', valueEncoding: 'json' }) -const core2 = store.get({ name: 'core-2' }) -const core3 = store.get({ name: 'core-3' }) -await Promise.all([core1.ready(), core2.ready(), core3.ready()]) - -console.log('main core key:', b4a.toString(core1.key, 'hex')) - -// Here we'll only join the swarm with the core1's discovery key -// We don't need to announce core2 and core3, because they'll replicated with core1 -swarm.join(core1.discoveryKey) - -// Corestore replication internally manages to replicate every loaded core -// Corestore *does not* exchange keys (read capabilities) during replication. -swarm.on('connection', conn => store.replicate(conn)) - -// Since Corestore does not exchange keys, they need to be exchanged elsewhere. -// Here, we'll record the other keys in the first block of core1. -if (core1.length === 0) { - await core1.append({ - otherKeys: [core2, core3].map(core => b4a.toString(core.key, 'hex')) - }) -} - -// Record all short messages in core2, and all long ones in core3 -process.stdin.on('data', data => { - if (data.length < 5) { - console.log('appending short data to core2') - core2.append(data) - } else { - console.log('appending long data to core3') - core3.append(data) - } -}) -``` - - -`reader.mjs` connects to the previous peer with Hyperswarm and replicates the local Corestore instance to receive the data from it. This requires the copied key to be supplied as an argument when executing the file, which will then be used to create a core with the same public key as the other peer (i.e., the same discovery key for both the reader and writer peers). - - -```javascript -reader.mjs -import Corestore from 'corestore' -import Hyperswarm from 'hyperswarm' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -// pass the key as a command line argument -const key = b4a.from(process.argv[2], 'hex') - -// creation of a Corestore instance -const store = new Corestore('./reader-storage') - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replication of corestore instance on every connection -swarm.on('connection', conn => store.replicate(conn)) - -// creation/getting of a hypercore instance using the key passed -const core = store.get({ key, valueEncoding: 'json' }) -// wait till all the properties of the hypercore instance are initialized -await core.ready() - -const foundPeers = store.findingPeers() -// join a topic -swarm.join(core.discoveryKey) -swarm.flush().then(() => foundPeers()) - -// update the meta-data of the hypercore instance -await core.update() - -if (core.length === 0) { - console.log('Could not connect to the writer peer') - process.exit(1) -} - -// getting cores using the keys stored in the first block of main core -const { otherKeys } = await core.get(0) -for (const key of otherKeys) { - const core = store.get({ key: b4a.from(key, 'hex') }) - // on every append to the hypercore, - // download the latest block of the core and log it to the console - core.on('append', () => { - const seq = core.length - 1 - core.get(seq).then(block => { - console.log(`Block ${seq} in Core ${key}: ${block}`) - }) - }) -} -``` - -### Hyperbee: Sharing Append-Only Databases - -[hyperbee.md](building-blocks/hyperbee.md) is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. - -The example consists of three files: `writer.mjs` , `bee-reader.mjs` and `core-reader.mjs`. - -`writer.mjs` stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and download the dictionary data into their local Hyperbee instances. - -> ℹ️ Download the `dict.json.gz` compressed file from the [GitHub repository](https://github.com/holepunchto/examples/blob/main/quick-start/hyperbee/dict.json.gz) to the folder where the `writer.mjs`is present. The compressed file contains 100K dictionary words. - -```javascript -writer.mjs -import fs from 'fs' -import zlib from 'zlib' - -import Hyperswarm from 'hyperswarm' -import Corestore from 'corestore' -import Hyperbee from 'hyperbee' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -// create a corestore instance with the given location -const store = new Corestore('./writer-storage') - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replication of corestore instance -swarm.on('connection', conn => store.replicate(conn)) - -// creation of Hypercore instance (if not already created) -const core = store.get({ name: 'my-bee-core' }) - -// creation of Hyperbee instance using the core instance -const bee = new Hyperbee(core, { - keyEncoding: 'utf-8', - valueEncoding: 'utf-8' -}) - -// wait till all the properties of the hypercore are initialized -await core.ready() - -// join a topic -const discovery = swarm.join(core.discoveryKey) - -// Only display the key once the Hyperbee has been announced to the DHT -discovery.flushed().then(() => { - console.log('bee key:', b4a.toString(core.key, 'hex')) -}) - -// Only import the dictionary the first time this script is executed -// The first block will always be the Hyperbee header block -if (core.length <= 1) { - console.log('importing dictionary...') - const dict = await loadDictionary() - const batch = bee.batch() - for (const { key, value } of dict) { - await batch.put(key, value) - } - await batch.flush() -} else { - // Otherwise just seed the previously-imported dictionary - console.log('seeding dictionary...') -} - -async function loadDictionary() { - const compressed = await fs.promises.readFile('./dict.json.gz') - return new Promise((resolve, reject) => { - // unzip the compressed file and return the content - zlib.unzip(compressed, (err, dict) => { - if (err) return reject(err) - return resolve(JSON.parse(b4a.toString(dict))) - }) - }) -} -``` - - -`bee-reader.mjs` creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as the above file. On every word entered in the command line, it will download the respective data to the local Hyperbee instance. - -Try looking at disk space the `reader-storage` directory is using after each query. notice that it's significantly smaller than `writer-storage`! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call **sparse downloading.** - -```javascript -bee-reader.mjs -import Hyperswarm from 'hyperswarm' -import Corestore from 'corestore' -import Hyperbee from 'hyperbee' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -// creation of a corestore instance -const store = new Corestore('./reader-storage') - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replication of the corestore instance on connection with other peers -swarm.on('connection', conn => store.replicate(conn)) - -// create or get the hypercore using the public key supplied as command-line argument -const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) - -// create a hyperbee instance using the hypercore instance -const bee = new Hyperbee(core, { - keyEncoding: 'utf-8', - valueEncoding: 'utf-8' -}) - -// wait till the hypercore properties to be intialized -await core.ready() - -// logging the public key of the hypercore instance -console.log('core key here is:', core.key.toString('hex')) - -// Attempt to connect to peers -swarm.join(core.discoveryKey) - -// Do a single Hyperbee.get for every line of stdin data -// Each `get` will only download the blocks necessary to satisfy the query -process.stdin.setEncoding('utf-8') -process.stdin.on('data', data => { - const word = data.trim() - if (!word.length) return - bee.get(word).then(node => { - if (!node || !node.value) console.log(`No dictionary entry for ${data}`) - else console.log(`${data} -> ${node.value}`) - }, err => console.error(err)) -}) -``` - -Importantly, a Hyperbee is **just** a Hypercore, where the tree nodes are stored as Hypercore blocks. Now examine the Hyperbee as if it were just a Hypercore and log out a few blocks. - -`core-reader.mjs` will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which we can easily import and use. - - -```javascript -core-reader.mjs -import Hypercore from 'hypercore' -import Hyperswarm from 'hyperswarm' -import Corestore from 'corestore' -import goodbye from 'graceful-goodbye' -import b4a from 'b4a' - -import { Node } from 'hyperbee/lib/messages.js' - -// creation of a corestore instance -const store = new Corestore('./reader-storage') - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replication of the corestore instance on connection with other peers -swarm.on('connection', conn => store.replicate(conn)) - -// create or get the hypercore using the public key supplied as command-line argument -const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) -// wait till the properties of the hypercore instance are initialized -await core.ready() - -const foundPeers = store.findingPeers() -// join a topic -swarm.join(core.discoveryKey) -swarm.flush().then(() => foundPeers()) - -// update the meta-data information of the hypercore instance -await core.update() - -const seq = core.length - 1 -const lastBlock = await core.get(core.length - 1) - -// print the information about the last block or the latest block of the hypercore instance -console.log(`Raw Block ${seq}:`, lastBlock) -console.log(`Decoded Block ${seq}`, Node.decode(lastBlock)) -``` - -### Hyperdrive: A Full P2P Filesystem - -[hyperdrive.md](building-blocks/hyperdrive.md) is a secure, real-time distributed file system designed for easy P2P file sharing. In the same way that a Hyperbee is just a wrapper around a Hypercore, a Hyperdrive is a wrapper around two Hypercores: one is a Hyperbee index for storing file metadata, and the other is used to store file contents. - -Now mirror a local directory into a Hyperdrive, replicate it with a reader peer, who then mirrors it into their own local copy. When the writer modifies its drive, by adding, removing, or changing files, the reader's local copy will be updated to reflect that. To do this, use two additional tools: [mirrordrive.md](helpers/mirrordrive.md) and [localdrive.md](helpers/localdrive.md), which handle all interactions between Hyperdrives and the local filesystem. - -This example consists of three files: `writer.mjs`, `drive-reader.mjs` and `bee-reader.mjs`. - -`writer.mjs` creates a local drive instance for a local directory and then mirrors the local drive into the Hyperdrive instance. The store used to create the Hyperdrive instance is replicated using Hyperswarm to make the data of Hyperdrive accessible to other peers. Copy the drive key logged into the command line for the `reader.mjs` execution. - - -```javascript -writer.js -import Hyperswarm from 'hyperswarm' -import Hyperdrive from 'hyperdrive' -import Localdrive from 'localdrive' -import Corestore from 'corestore' -import goodbye from 'graceful-goodbye' -import debounce from 'debounceify' -import b4a from 'b4a' - -// create a Corestore instance -const store = new Corestore('./writer-storage') -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replication of the corestore instance on connection with other peers -swarm.on('connection', conn => store.replicate(conn)) - -// A local drive provides a Hyperdrive interface to a local directory -const local = new Localdrive('./writer-dir') - -// A Hyperdrive takes a Corestore because it needs to create many cores -// One for a file metadata Hyperbee, and one for a content Hypercore -const drive = new Hyperdrive(store) - -// wait till the properties of the hyperdrive instance are initialized -await drive.ready() - -// Import changes from the local drive into the Hyperdrive -const mirror = debounce(mirrorDrive) - -const discovery = swarm.join(drive.discoveryKey) -await discovery.flushed() - -console.log('drive key:', b4a.toString(drive.key, 'hex')) - -// start the mirroring process (i.e copying) of content from writer-dir to the drive -// whenever something is entered (other than '/n' or Enter )in the command-line -process.stdin.setEncoding('utf-8') -process.stdin.on('data', (d) => { - if (!d.match('\n')) return - mirror() -}) - -// this function copies the contents from writer-dir directory to the drive -async function mirrorDrive () { - console.log('started mirroring changes from \'./writer-dir\' into the drive...') - const mirror = local.mirror(drive) - await mirror.done() - console.log('finished mirroring:', mirror.count) -} -``` - -`drive-reader.mjs` creates a local drive instance for a local directory and then mirrors the contents of the local Hyperdrive instance into the local drive instance (which will write the contents to the local directory). - -Try running `node drive-reader.mjs (key-from-above)`, then add/remove/modify files inside `writer-dir` then press `Enter` in the writer's terminal (to import the local changes into the writer's drive). Observe that all new changes mirror into `reader-dir`. - - -```javascript -drive-reader.mjs -import Hyperswarm from 'hyperswarm' -import Hyperdrive from 'hyperdrive' -import Localdrive from 'localdrive' -import Corestore from 'corestore' -import goodbye from 'graceful-goodbye' -import debounce from 'debounceify' -import b4a from 'b4a' - -// create a Corestore instance -const store = new Corestore('./reader-storage') - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replication of store on connection with other peers -swarm.on('connection', conn => store.replicate(conn)) - -// create a local copy of the remote drive -const local = new Localdrive('./reader-dir') - -// create a hyperdrive using the public key passed as a command-line argument -const drive = new Hyperdrive(store, b4a.from(process.argv[2], 'hex')) - -// wait till all the properties of the drive are initialized -await drive.ready() - -const mirror = debounce(mirrorDrive) - -// call the mirror function whenever content gets appended -// to the Hypercore instance of the hyperdrive -drive.core.on('append', mirror) - -const foundPeers = store.findingPeers() - -// join a topic -swarm.join(drive.discoveryKey, { client: true, server: false }) -swarm.flush().then(() => foundPeers()) - -// start the mirroring process (i.e copying the contents from remote drive to local dir) -mirror() - -async function mirrorDrive () { - console.log('started mirroring remote drive into \'./reader-dir\'...') - const mirror = drive.mirror(local) - await mirror.done() - console.log('finished mirroring:', mirror.count) -} -``` - - -Just as a Hyperbee is **just** a Hypercore, a Hyperdrive is **just** a Hyperbee (which is **just** a Hypercore). Now inspect the Hyperdrive as though it were a Hyperbee, and log out some file metadata. - -`bee-reader.mjs` creates a Hyperbee instance using the Hypercore instance created with the copied public key. Every time the Hyperbee is updated (an `append` event is emitted on the underlying Hypercore), all file metadata nodes will be logged out. - -Try adding or removing a few files from the writer's data directory, then pressing `Enter` in the writer's terminal to mirror the changes. - - -```javascript -bee-reader.mjs -import Hyperswarm from 'hyperswarm' -import Corestore from 'corestore' -import Hyperbee from 'hyperbee' -import goodbye from 'graceful-goodbye' -import debounce from 'debounceify' -import b4a from 'b4a' - -// create a Corestore instance -const store = new Corestore('./reader-storage') - -const swarm = new Hyperswarm() -goodbye(() => swarm.destroy()) - -// replicate corestore instance on connection with other peers -swarm.on('connection', conn => store.replicate(conn)) - -// create/get the hypercore instance using the public key supplied as command-line arg -const core = store.get({ key: b4a.from(process.argv[2], 'hex') }) - -// create a hyperbee instance using the hypercore instance -const bee = new Hyperbee(core, { - keyEncoding: 'utf-8', - valueEncoding: 'json' -}) - -// wait till the properties of the hypercore instance are initialized -await core.ready() - -const foundPeers = store.findingPeers() -swarm.join(core.discoveryKey) -swarm.flush().then(() => foundPeers()) - -// execute the listBee function whenever the data is appended to the underlying hypercore -core.on('append', listBee) - -listBee() - -// listBee function will list the key-value pairs present in the hyperbee instance -async function listBee () { - console.log('\n***************') - console.log('hyperbee contents are now:') - for await (const node of bee.createReadStream()) { - console.log(' ', node.key, '->', node.value) - } -} -``` - diff --git a/readme.md b/readme.md index 0796585..d98072d 100644 --- a/readme.md +++ b/readme.md @@ -28,4 +28,98 @@ Welcome to the Internet of Peers * [Starting a Pear Terminal Project](./guide/starting-a-pear-terminal-project.md) * [Making a Pear Terminal Application](./guide/making-a-pear-terminal-app.md) * [Sharing a Pear Application](./guide/sharing-a-pear-app.md) -* [Marking a Release](./guide/releasing-a-pear-app.md) \ No newline at end of file +* [Marking a Release](./guide/releasing-a-pear-app.md) + +## Building blocks + +The Holepunch Ecosystem is constructed utilizing the following structural components. + +1. [hypercore.md](./building-blocks/hypercore.md): A distributed, secure append-only log for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. +2. [hyperbee.md](./building-blocks/hyperbee.md): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. +3. [hyperdrive.md](./building-blocks/hyperdrive.md): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. +4. [autobase.md](./building-blocks/autobase.md): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. +5. [hyperdht.md](./building-blocks/hyperdht.md): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. +6. [hyperswarm.md](./building-blocks/hyperswarm.md): A high-level API for finding and connecting to peers who are interested in a "topic." + +## Helpers + +Helper modules can be used together with the building blocks to create cutting-edge P2P tools and applications. + +1. [corestore.md](helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. +2. [localdrive.md](helpers/localdrive.md): A file system interoperable with Hyperdrive. +3. [mirrordrive.md](helpers/mirrordrive.md): Mirror a [hyperdrive.md](./building-blocks/hyperdrive.md) or a [localdrive.md](helpers/localdrive.md) into another one. +4. [secretstream.md](helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. +5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. +6. [protomux.md](helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. + +## Tools + +| Tools | Description | +| :----------------------------------------------------------: | :---------------------------------------------------------: | +| **[Hypershell](./tools/hypershell)** | A CLI to create and connect to P2P E2E encrypted shells.. | +| **[Hypertele](./tools/hypertele)** | A swiss-knife proxy powered by [HyperDHT](./building-blocks/hyperdht). | +| **[Hyperbeam](./tools/hyperbeam)** | A one-to-one and end-to-end encrypted internet pipe. | +| **[Hyperssh](./tools/hyperssh)** | A CLI to run SSH over the [HyperDHT](./building-blocks/hyperdht). | +| **[Drives](./tools/drives)** | CLI to download, seed, and mirror a [hyperdrive](./building-blocks/hyperdrive) or a [localdrive](./helpers/localdrive). | + +> ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). + + +## Applications built on Pear + +* [keet.io.md](apps/keet.io.md)**:** A peer-to-peer chat and video-conferencing application with end-to-end encryption. + +### Hypercore + +* The [`session`](./building-blocks/hypercore.md#core.session-options) and [`snapshot`](./building-blocks/hypercore.md#core.snapshot-options) methods for providing multiple views over the same underlying Hypercore, which simplifies resource management. +* A [`truncate`](./building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](./building-blocks/autobase.md), as described below. +* An improved fork detection in the replication protocol, to improve resilience. +* Optional on-disk encryption for blocks (in addition to the existing transport encryption). +* The storage layer now uses a write-ahead log to ensure that power loss or unexpected shutdown cannot lead to data corruption. + +### Hyperswarm + +* An improved UDP holepunching algorithm that uses arbitrary DHT nodes (optionally selected by the connecting peers) to proxy necessary metadata while being maximally privacy-preserving. +* A custom-built transport protocol, [UDX](https://github.com/hyperswarm/libudx), that takes advantage of the holepunching algorithm to avoid unnecessary overhead (it doesn't include handshaking since holepunching takes care of that, for example). It's blazing fast. +* A simplified DHT API that closely resembles NodeJS's `net` module, but using public keys instead of IP addresses. + +### Hyperdrive + +* Uses Hyperbee internally for storing file metadata +* Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. +* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. + +### Autobase (experimental) + +Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](./building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view members see on the screen. + +As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. + +> Autobase is still experimental and is likely to change significantly in the near future. + +## Stability indexing + +Throughout the documentation, indications of a module's stability are provided. Some modules are well-established and used widely, making them highly unlikely to ever change. Other modules may be new, experimental, or known to have risks associated with their use. + +The following stability indices have been used: + +| Stability | Description | +| :----------------------------------------------------------: | :---------------------------------------------------------: | +| **stable** | Unlikely to change or be removed in the foreseeable future. | +| **experimental** | New, untested, or have known issues. | +| **deprecated** | Being removed or replaced in the future. | +| **unstable** | May change or be removed without warning. | + +#### Stability overview + +| Module | Stability | +| -------------------------------------------------------- | :----------------------------------------------------------: | +| [hypercore.md](./building-blocks/hypercore.md) | **stable** | +| [hyperbee.md](./building-blocks/hyperbee.md) | **stable** | +| [hyperdrive.md](./building-blocks/hyperdrive.md) | **stable** | +| [autobase.md](./building-blocks/autobase.md) | **experimental** | +| [hyperswarm.md](./building-blocks/hyperswarm.md) | **stable** | +| [hyperdht.md](./building-blocks/hyperdht.md) | **stable** | + + + > Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. From 0908effb105cf796e2b4c3f451e19ae3d20cc619 Mon Sep 17 00:00:00 2001 From: dmc Date: Fri, 19 Jan 2024 19:38:05 +0100 Subject: [PATCH 33/35] link fixes, tweaks --- readme.md | 38 +++++++++++++++++--------------------- 1 file changed, 17 insertions(+), 21 deletions(-) diff --git a/readme.md b/readme.md index d98072d..07c0086 100644 --- a/readme.md +++ b/readme.md @@ -32,7 +32,7 @@ Welcome to the Internet of Peers ## Building blocks -The Holepunch Ecosystem is constructed utilizing the following structural components. +The following structural components form the backbone of the Pear Ecosystem. 1. [hypercore.md](./building-blocks/hypercore.md): A distributed, secure append-only log for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. 2. [hyperbee.md](./building-blocks/hyperbee.md): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. @@ -45,12 +45,12 @@ The Holepunch Ecosystem is constructed utilizing the following structural compon Helper modules can be used together with the building blocks to create cutting-edge P2P tools and applications. -1. [corestore.md](helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. -2. [localdrive.md](helpers/localdrive.md): A file system interoperable with Hyperdrive. -3. [mirrordrive.md](helpers/mirrordrive.md): Mirror a [hyperdrive.md](./building-blocks/hyperdrive.md) or a [localdrive.md](helpers/localdrive.md) into another one. -4. [secretstream.md](helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. -6. [protomux.md](helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. +1. [corestore.md](./helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. +2. [localdrive.md](./helpers/localdrive.md): A file system interoperable with Hyperdrive. +3. [mirrordrive.md](./helpers/mirrordrive.md): Mirror a [hyperdrive.md](./building-blocks/hyperdrive.md) or a [localdrive.md](./helpers/localdrive.md) into another one. +4. [secretstream.md](./helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. +5. [compact-encoding.md](./helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. +6. [protomux.md](./helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. ## Tools @@ -62,17 +62,13 @@ Helper modules can be used together with the building blocks to create cutting-e | **[Hyperssh](./tools/hyperssh)** | A CLI to run SSH over the [HyperDHT](./building-blocks/hyperdht). | | **[Drives](./tools/drives)** | CLI to download, seed, and mirror a [hyperdrive](./building-blocks/hyperdrive) or a [localdrive](./helpers/localdrive). | -> ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Holepunch, like [Keet.io](https://keet.io/). +> ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Pear, like [Keet.io](https://keet.io/). -## Applications built on Pear - -* [keet.io.md](apps/keet.io.md)**:** A peer-to-peer chat and video-conferencing application with end-to-end encryption. - ### Hypercore * The [`session`](./building-blocks/hypercore.md#core.session-options) and [`snapshot`](./building-blocks/hypercore.md#core.snapshot-options) methods for providing multiple views over the same underlying Hypercore, which simplifies resource management. -* A [`truncate`](./building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [autobase.md](./building-blocks/autobase.md), as described below. +* A [`truncate`](./building-blocks/hypercore.md#await-core.truncate-newlength-forkid) method for intentionally creating a new fork, starting at a given length. We use this method extensively in [`autobase`](./building-blocks/autobase.md), as described below. * An improved fork detection in the replication protocol, to improve resilience. * Optional on-disk encryption for blocks (in addition to the existing transport encryption). * The storage layer now uses a write-ahead log to ensure that power loss or unexpected shutdown cannot lead to data corruption. @@ -87,11 +83,11 @@ Helper modules can be used together with the building blocks to create cutting-e * Uses Hyperbee internally for storing file metadata * Major API simplification. Instead of mirroring POSIX APIs, the new API better captures the core requirements of P2P file transfer. -* Auxiliary tools, [localdrive.md](helpers/localdrive.md) and [mirrordrive.md](helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. +* Auxiliary tools, [`localdrive`](./helpers/localdrive.md) and [`mirrordrive`](./helpers/mirrordrive.md), that streamline import/export flows and make it easy to mirror drives to and from the local filesystem. We use these every day when deploying Keet. ### Autobase (experimental) -Hypercores are single-writer data structures, but collaboration is crucial. [autobase.md](./building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view members see on the screen. +Hypercores are single-writer data structures, but collaboration is crucial. [`autobase`](./building-blocks/autobase.md "mention") is an experimental module that allows to turn many Hypercores, owned by different people, into a single 'virtual' Hypercore. In Keet, every member of a room has their input Hypercore where they write chat messages, and Autobase merges these into the linear view members see on the screen. As Autobase's output shares the familiar Hypercore API, it is possible to plug it into higher-level modules like Hyperbee and Hyperdrive, getting a multi-user collaboration with little additional effort. @@ -114,12 +110,12 @@ The following stability indices have been used: | Module | Stability | | -------------------------------------------------------- | :----------------------------------------------------------: | -| [hypercore.md](./building-blocks/hypercore.md) | **stable** | -| [hyperbee.md](./building-blocks/hyperbee.md) | **stable** | -| [hyperdrive.md](./building-blocks/hyperdrive.md) | **stable** | -| [autobase.md](./building-blocks/autobase.md) | **experimental** | -| [hyperswarm.md](./building-blocks/hyperswarm.md) | **stable** | -| [hyperdht.md](./building-blocks/hyperdht.md) | **stable** | +| [`hypercore`](./building-blocks/hypercore.md) | **stable** | +| [`hyperbee`](./building-blocks/hyperbee.md) | **stable** | +| [`hyperdrive`](./building-blocks/hyperdrive.md) | **stable** | +| [`autobase`](./building-blocks/autobase.md) | **experimental** | +| [`hyperswarm`](./building-blocks/hyperswarm.md) | **stable** | +| [`hyperdht`](./building-blocks/hyperdht.md) | **stable** | > Any part of a module (method, event, or property) that is not documented as part of that module's public API is subject to change at any time. From e62fdc8f294e3d44e41f1ecd7938620e06a2d8b1 Mon Sep 17 00:00:00 2001 From: dmc Date: Fri, 19 Jan 2024 19:38:57 +0100 Subject: [PATCH 34/35] more link fixes --- readme.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/readme.md b/readme.md index 07c0086..afe9241 100644 --- a/readme.md +++ b/readme.md @@ -34,23 +34,23 @@ Welcome to the Internet of Peers The following structural components form the backbone of the Pear Ecosystem. -1. [hypercore.md](./building-blocks/hypercore.md): A distributed, secure append-only log for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. -2. [hyperbee.md](./building-blocks/hyperbee.md): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. -3. [hyperdrive.md](./building-blocks/hyperdrive.md): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. -4. [autobase.md](./building-blocks/autobase.md): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. -5. [hyperdht.md](./building-blocks/hyperdht.md): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. -6. [hyperswarm.md](./building-blocks/hyperswarm.md): A high-level API for finding and connecting to peers who are interested in a "topic." +1. [`hypercore`](./building-blocks/hypercore.md): A distributed, secure append-only log for creating fast and scalable applications without a backend, as it is entirely peer-to-peer. +2. [`hyperbee`](./building-blocks/hyperbee.md): An append-only B-tree running on a Hypercore that provides key-value store API, with methods for inserting and getting key/value pairs, atomic batch insertions, and creation of sorted iterators. +3. [`hyperdrive`](./building-blocks/hyperdrive.md): A secure, real-time distributed file system that simplifies P2P file sharing that provides an efficient way to store and access data across multiple connected devices in a decentralized manner. +4. [`autobase`](./building-blocks/autobase.md): An experimental module used to automatically rebase multiple causally-linked Hypercores into a single, linearized Hypercore for multi-user collaboration. +5. [`hyperdht`](./building-blocks/hyperdht.md): A DHT powering Hyperswarm. Through this DHT, each server is bound to a unique key pair, with the client connecting to the server using the server's public key. +6. [`hyperswarm`](./building-blocks/hyperswarm.md): A high-level API for finding and connecting to peers who are interested in a "topic." ## Helpers Helper modules can be used together with the building blocks to create cutting-edge P2P tools and applications. -1. [corestore.md](./helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. -2. [localdrive.md](./helpers/localdrive.md): A file system interoperable with Hyperdrive. -3. [mirrordrive.md](./helpers/mirrordrive.md): Mirror a [hyperdrive.md](./building-blocks/hyperdrive.md) or a [localdrive.md](./helpers/localdrive.md) into another one. -4. [secretstream.md](./helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. -5. [compact-encoding.md](./helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. -6. [protomux.md](./helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. +1. [`corestore`](./helpers/corestore.md): A Hypercore factory designed to facilitate the management of sizable named Hypercore collections. +2. [`localdrive`](./helpers/localdrive.md): A file system interoperable with Hyperdrive. +3. [`mirrordrive`](./helpers/mirrordrive.md): Mirror a [`hyperdrive`](./building-blocks/hyperdrive.md) or a [`localdrive`](./helpers/localdrive.md) into another one. +4. [`secretstream`](./helpers/secretstream.md): SecretStream is used to securely create connections between two peers in Hyperswarm. +5. [compact-`encoding`](./helpers/compact-encoding.md): A series of binary encoding schemes for building fast and small parsers and serializers. We use this in Keet to store chat messages and in Hypercore's replication protocol. +6. [`protomux`](./helpers/protomux.md): Multiplex multiple message oriented protocols over a stream. ## Tools From 09f603447be3c0177f995047465905c4173c049c Mon Sep 17 00:00:00 2001 From: dmc Date: Fri, 19 Jan 2024 19:40:16 +0100 Subject: [PATCH 35/35] tweak --- readme.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/readme.md b/readme.md index afe9241..fa67ae5 100644 --- a/readme.md +++ b/readme.md @@ -54,6 +54,8 @@ Helper modules can be used together with the building blocks to create cutting-e ## Tools +The following tools are used extensively employed in the day-to-day development and operation of applications built on Pear + | Tools | Description | | :----------------------------------------------------------: | :---------------------------------------------------------: | | **[Hypershell](./tools/hypershell)** | A CLI to create and connect to P2P E2E encrypted shells.. | @@ -62,8 +64,6 @@ Helper modules can be used together with the building blocks to create cutting-e | **[Hyperssh](./tools/hyperssh)** | A CLI to run SSH over the [HyperDHT](./building-blocks/hyperdht). | | **[Drives](./tools/drives)** | CLI to download, seed, and mirror a [hyperdrive](./building-blocks/hyperdrive) or a [localdrive](./helpers/localdrive). | -> ✔️ These tools are extensively employed in the day-to-day development and operation of applications built on Pear, like [Keet.io](https://keet.io/). - ### Hypercore