> The firehose is all public data going into the network right?
It's the "main subset" of the public data, being "events on the network": for the Bluesky app that's posts, reposts, replies, likes, etc. Most of those records are just metadata (e.g. a repost record references the post being reposted, rather than embedding it). Post / reply records include the post text (limited to 300 graphemes).
In particular, the firehose traffic does _not_ include images or videos; it only includes references to "blob"s. Clients separately fetch those blobs from the PDSs (or from a CDN caching the data).
> > If you want large scale social networks, you need to work with a large scale of data. Since federated open queries aren’t feasible, you need big machines.
> Thats just simply not true.
> [snip]
> This is why ActivityPub has a discovery problem by the way, but it's just a symptom of real federation.
You're actually agreeing with them! The "discovery problem" is because "federated open queries aren't feasible".
> With ATProto you NEED to connect to some centralized relay that will broker your messages for you.
You can connect to PDSs directly to fetch data if you want; this is exactly what the relays do!
If you want to build a client that behaves more like ActivityPub instances, and does not depend on a relay, you could do so:
- Run your own PDS locally, hosting your repo.
- Your client reads your repo via your PDS to see the accounts you follow.
- Your client looks up the PDSs of those accounts (which are listed in their DID documents).
- Your client connects to those PDSs, fetches data from them, builds a feed locally, and displays it to you.
This is approximately a pull-based version of ActivityPub. It would have the same scaling properties as ActivityPub (in fact better, as you only fetch what you need, rather than being pushed whatever the origins think you need). It would also suffer from the same discovery problem as ActivityPub (you only see what the accounts you follow post).
At that point, you would not be consuming any of the _output_ of a relay. You would still want relays to connect to your PDS to pull data into their _input_ in order for other users to see your posts, but that's because those users have chosen to get their data via a relay (to get around the discovery problem). Other users could instead use the same code you're using, and themselves fetch data directly from your PDS without a relay, if they wanted to suffer from the discovery problem in exchange for not depending on a relay.
It doesn't change the fact that if someone were to do that, it wouldn't be supported by anyone let alone the main bluesky firehose. I think it's pretty disingenuous to just say "You can do it" when what your suggesting is so far off the intended usage of the Protocol that it might as well be a brand new implementation. As a matter of fact, people DO already do this. they use ActivityPub and talk to bluesky using a bridge.
The core of the issue is, Bluesky's current model is unsustainable. The cost of running the main relay is going to keep rising. The barrier to discovery keeps getting higher and higher. It might cost 150$/month now to mirror the relay, but what's going to happen when it's 1000$?
By your own numbers it averages 2.7 MB/s. This is manageable with a good cable internet connection or a small vps. This is a small number for 5.5 million active users.
What happens if it expands to 10 times its current active users? Who knows, maybe only the 354 million people around the world with access to gigabit broadband can run a full server at home and the rest of the people and companies that want to run full servers will have to rent a vps.
Additionally, a user _can_ root their identity in DNS if they want, by using did:web instead of did:plc [0]. The main Bluesky client doesn't expose this (presumably because did:web cannot provide a mechanism for automatic migration between PDSs (due to the PDS having no control over the DID document) or recovering from loss of control of the domain name, so it requires more technical expertise), but there are users successfully using this method for their identities.
For a lower-friction firehose experience, you can use Jetstream [0] (by one of the Bluesky devs) which supports subscribing to specific Collection NSIDs and user repositories, and converts records to JSON for you.
There's a public instance URL in the README (with bandwidth limits), or you can self-host.
The firehose itself isn't really the fiddly part since it's just a WebSocket connection. Setting up the feed server, publishing the DID for its web host, then publishing the feed generator to the network were all kind of a low-grade hassle that killed a lot of my enthusiasm. Like none of it was especially complicated if you're doing it for a professional project or whatever, but I was just trying to goof around while watching episodes of Highlander: The Series, and it was taking me away from Duncan.
I'll check out this Jetstream project for sure, though.
For non-Bluesky apps built in ATProto, in addition to White Wind (blogging), there is also Smoke Signal (events, only Lexicons are open source currently AFAICT) [0], and Frontpage (link aggregation) [1].
> But I’m still wondering how one would aggregate reactions. It seems it would not be enough just to have a PDS, but also to run an Indexer/Relay, which indexes other peoples PDS. And not just the PDSs of the people you are following, but also the rest of the world – after all a Like or such can come from everyone. When running an Indexer you are pretty soon into Firehose territory. Too much for a single person website.
You don't need to run a Relay yourself in order to see reactions; you instead subscribe to an existing Relay's firehose (such as the one that BlueSky PBC operates), and ignore anything that isn't "reaction-shaped" (matches the relevant Lexicon).
> Everyone talking about ATP seems to [..] never talk about the transmission mechanisms. Whereas I’m sitting here and thinking that an ActivityPub Actor’s Outbox is already a form of PDS – and it has an Inbox. Addressable via the Web and findable via Webfinger.
Here's my attempt to summarise the transmission difference between ActivityPub and AT Protocol: ActivityPub is symmetric and push-based, AT Protocol is asymmetric and pull-based.
With ActivityPub, your client talks to your server. The server handles both outgoing data (you making posts) and incoming data (viewing other people's posts).
When you create a post:
- The post is placed in your server's outbox.
- Your server looks up the servers for every one of your followers, and places your post in their inbox.
When someone creates a reaction:
- The reaction is placed in their server's outbox.
- Their server looks up your server, and places the reaction in its Inbox.
- Your website checks the Inbox, discovers the reaction, and handles it.
With AT Protocol, your client talks to your PDS and the client's App View. Your PDS handles outgoing data (you making posts), and your client's App View handles incoming data (viewing other people's posts). Relays are what provide transmission between the two: Relays subscribe to your PDS, and App Views subscribe to the firehose of at least on Relay.
When you create a post:
- The post is added to your repository on your PDS.
- The Relay pulls the post from your PDS and broadcasts it on its firehose.
- App Views see the post in the firehose, and include it within the feeds of users that follow you.
When someone creates a reaction:
- The reaction is added to their repository on their PDS.
- The Relay pulls the reaction from their PDS and broadcasts it on its firehose.
- Your website's App View (which might be your client's App View if "reactions" are also posts, or some other App View if this is a different kind of semantic data) sees the reaction in the firehose, and handles it (e.g. includes it within the feed of reactions that your website uses).
But in a way depressing: I fear subscribing to a firehose is not that realistic for small single person websites. Maybe you can filter the firehose subscription.
> The indexer is only necessary because the user ids do not remain static on the system. Instead there is some weird cryptographic reshuffling going on where DID:PLC starts as a substring of a hash and then because of 'key rotation' the user id of every user actually changes out from under the user themselves over time.
The user IDs (e.g. did:plc or did:web) do not change over time. You are correct that a user that uses did:plc has that generated from a hash (of their initial repository state), but subsequent additions to or deletions from the repo don't change the DID. Likewise, if you set up your repository with did:web, the domain you pick is fixed, and you lose access to your repo if you lose control of that domain.
What is changing with key rotation is the repo signing keys that authorize updates to the user repo. If you run an entirely self-hosted PDS then you can choose to never rotate that key, and then you have a "static private key" repository. But key rotation is a necessary part of the design, in part to enable account migration (you wouldn't want an old PDS to still be allowed to make changes to your repository, so you'd rotate those keys to remove the old server's key and add the new server's key).
The comparison here that comes to mind is imagine if Github issued you a keypair to use over ssh to push/pull from a repo, but they change that keypair for you periodically and also keep track of that keypair for you.
I'd much prefer to generate my own ssh key and upload the public key to Github, Gitlab, and Sourcehut, and Codeberg, etc. Forgive me if I forgot a forge.
A useful way to think about it is that the PDS is a "user agent" - they act on behalf of the user, and with the user's implicit trust. This is much the same way that a user trusts webserver software (and the VPS running it) to correctly serve their website, hold onto the private keys for the site's TLS certificates, use those private keys to correctly set up encrypted connections to the website, etc.
The AT Protocol itself does technically allow for all private key material to be held at all times by the user, but that means the user needs to be interactively involved in any operation that requires making a signature (and online / available whenever such an operation is necessary, which isn't necessarily the same times as when the user is interacting with a particular app like BlueSky running on ATProto). The PDS software that BlueSky have implemented instead requires (IIRC) that the PDS has the private key for at least one of the public signing keys (but there can be multiple signing keys, so the user can also hold onto one locally).
> There is a sub-curve (Ristretto) which is of prime order which can be used on top of curve25519.
I'm sure you are aware of this and are just abbreviating, but I think it's important to clarify this for other readers: Ristretto255 is not a sub-curve of Curve25519. It is not a curve at all. Ristretto is abstractly a prime-order group, and concretely a scheme for encoding group elements such that round-trip decode-and-encode produces the same element encoding.
It happens to have been designed to have the same order as the prime-order subgroup of Curve25519, for convenience of implementation, but it is possible to implement it using a different elliptic curve under the hood (though in general you're better to use Curve25519 under the hood for the same reason as avoiding Ed448: there are many more eyes working on solid implementations of that curve's arithmetic).
> But doing zero-knowledge proofs of Ristretto operations is significantly more complicated due to the fact that it is a sub-curve
Using Ristretto255 in zero-knowledge proofs is not much more complex than using an elliptic curve. It just isn't as _simple_ as using a plain elliptic curve, because you cannot witness a Ristretto element as a coordinate pair (x, y), again because Ristretto255 is not a curve. You instead have to witness the encoding of the group element, and then implement the decoding operation inside the circuit to constrain the encoding to its arithmetic representation (e.g. a Curve25519 curve point). This would actually be pretty straightforward to implement in a PLONK-style arithmetisation, but it's also only 12 field element multiplications and an inversion constraint, so it's perfectly feasible in R1CS as well. And thanks to the decoding constraints providing a firewall between the group element encoding and its arithmetic representation, once you have the latter you can safely just use it, and don't need to propagate subgroup checks or cofactor elimination throughout your protocol (like you would using Curve25519 directly).
In practice though, I agree that if you can use a prime-order elliptic curve, then you should definitely do so. Ristretto255 was designed at a time when people wanted to use cofactor curves for performance and curve-safety at the cost of group-safety; Ristretto255 provides that group-safety. Nowadays we have complete and performant formulae for all of the prime-order elliptic curves people want to use, so if you have the option then both secp256r1 and secp256k1 are decent options. But if you're in an ecosystem where Curve25519 is a necessary primitive, then Ristretto255 is a great option.
(Disclosure, since I'm not using a pseudonym here: Filippo and I are authors on the Ristretto IETF draft [0].)
It's the "main subset" of the public data, being "events on the network": for the Bluesky app that's posts, reposts, replies, likes, etc. Most of those records are just metadata (e.g. a repost record references the post being reposted, rather than embedding it). Post / reply records include the post text (limited to 300 graphemes).
In particular, the firehose traffic does _not_ include images or videos; it only includes references to "blob"s. Clients separately fetch those blobs from the PDSs (or from a CDN caching the data).