Hacker News new | past | comments | ask | show | jobs | submit login
So, you want to deploy on the edge? (zknill.io)
251 points by zknill on July 31, 2023 | hide | past | favorite | 131 comments



This isn't really edge. It's multi-region. It's a great intro to multi-region considerations, but it's not edge.

Edge implies putting your stuff in a CDN-sized datacenter that has only a subset of the full regional services, may not be able to scale up significantly on demand, may be more expensive, have less local storage and failover redundancy, etc. The multi-region considerations come in here too, but there's a whole extra set of things to worry about.

Basically you rarely want to deploy your whole app to edge; have a central or multi-region thing under the hood, and let the edge handle some very specific functionality. And even then, only if a few ms of latency makes a big difference to your clients (and they can deal with the lack of failover redundancy).


Edge IMHO implies being (very) close to the customer. As in, for example, IoT or running a service in an Internet Point of Presence maybe. But once your "in the cloud", you're not really on the Edge anymore.

But the term seems to get used for everything and anything apparently.


Yeah I don’t get how anything in a data center can be called edge.


That's just moving goal posts. Now you have to define what's a data center.

Your home or office with a bunch of computers can be classified as a data center. It might not be very redundant or efficient, but still.


I think a reasonable reader can figure out from context what “data center” means. If you have a bunch of computers in a home or office you might call it a rack, or a network closet, maybe a server room. Most people won’t call it a data center.

If you complain that the definition of “data center” is inexact, you’re missing the point—the definition of “edge” is inexact, too. That’s okay. People have a good enough definition of “data center” that it doesn’t need to be explained here. And people also understand that given definitions are not perfectly precise—there are exceptions and unusual cases.


> Your home or office with a bunch of computers can be classified as a data center.

Only in the way that a plucked chicken is a man. (http://www.artandpopularculture.com/Featherless_biped)


Well I could see it if it’s the customer’s data center.


How would you describe hosting a static website in Singapore and in New York on AWS?


Multi region, definitely not even close to edge. If you want static site on the edge why not just use Cloudflare Pages


This was just an example. If the users are in Singapore and the USA. I'm not sure what would be more 'edge' than this configuration.


More 'edge' would be colocating your equipment in each major New York ISP's data center. One box for RCN, one for Optimum, one for Verizon...

But some people would say if it's in any kind of data center, it's not true edge.


You do realize that users travel right? And that the US is absolutely massive and there are wide latency differences between coast to coast? Using two DCs isn't "edge"


Multi-region redundancy? Still not at the “edge” because I’m using your NY instance from Atlanta.


Geolocation-based load balancing, geo steering, something along these lines.

Depends a bit on wether traffic from Singapore would be allowed to go to NY if the Singapore host goes down.


Eh, I've always defined "edge" as the closest piece of compute infra to your client. Could be a VM, a load balancer, an API gateway, a service worker, whatever. Arguing it's part of your CDN is just letting Cloudflare's marketing department coopt the term.


The closest piece is the client router. Then a ton of switches on the way. I get in gist what is meant here, but it's really hard to define so simply.


Here's the definition of "Edge Computing" from State of Edge 2022

> The delivery of computing capabilities to the logical extremes of a network in order to improve the performance, operating cost and reliability of applications and services. By shortening the distance between devices and the cloud resources that serve them, and also reducing network hops, edge computing mitigates the latency and bandwidth constraints of today’s Internet, ushering in new classes of applications. In practical terms, this means distributing new resources and software stacks along the path between today’s centralized data centers and the increasingly large number of devices in the field, concentrated, in particular, but not exclusively, in close proximity to the last mile network, on both the infrastructure and device sides.

Ps. I work at https://edgenode.com


The meaning of "edge" keeps changing but currently it just means when your CDN provider offers compute services.


To CDNs marketing their compute-in-CDN products, sure.

But to people selling "Edge AI" products, an edge device is something like a smart speaker or industrial camera.


With industrial camera it is probably the device where multiple cameras connect to. With industrial solutions you often want a gateway that is edge device between the internal and external network.

At least that is my current IIoT definition. You could have bunch of pretty dump sensors and then somewhat beefy these days gateway. And the gateway here could do some computation.


There is a clear distinction between edge and a region. That distinction is defined by the distance and by relation latency. Edge locations are not heavy regions or data centres, per my understanding and hence limited by compute/storage capability. They are also high in number. Cloudflare has 100 odd edge locations all around the world.

Cloudflare and Akamai both have the concept of edge workers - light weight application APIs that can serve modest capacity. Cloudflare also has a cache on the edge.

https://blog.cloudflare.com/introducing-cloudflare-workers/ https://developers.cloudflare.com/workers/learning/how-the-c...


They're all data centers in essence; some are way smaller, less functional and some way bigger. But still much smaller than regional DC's (the cloud). Akamai has smaller ones, and hence 1000's of them. While Cloudflare and Fastly have larger PoP(Points of Presence) and a few 100's of them.


I think this is a network architecture detail, and is using the term "edge" in a subtly different sense than the article is. We have "edge" hosts and "worker" hosts where I work, but "edge" means "the edge of our network, where requests from the Internet land", not "deployed as close as possible to users".


> Basically you rarely want to deploy your whole app to edge; have a central or multi-region thing under the hood, and let the edge handle some very specific functionality.

Exactly. Example: We wanted / needed to deploy on the edge. Use case: A very very small part of a live streaming service, basically cache preloading functionality, as well as authentication / session management / DRM. All the rest would be handled by some upstream service.


In terms of the theory covered here it doesn’t really matter whether the edge is a cloud region, CDN, server in your building, or personal device. I don’t think any of these havent been tried as nodes in a distributed transactional database. It’s kind of just a semantic argument - the network failure rate, availability, and capacity to scale horizontally within your “edge” node are just parameters.

Imagine this: within a CDN’s compute you try to serve distributed db transactions optimistically then fall back to your big cloud region when it fails. The CDN is kind of a write-through cache. You still have to address the CAP theorem distributed state challenges of synchronizing state and making a consistency/availability/partition tolerance tradeoff, and it seems like a reasonable use of a CDN.


Exactly .. looks like the author don't really have experience in the possible capabilities of a good CDN network and more so in designing the software architecture in a way to take advantage of a good CDN network.


Thank you, and this actually is an important distinction, and not just pedantry

Hardware becomes the limiting factor when you’re talking edge versus multi region where hardware is one of many limiting factors, but often network is the more important factor as indicated in this link

Edge also has the distinction of being hardest to instrument for observability and often is required to work in multiple environments.


Huh, I thought edge was essentially IoT devices. Smart toothbrushes. Calling a datacenter edge seems odd.


The DC guys have co-opted the term "edge" for marketing and fund raising purposes. A former colleague was talking about a 10's of megawatts "edge" DC in Jakarta that they were building, I couldn't help but chuckle.


Edge is always one hop further than you currently are. If


> Most internet apps are read-heavy, there are more reads than writes, so largely it makes sense to deal with the latency on writes. We can do this by forwarding the writes to some leader for that piece of data (e.g. the log for usernames), and waiting until that write is replicated back to us (so we can read our own writes).

This is precisely the point. For most CRUD applications, writes are usually so sporadic and user-intentioned that you can show a spinning wheel for it, and users will forgive you for that slowness because it's so sporadic.

Edge computing is basically reinventing insecure HTTP caches and mirrors, but allowing you to include authn/authz so that user data stays private to them. If your edge data model is that much more complicated than that, you're probably doing it wrong.


I had heard authn/authz before, never realized/read into what they were:

> How are authn and authz different? To put it simply, authn has to do with identity, or who someone is, while authz has to do with permissions, or what someone is allowed to do.


Just to be clear, these two are short for authentication and authorization, right?

I guess it's awkward that those two related but different concepts share the same first 4 letters. But I think authc would have been clearer for disambiguation than authn.


It would be even clearer if we just called them “identity” and “access” (or “permission”) instead of abbreviating longer terms with a common prefix.

And we kind of do, in other (but closely related) contexts; e.g., the common cloud systems for managing both are “identity and access management systems" not “authentication and authorization management systems”.

(Though, yes, “access" sometimes means something else; nothing is perfect.)


"Authn" and "authz" have over the last 10 years become the standard terms of art in software security for distinguishing between identification controls and access management controls, so whatever other terminology one might want to use --- authc, AAA, whatever --- you should probably keep authn and authz in your resident set.


Yes, that's what they're short for.


authn/autho are much clearer in my opinion


In one of my crazy experimentation repos I played with just dropping the confusing "auth" portion, leaving "orization" and "entication". I don't know that those exact terms are a winner, though they did work quite effectively for me, but I think dropping the "auth" bit has some promise in the general sense. Leaving the redundant and confusing part at the beginning and lowering the disambiguation down to one letter seems like the wrong direction, no matter what the one letter is.


The ambiguity of just "auth" is useful when describing it at a higher level.

I can say "auth service" and it's well understood, even to the user, that service identifies them and determines their permissions.


Yep, "authn" is good because it sounds like the first two syllables of "auth-en-ti-ca-tion".

I now agree with you that "autho" would have been better for the same reason. Although "authz" sounds cooler :)


good ol' auth'n 'n' autho'


I wish it were authi and authp for identity and permissions


To add to the other two sibling comments pointing out flaws in those shortened terms, in British English it's spelled authorisation, so it would give "auths", which is even more confusing.


One more wrinkle: authorization starts with “author” but in computer security is no longer about who wrote the data but rather if they are allowed to. It has its origin in “to determine the author of” but the modern usage is at least twice removed from that, in that it’s really checking if the _token_ was written by the _system_ not if the user was the content author.


Author stems from the same Latin root as authority, namely auctor, meaning "master, leader, author".

<https://www.etymonline.com/word/authority>

<https://www.etymonline.com/word/author>

So an author is one who creates, but also a "source of authoritative information or opinion".

Confusingly, authentic seems to have a different etymology, *autos "self" (see auto-) + hentes* "doer, being".

<https://www.etymonline.com/word/authentic>

So to authenticate an author as an authority derives three meanings from two separate roots.


I see it more about having the "authority" semantic root, ie: who has the authority to act on something.


IMO it's a case where shortening the words is pretty detrimental because the point of authentication and authorization are already in the words and makes sense. Plus both words end with "n" so "authn" doesn't make sense at first glance.


It’s historically been abbreviated AAA “triple A” for Access, Authentication, and Authorization. Each being a separate system, VPN’s and Firewalls traditionally controlling access for example.


I've also seen AAA used, but standing for Authentication, Authorization, and Accounting. The Wikipedia page [1] is short, but it refers to the context I've seen that in.

[1] https://en.wikipedia.org/wiki/AAA_(computer_security)


> If your edge data model is ... more complicated than [a read-dominated workload] you're probably doing it wrong

Huh?

There are a huge number of applications that require writes at the edge.

This access pattern in no way indicates that the application is "doing it wrong".


I've seen people debating what an 'edge' even means, and wanted to chime in. Note that 'edge' is poorly defined in the industry right now. Consider the following

• Remote Region Datacenter — It's certainly not this

• Local Region Datacenter — It's not this

• Local Municipality/County Datacenter/Co-lo/Central Office — Could it be this?

• Local mini-datacenters attached to mobile towers or fiber digital loop carriers? — Could it also be this?

• On-prem Datacenter — It's not this either

A lot of people argue that an 'edge' has to be in the field — like a local mobile tower or a digital loop carrier. I'd argue that it could also be at a nearby co-lo facility or CO. Basically anything ≤300 km should be ≤1 ms latency. Within 75-100 km you're talking 0.25-0.33 ms latencies. And for most applications these days that seems "real-time" enough. YMMV.


This kind of load distribution is important but please, let’s not redefine the term “edge”. TCP/IP was specifically designed with the devices at the edge to function as peers. Computing on the edge means, say, doing the computing in your phone or laptop or iot frob.

This is longstanding use and to pollute a useful technical term just leads to unnecessary confusion.


I think that ship has sailed. “Edge computing” is a well established phrase at this point. When I need to refer to the scenario you’re describing I usually say “local computation”.


If that's were the case, why would the entire first page of Google search results for "edge computing" refer to IOT, and processing data where it's created?


You’re right of course, IoT is edge computing because it’s treated as company’s edge, while we can’t say the same for end users’ phones and computers.


But that's only because there aren't any compelling IOT applications in the consumer space. Companies are the users with distributed assets, not consumers.


Just wait until we're running transformers on the edge, using DSL and stacks.


Apple does a lot of that kind of thing on phone/tablet.

By “a lot” I mean they have a lot of devices dining so frequently; I imagine the workload is very carefully managed for battery/heat reasons.


My point was more that "transformers", "DSL", and "stacks" all have long established meanings that have been bastardized by unimaginative software companies.


At last, an examination of the true nature of 'edge' computing is presented. Despite the appealing promises made by posts from Fly.io and others that depict 'edge' computing as a simple success, the reality can be more complex.

I have recently spent a fair bit of time experimenting with this on Fly for my application (https://www.ssrfproxy.com). It's hard to beat the straightforwardness of deploying in a single region, with the database in close proximity. This approach probably to meets the needs of what 99% of developers require. Aka Heroku.


Actually on Fly it's trivial to set up PG read replicas that your edge apps can read from.


My suspicion has always been that edge computing was primarily offered by CDNs as away to increase utilisation of hardware rather than something there was significant customer demand for.


LiteFS author here. I think single-node deployments are great and it's a model that fits many (or even most) apps out there. However, the two most common feature requests when I was developing Litestream were HA & replication. While most apps won't need those features, it's good to have it available as an option down the road. I've seen many devs choose Postgres or MySQL over SQLite simply because they didn't have these features. Once those are solved, edge computing is really more of a nice side effect.

I think it's also useful to consider that tools like LiteFS aren't just for your main database. We have several internal tools at Fly.io that fan out their data to other internal apps to consume so they can share data without having to build an API or worry about rate limiting.


HA?


Sorry, I should have been more clear. HA = High Availability. Essentially, folks wanted the ability to automatically failover to a secondary node if the current primary failed.


The real "edge" is the users device, I'm 100% sold on the concept of "local first". There is a great community building around this: https://localfirstweb.dev/

The point is, by the time you have solved all the problems with consultancy and sync to an edge node, you are most of the way to local first.


> The real "edge" is the users device

Isn't that the "client"?


I'm glad there's a community working on this; I read the Ink & Switch article a while back, the local-first paradigm is very appealing.


I'm really wanting to investigate what our SaaS app looks like being fully offline, with sync on connection.


AKA heavy client


Without getting into a discussion on what is the correct definition of "edge", I think the article offers 2 solutions to this problem: you have an internet app and you need to provide your users with low-latency reads while maintaining uniqueness constraint and read-your-writes consistency.

The article favours dealing with latency during writes while making some assumptions, - most internet apps are read-heavy (reasonable, but what if it's not the case for you?) - you do not need sharding, all regions have all the data (what if you have a ton of data that is growing quickly? what if you have durations where some data is heavily contended for?) - the primary replica used for writes is highly available (how do you automate handling of the primary replica failure or primary replica getting partitioned?)

Another approach to consider would be Amazon Dynamo style databases (Cassandra, Riak etc.) which can help when the above assumptions are not met, - Dynamo was designed to be always-writable and suitable for heavy writes and reads as well (e.g. the paper mentions teams using it for catalog data with R=1 and W=N) - data is partitioned/sharded using a variant of consistent hashing, which helps with large data volumes and limiting the impact of hot spots as well - failure detection and leader election help automate handling the partitions for which a failed node was holding a primary replica


The best use case for edge deployment is the deployment of static resources for the initial web app load. It achieves a latency of around 40ms and almost instantly shows the user the initial render. The later database CRUD actions are much less important when the application has already loaded. Not only because much of data can be cached in the browser or aggressively pre-fetched, but also because many CRUD actions run in the background only show errors if something went wrong.


I think this depends on the application. For a complex spa, sure. But for something like a news site or social media the extra load time feels horrible. It's preferable to use server side rendering as much as possible. That also means you gotta speed up your database queries, which is where techniques like in the OP come into play.


Even for mostly static, edge can add a cache layer, and also do doughnut rendering for sites...

By doughnut rendering, I mean you can displace certain static placeholders with user-level content... such as replacing the "login" section of the header with the markup for the User's name and profile image.

These are both common use cases for Cloudflare Workers. Not just for complex compute but for near-line cache and simple transforms.


What about local read only replicas of the db in each region, and one primary in your primary region? Or a write through cache in each region.


This design is really similar to the approach taken by litefs[1] and turso[2]. Writes are forwarded to the primary (like you would with a write-through cache) and a local copy of the data is held in each region. These local copies are only guaranteed to be up-to-date with the most recent write forwarded by that local copy.

[1] https://fly.io/docs/litefs/proxy/ [2] https://docs.turso.tech/reference/data-consistency#on-the-re...


From what I understand, that should be possible with Fly.io, using the fly-replay header to denote requests that need to be run against the primary database. I'm not sure if that's still their recommended approach, though.

EDIT: https://fly.io/blog/globally-distributed-postgres/ talks about how this can work. The approach it suggests for implementing this is to try running all database queries locally; if there's an error from the local database being a read-only replica, add the fly-replay header pointing to the primary region and return an HTTP 409, and Fly's proxy will rerun the request in the specified region. There's also some commentary on handling consistency issues to at least get read-your-own-writes.


Yes, it is. It's how we think about multi-region replication in our own tooling. It's not the only way to handle it (Cockroach would be an example of another approach; so would rqlite). We're not evangelists for multi-reader single-writer --- it's just the most common approach big Postgres shops tend to use, so we worked on making it effective. But the other approaches should work fine on Fly.io too.

What we especially like about multi-reader single-writer clusters is that you can mostly keep all the pieces in your head at the same time. If you understand Postgres or SQLite, and can get your head around the `fly-replay` header (which is just "replay this request over there please"), you've got the whole system.


rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, but it is an option.

"rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes."

[1] https://rqlite.io/

[2] https://rqlite.io/docs/clustering/read-only-nodes/


Thanks for confirming this!


We do new stuff all the time but `fly-replay` is unlikely to go away, since I think it's Kurt's favorite thing we've ever come up with. :)


This is relatively easy to orchestrate with MongoDB replica sets (and I'm sure with other databases, too). In a typical MongoDB deployment you want at least 3 nodes (1 primary, 2 or more secondaries).

You can put one of the secondaries on the other side of the world.

By default, all reads are sent to the primary. But you can easily specify "nearest" as your "read preference" on a query-by-query basis which would allow the app to read from the nreplica with the lowest latency (whether it be the primary or one of the secondaries).

The ability to specify read preference at the query level lets you have 2 categories of queries, 1) queries that must always be served with most recent writes, sent to primary at the expense , and 2) queries that are allowed to occasionally send stale data but benefit from latency improvements by reading from the nearest node.


PolyScale [1] focuses on many of these issues. It provides a globally distributed database cache at the edge. Writes pass through to the database and reads are cached locally to the app tier. The Smart Invalidation feature inspects updates/deletes/inserts and invalidates just the changed data from the cache, globally.

1. https://www.polyscale.ai/


> Or a write through cache in each region.

This is something I want to play around with using Cloudflare. I wonder if there’s enough of a guarantee for a user to “stick” to a node that you could use Workers KV as an eventually consistent write through cache for user unique data and a Durable Object as a strongly consistent write through cache.

I feel like that would give a pretty solid balance, wouldn’t it?


Cloudflare has Durable Objects/workers for consistent write-through... You can combine them. You can also consider D1, which might be the better option all around for many use cases.


Or just having a completely separate database for each region, much like the average MMO does it.


>If a user makes a request from Europe, and the apps run in US East, that adds an extra 100-150ms of latency just by round-tripping across the Atlantic.

These numbers seem high. This site shows transatlantic RTT of around 75ms: https://wondernetwork.com/pings

If a CDN or edge computing platform reduces that to 20ms then the difference is 55ms. And it matters only for read requests that cannot be cached locally.

Whether or not its worth it depends largely on the number of requests. Perhaps reducing that number should be prioritised.


If your application isn't real-time (like online gaming or multi-way voice/video), and you aren't making a bunch of requests serially from the client (very rare), I'm convinced you can serve the whole world with at most 2 datacenters - e.g. one in NYC and another in Singapore - coupled with a CDN to serve static assets.

The notion that you need to replicate your infra in 10 regions in order to provide the best user experience sounds like propaganda pushed by cloud companies to make you use more product than you need.


It's usually just as straightforward to serve in NA and APAC as it is to serve in 10 different regions, and it's a very sane strategy --- I'm rolling out a Vault replacement inside Fly.io right now, which is in the hot path for booting VMs up anywhere, and my initial deployment strategy is just Sydney, Ashburn, and Frankfurt. So you're right, and the only nit I have is the idea that there's a conspiracy to get you to do lots of regions. I assume, like us, most cloud providers are equally psyched at the prospect of getting you up and running in 2 regions as they would be about 2 dozen.

The one thing to keep in mind here is that the software architecture considerations of going to 1 to 2 regions cover about 95% of what it takes to go from 2 to 10. To a first approx., the hard part is being plural, not how plural you are.


You've linked to ping/ICMP statistics, TCP is a more common use case and thus tends to be more representative of real-world applications. 100ms is a fairly realistic 90th percentile in my experience, 150ms could be in the 95th.


The extra time it takes to process a TCP request compared to ICMP traffic is the same regardless of whether or not you're using an edge computing platform. It seems to me that what actually matters is the difference in latency, not the absolute numbers.

That's why I think the article is incorrect when it claims that running an app in US East "adds an extra 100-150ms of latency". Not all of this is "extra". Only about 55ms are.


Leaving aside what "edge" really means, for me personally, this is about connecting a website running on Deno Deploy with a database.

As a hobbyist doing recreational programming, I don't want to pay for a website when it gets no traffic. Everything needs to start up on demand and shut down when idle.

This means that cold start time is what really matters. Since Deno Deploy has no persistent storage yet, both the frontend and the database need to start up on first request. Having a multi-region database doesn't help if needs to be running all the time and doesn't start up very fast.

As for latency when warmed up, I plan to deal with that by avoiding round trips with stored procedures and caching.

Here's my "hello world" website. Any advice on technologies to check out would be welcome.

https://postgres-counter-demo.deno.dev/


"If a user makes a request from Europe, and the apps run in US East, that adds an extra 100-150ms of latency just by round-tripping across the Atlantic."

laughs in Rural New Zealand

But seriously really enjoyed this article. I was always a bit confused how the recent influx of edge services handled writes/consistency, and yeah they all seem to have a single leader that routes writes to them - the edge is all about fast reading.


In 2006 or so I was working for an agency that was angling for a contract for the City of Ithaca to develop a system for building inspectors to use cell-phone connected PDAs to do their paperwork on the go.

At the time I advocated for an “always connected” model but we were concerned that cellular connections wouldn’t be that reliable so we thought disconnected operation would be necessary.

A few year back I was thinking about the future of “low code” and thought an open-source project similar to Lotus Notes but JSON-centric would be a good thing

https://en.wikipedia.org/wiki/HCL_Domino

In particular, Lotus notes has a model for disconnected operation that works quite well. My take is that “progressive web apps” haven’t gotten that far because they don’t have enough batteries included, a really good bidirectional sync model would make a difference.

For years you could say ideas from Lotus Notes didn’t make it into the open source world because they were patented but the patents have long since expired and the main reason they are obscure is ignorance.


Might be interested in PouchDB/CouchDB which can be offline and resync relatively easily. Slightly different approach than other SQL based RDBMS though.

In the end, it gets complicated to support offline web-apps and really depends on your usage. I had to work on a water dispensing system over a decade ago that needed to be able to work "offline" at a given site... I used firebird embedded locally, and a remotely available server for syncing against. All accounts and balances were synced to each site, so that they could operate in offline for dispensing water to carrier trucks. There were rules on offline term (up to a week) and balance carry to reduce the risk of a negative balance. In the end it worked well, but it wasn't at all simple.

Active/Inactive and syncing transactions was at least interesting, and that was for a relatively simple data model. More complex models would be, of course, more complicated... and almost exponentially so.


Perhaps it's better to launch multiple, region-specific versions of a site, with region-specific data. Something like amazon.com vs amazon.nl vs amazon.de. Account data would still be global but that doesn't change much so you can get away with strong consistency.

Added benefit is that it clarifies how to do local compliance. With a global model, the complexity of local laws can really overwhelming. For example a global site based in the US needs to be GDPR-compliant, while a global site based in the EU has to figure out how to file multiple forms of taxes (federal, state, sales, city, district) for the 16000+ US tax jurisdictions. As a European I am more afraid of US taxes than GDPR.


There are merchants of record which will resell your product, and take care of levying and remitting taxes, such as paddle.com for SaaS software.


> federal, state, sales, city, district

Really? I've never really been to the US, other than a couple of brief work trips, and that is astonishing.


Really! For some locations you need a street address to be sure you're collecting the correct Sales Tax for that particular type of product.

Also there are Sales Tax holidays where the rate of sales tax is reduced for a set period: https://www.salestaxinstitute.com/resources/sales-tax-holida...

Also certain customers are exempt from Sales Tax...


District is unusual.

There's no federal sales tax, but if you're selling internationally, there can be border fees of various kinds. State sales taxes are nearly ubiquitous. Large cities or counties might have a sales-tax addition for local funding.

Special-purpose sales taxes exist, infrequently: hotels are probably most common.


Sure, there are also excise taxes levied on different kinds of products (fuel and alcohol being the most common), fees (like disposal deposits on drink containers, tires, and the like) and just in general differential tax rates (certain kinds of foods are not taxed, taxed at a different rate or even, I’ve seen proposed, taxed by the size of container).


Aren't import taxes essentially federal sales taxes? Granted, they're applied long before the product sits in a consumer's cart.


I think the footnote is pretty much the solution.. imagine you implement replication within a DC. Yes you risk data loss and maybe inconsistency (if you fail over to a new region and start doing writes without a way to fix this later) when the DC becomes unavailable but you can now offer strong consistency without expensive cross DC hops. Pretty sure this is how Google Cloud Spanner is implemented: you have a single region for writes and everywhere else is just a read partition? And it’s probably nothing fancy to set the read partition, just pick the first hop at the time of creation

Basically: implementing a distributed global db without structuring the topology to minimize hops/making engineering trade offs can’t be fast. It’s all about deciding which part of the CAP theorem you want to relax.


Great post highlighting important considerations before you make the decision to go on edge!

If you do though, you should take a look at https://edgenode.com, which I helped build (we host server-less containers on edge)


Definitely some good points here. Using a single primary database seems easier for a lot of more straightforward use cases, and read replicas are probably sufficient for read-heavy workloads that can tolerate some loss of consistency.

I think either the Cloudflare Workers team or the Fly folks have talked about how they envision edge apps as sharding data by region or by tenant, then sticking each shard in the optimal location. That sounds potentially useful for some cases, but then there's the complexity of sharding to deal with.


We host applications that do sharding like this, but when we build our own stuff at Fly.io, or publish tooling like LiteFS, we're generally not thinking about sharding right now. We like things that are easy to keep our heads around.

We're a hosting provider, not really a database provider. Systems that do sharding, Raft consistency, and stuff like that are built on top of systems like ours, not so much inside of them. :)


Gotcha, that makes sense; I'm probably thinking of Cloudflare Workers, then. It might've come up from kentonv's comments in the recent post about Cap'n Proto 1.0.


Hmm, not in that thread, but maybe in a thread about D1 or Durable Objects? We are indeed trying to make it as easy as possible for applications to shard databases by user or by "document".


There's various options... my current direction is CockroachLabs for the DB side... but Cloudflare offers D1 (sqlite interface over durable workers/kv) which is kind of what you are talking about in practice.

Actual data sharding, without persistence in different regions is more difficult. Different database systems lend themselves differently for different loads and sharding characteristics... I've seen MS-SQL in master-master across the globe, postgresql with sharding that is replicated to base host, and various ScyllaDB/Cassandra setups. It's always a mixed bag.


Nitpicky terminology choices aside, this is a very good, easy-to-follow writeup on multi-region data system design. Understanding that the trick is shuffling around the latency makes it much easier to reason about where you want to focus your latency savings. One thing I wasn't quite sure on, though, is why Database A would care if Database B had a read of a certain kind (vs. a write). Reads don't change state, so if a million reads occur with no writes, the data should still be consistent, no?


If you're talking about the case where a local database forwards a read to the 'leader', and that read is put on the leader's log, but that read is only served when the read is replicated in the log back to the original local database then: It's actually less about the the leader caring about the read, and more about creating a point in time that the read will happen.

The database is using the read in the log as the point-in-time where that read happened. The writes before that point will be viewed by the read, and the writes after won't be.

By sending the read to the leader, when we receive that read back, we know for sure that all the writes that should have been replicated to our local copy have been replicated (because our read has made the full round-trip that all our writes would have, so we're guaranteed to have received all the writes before that read's "point-in-time").


Oh, the author! Great work on the article, and thanks for the answer!


Another pattern here is to have your edge provider automatically move your compute closer to your backend. Cloudflare released Smart Placement a few months ago to do this exact thing (with surprisingly similar diagrams!) https://blog.cloudflare.com/announcing-workers-smart-placeme...

(disclaimer: I was the tech lead on this feature)


My takeaway from this is that a lot of people disagree on what "edge" is. IMO, "edge" is the furthest resource that you have some computational level of control over. Could be a data center, could be a phone, could be an IoT device, could be a missile.

EDIT: I think I'm realizing people will disagree with me because I have a different perspective. For my use cases, my data comes from sensors on the edge, and so for me, I want my edge computing to be as close to those sensors as possible.


Not quite. Nearly every service out there has computational control over the end client (whether a mobile app, browser JS etc.), but very few are focused on edge compute at that level.

It is more helpful to think of it in terms of a traditional client-server architecture, where you want to move the server as close to the client as possible. This covers 95% of what people mean when they say edge compute.


I do see your point, and I agree that covers most cases, but can it really be a definition if it doesn't cover all cases? Are just talking about a distinction between "edge" in an architecture vs "edge computing?"


It's actually really helpful. Looking at a lot of js frameworks I thought I understood what they meant, but now I understand that the term is actually pretty ambiguous.


That's a terrible "definition" of edge


I'd love to hear a better one. Care to comment?


To me edge is when you are doing most computation and data retrieval required for the given request in a data center close to each user. How to define close? Probably when you have at least several dozen POPs spread out across the world


Most applications tend to benefit from serving an Early Hints[1] from a CDN, rather than actual edge computing. Early hints takes care of the delay in response time by ensuring that the page, when loaded, will tend to have all the resources it needs to load instantly.

[1] https://httpwg.org/specs/rfc8297.html


I extend my prayers out to those of you who have pointy-haired bosses looking to seem trendy in their public reports by implementing this.


Edge is just where something is hosted. It has nothing to do with data consistency. User signup is a unique problem that any large site has. Some sites like HN also actually needs usernames which makes things more complicated. Owasp always recommend to add usernames to any login. It is such a naive approach. User names are garbage in most cases.


No, I do not want to deploy on the edge, and I wish people would stop making noise over it because it’s getting really had to convice buzzword happy business people that there’s nothing wrong with running a monolith on a single big server.


Great intro summary to this topic!

I remember these distributed db scenarios being describe when reading "Designing Data Intensive Applications" and this was a really helpful supplementary piece to jog that memory.

Cheers


I get the premise, but something must be said for caching cutting down on those DB transactions. Do the math for YOUR app and see what makes sense.


One definition of edge I found was '5ms average latency to the customer' , does that seem like a good definition?


Sorry for the unrelated question, but can someone tell me how this blog is built? What tech stack is used here?


A CDN makes a lot of sense to me, having static assets that get cached close to users is a brilliant way to speed up requests. As the article points out, this is a very hard problem to solve for anything that touches a database or is not static/cached content. This is also why I don't believe in the idea that Fly.io pitches. Additionally you have the legal things to take care of with GDPR, etc. so a local database is probably the way to go for most projects.


I had no idea edge was so varied. Isn't Edge primarily involved in IOT?


Okay




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: