Related tangent: highest possible recommendation for https://hpbn.co/ (High-Performance Browser Networking), an extraordinarily helpful book and website.
This is a nice post, but it feels a little weird because while HTTP/3 gets them 10ms on average, they could get 30ms on average by reducing Asia latencies (15% of traffic, 300ms) to North America latencies (100ms). I'm sure they have some internal constraint that makes that difficult, but just stuck out at me.
That might require them to put replicas of the search service in Asia, and operating a distributed system over distant geographies is a hard operational problem.
Not necessarily: they could terminate the initial connection close to the user and then run a new connection. While this sounds like it shouldn't help, because the second connection is fully under their control they can make it highly reliable and very fast. Additionally, any recovery from dropped packets (on the user's home network) only has the latency of an in-region connection.
This also sets them up for ruining a light cache in-region, and potentially scaling that up to handle more kinds of API calls.
Yes I'm sure they've considered doing that as it seems so obvious. I would also think that the use of any one account was very region specific so could you actually base their whole data out near them. Be interesting to hear how they've ended up at the compromise that they have.
It would be nice to see that 99th percentile, the 99.9th percentile, and the 99.99th percentile.
My browser has made over 100,000 network requests so far today, and I expect yours has too. I'm sure at least some of those will be in the 99.99th percentile.
Note that some of the users on the worst networks are also users that won't be able to use HTTP3 due to corporate firewall restrictions that block UDP and downgrade TLS.
Source: I've worked on network protocols for video conferencing and have seen my share of corporate network horrors.
One thing I'm wondering about is that http3 seems to be from end user to search, which is a general serving problem and not really about search. Do dropbox clients call the search service directly?
I have wondered recently about head of line blocking's impact to latency at AZ level within AWS. How reliable are TCP connections within the same datacenter and do these issues noticeably impact performance at the tail end?
It would be interesting to see similar experiments published from big edge networks like Cloudflare and Fastly. Do they have something similar published?
Why not? As long as you can send and receive UDP traffic, there's no reason why you can't do http/3.
Caddy and HAProxy both support the protocol. nginx doesn't have it in the public stable version yet, though there are preview packages available for modern server platforms.
HTTP/3/QUIC supports migrating connections between two networks, such as if a user switches from WIFI to LTE. IPVS or any UDP load balancer won't handle this scenario properly since it doesn't introspect the QUIC header and load balance based on the QUIC connection ID. This QUIC connection ID allows for a stable connection when the device needs to switch networks. If operators have any sort load balancer (like IPVS) between the client and the point the HTTP/3 connection is terminated, they will need to ensure that it has proper support for QUIC. One example is Katran[1] which has support for this method of load balancing.
Yes. Google and Facebook have deployed HTTP/3 across their properties for quite some time now. Support in popular servers is not mature but if you're using a CDN like Cloudflare, it's trivial to enable HTTP/3 for your users.
We use it on our rinky-dink ecommerce site, and it represents 40% of requests at the moment. HTTP/1 and HTTP/2 are at 30% each. Our users skew a bit older, and definitely aren't techno-savvy (one scrolled past all the products and used the contact us page to ask for a printed product catalogue and order form...).
Wondering the same- how’s the compatibility with very restrictive networks? Say a university/corporate network that only allows port 443 and DPI’d 80 egress. QUIC won’t pass through that since it’s UDP.
Can server and client reliably negotiate a fallback to http/2 in such a case?
This. There are also clients that with a little config will let cache the support level per-host, and even provide a list of hosts that the initial request should race TCP and QUIC to.
What if your browser receives Alt-Svc header and switches to http/3 on one network (say mobile data), but then you switch to a restrictive WiFi that has UDP disabled. All without restarting your browser/within one "session" of your http client. Wouldn't you start having connectivity issues that would be hard to troubleshoot? In that scenario, having http3 disabled is beneficial.
Changing network interfaces breaks connections and causes a new handshake. Browser session works at a different layer and doesn't prevent that.
QUIC actually lets you migrate between connections (because the packets are identified by a connection ID in each UDP packet rather than a 5-tuple). Clients will typically re-test a connection occasionally and downgrade as needed for this to work.
HTTP/1.1 isn’t going anywhere in any popular server, so yeah, HTTP/2 fallback isn’t a problem. (HTTP/1.1 is basically required even if you just want to serve an https redirect on port 80, since h2 requires TLS. The clear text h2c protocol has no adoption AFAICT.)
Sadly, 1.0 is _still_ around. Rare, but it happens.
I remember being gobsmacked several years ago that F5 LBs only supported HTTP/1.0 health checks. Doing HTTP/1.1 health checks required writing one specifically for it. Something the community had got sorted a long time ago.
IPv4 is not going away because it's valuable (as in limited supply creates a market). HTTP 1.1 is already unusuable in modern web as you'll be instantly blocked by cloudflare and gang. HTTP2 is likely to follow in near future as replacing it will be much easier than replacing http1.1.
I'm not sure you understand. IP scarcity means they are more valuable in the web automation context. IPv6 availability will make it harder to "price" web automation -> more bots online -> more captchas and privacy invasions. That's a real challenge that is hard to solve despite your snarky comments.
The IPv4 address scarcity means that bots cause a lot more collateral damage, because if your neighbor runs a bot from home, you'll most likely get banned too.
Yeah, there's value in convenience of being able to block misbehaving IPv4 addresses. There's also value in burning CPU cycles to produce a transaction on a blockchain.
Alternatively, open up developer tools, go to the network panel, right-click on the headings and enable the protocol field. It will tell you which protocol was used for each request made by your browser. Obviously the server also needs to support HTTP/3 for a request to use HTTP/3.
Another user here commented that it's been enabled randomly for a subset of users. But not for me. And there's nothing on the web I've found about how to enable it. It was once on the Experimental menu, but no longer.
You can enable http/3 just fine even if you also serve clients that are falling behind on web standards, as long as you also serve http/2 as a fallback.
Safari not supporting isn't a problem unless your business only and specifically targets Safari, and even then it's a relatively painless way to help the few customers who run preview versions get a nicer experience.
While HTTP3 does provide some improvements, it's clear that further optimizations are needed, particularly for users in Europe and Asia.
One potential solution that hasn't been discussed much here is leveraging AWS managed services like AWS CloudFront or AWS Global Accelerator. It's worth noting that Dropbox's website already uses AWS CloudFront, so they are already leveraging AWS in some capacity. Based on my cost calculations, using AWS CloudFront would cost around $40k a month, while AWS Global Accelerator would be around $22k a month.
As of August 22, 2022, AWS CloudFront supports terminating HTTP/3 in a Point of Presence (POP), which could potentially help with the latency issues Dropbox is facing. AWS Global Accelerator, on the other hand, is designed to improve the performance of applications by terminating UDP/TCP as close to users as possible then routing user traffic through the AWS global network infrastructure. This could help reduce latency by ensuring that user traffic is routed through the most optimal path, even if the user is located far from a Dropbox data center.
It's hard to estimate what the potential latency reduction of using e.g. AWS Global Accelerator is, especially at higher percentiles. However, using https://speedtest.globalaccelerator.aws/, and assuming symmetry, my connections to Asia are 35-40% lower latency.
Of course, there are trade-offs to consider when using managed services like AWS CloudFront and AWS Global Accelerator. While they can provide significant performance improvements, they also come with additional costs and potential vendor lock-in. However, given the scale of Dropbox's operations and the importance of providing a fast, reliable search experience for their users, it may be worth exploring these options further.
---
Cost estimates
Assumptions:
1. Dropbox's peak traffic is 1,500 queries per second (QPS).
2. Average data transfer per query is 100 KB.
3. 50% of the traffic comes from North America, 25% from Europe, and 15% from Asia (remaining 10% from other regions, for pricing purposes put it into Asia's calculations).
---
1) AWS CloudFront cost estimation:
Data transfer:
- North America: 1,500 QPS * 0.5 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 194.4 TB
- Europe: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB
- Asia: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB
Data transfer cost:
- North America: 194.4 TB * $0.085/GB = $16,524
- Europe: 97.2 TB * $0.085/GB = $8,262
- Asia: 97.2 TB * $0.120/GB = $11,664
Total data transfer cost: $16,524 + $8,262 + $7,006 = $36,200
HTTP requests:
Total requests: 1,500 QPS * 60 seconds * 60 minutes * 24 hours * 30 days = 3,888,000,000
Using the updated AWS CloudFront pricing (as of May 22, 2023):
- HTTP requests cost: 3,888,000,000 * $0.0075/10,000 = $2,916
Total estimated monthly cost for AWS CloudFront: $36,200 (data transfer) + $2,916 (HTTP requests) = $40k
---
2) AWS Global Accelerator cost estimation:
Data transfer:
- Total data transfer: 194.4 TB (NA) + 97.2 TB (EU) + 97.2 TB (Asia) = 388.8 TB
Using the updated AWS Global Accelerator pricing (as of May 22, 2023):
- Data transfer cost (averaged across regions): 388.8 TB * $0.035/GB = $13,608
- (Also need to add EC2 egress cost, 388.8 TB * $0.02/GB = $7,776
- Total data transfer cost = $21,384
Accelerator:
- Assuming 1 accelerator with 2 endpoints (1 for HTTP/2 and 1 for HTTP/3)
- Accelerator cost: 1 * $18/accelerator/day * 30 days = $540
Total estimated monthly cost for AWS Global Accelerator: $21,384 (data transfer) + $540 (accelerator) = $22k
Former Dropbox employee, just correcting one assumption:
> Dropbox's peak traffic is 1,500 queries per second (QPS).
I can't speak to search QPS directly, but most individual serving hosts for file sync/retrieval were receiving tens of thousands of QPS. The overall edge QPS peaked at several hundreds of thousands QPS every day across all the hosts. So I'd guess that even just search is an order of magnitude higher than 1,500 :)
