Hetzner cuts traffic on US VPSs, raises prices

themgt · 2024-11-28T22:37:51 1732833471

As a Hetzner bandwidth enjoyer affected by this, this is why (HN cough) multi-cloud/dedi k3s is great, because if you get rug pulled you just migrate to another provider with better prices.

That said, $1/TB for bandwidth overage seems pretty fair. I empathize with the complaining but if the new price is such a ripoff everyone should be recommending what cloud VM provider they're migrating to for a better deal.

fastily · 2024-11-28T23:34:29 1732836869

I use OVH (VPS’s specifically), which offers unlimited bandwidth. In my experience they’ve been both reliable and affordable which is a rarity. I run a few applications that require high amounts of bandwidth, so silly caps like the ones that Hetzner are imposing are a non-starter for me

Aachen · 2024-11-29T00:28:07 1732840087

That's what a friend said also. If you look beyond the marketing material, hoever, the ToS says:

> OVHcloud reserves the right to restrict the VPS Service bandwidth to 1 Mbps (1 Megabit per second) until the end of the current billing period in cases of excessive use by the Client

but it advertises with "unmetered"... so is a meter attached by which they can tell whether your bandwidth use is excessive or not? Would they eat those costs for you?

I checked out some numbers. Quoting myself from chat history:

> it begs the question: what's "excessive"? I dunno but if they charge $5/month for the VPS and, while AWS may be ~1/3rd cheaper [than some other thing], that's still on the order of 70$/month. And AWS has insane economies of scale working for them, maybe their cost price is $7/month if they don't need to have a competitive price but that's still a loss then

> I bet you'd win the lawsuit where [OVH] falsely advertised with unmetered 500mbps and a terms of service saying "excessive", so when you transfer 2 TB/day on a connection advertised to be capable of 500mbps×24h = 5.4TB/day... that's reasonable right? But then you're having a lawsuit over a 5$/month VPS

matt-p · 2024-11-29T02:12:53 1732846373

> is a meter attached

Yes, of course. Having flow data (or monitoring ports/interfaces) for traffic engineering and management is pretty essential, not least for determining when capacity upgrades are needed.

I understand both sides of the argument here. The idea of offering "unlimited" is appealing because most users of a typical 2GB RAM virtual machine (as an example) consume less than 1TB of bandwidth per month. Offering unlimited bandwidth removes the hassle of overage charges/billing queries and eases customer concern/friction. Both sides benefit from this.

However, on the other hand, is it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth?

Should providers avoid offering unlimited bandwidth unless it's truly unlimited? From an engineer's perspective, yes, I agree. But this stance also risks degrading the experience for the 99.5% of "normal" customers—those who don’t exploit this simplification of "free bandwidth"—just to address a handful of users who take full advantage of it.

It's tough, so IME most such providers leave something in their terms that allows them to intervene in extreme cases but typically exercise restraint in doing so, usually only doing it manually if they notice that 'extreme' usage is damaging other users experience e.g it's serious and prolonged usage.

rendaw · 2024-11-29T05:35:18 1732858518

> it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth

It's reasonable for OVH to prevent that, and it's also reasonable for OVH to explicitly and clearly define the limits up front.

oarsinsync · 2024-11-29T15:07:37 1732892857

It’s also reasonable for OVH to not do that, as most of their customers don’t understand 95th percentile billing, which is the model that they’re being charged at by their transit suppliers.

It’s also reasonable for OVH to not do that, as most of their customers dont understand that transit costs blend with port costs depending on destination, and some destinations are effectively ‘free’ to send/receive from (fixed port costs only, no marginal costs), and other destinations are not (marginal costs associated with transit supplier fees).

The billing model consumers want is a simple BW used calculation, without facing the reality that if they consume their entire BW allowance as quickly as possible, it incurs order of magnitude higher costs than if they consume it at a trickle over the whole month.

rendaw · 2024-11-29T19:45:26 1732909526

You're acting like it's complicated, but they could just not say "unlimited" here: https://www.ovhcloud.com/en/bare-metal/bandwidth/

Like:

> Generous bandwidth limits*

> *see this link for details

oarsinsync · 2024-11-30T17:38:39 1732988319

It’s worth going back to the start of this thread and seeing that this all started with someone complaining that their provider had reduced the BW allowance and wanting somewhere with more generous allowances. When the provider sells things for what it actually costs, the customer gets upset and looks for someone selling a subsidised product in a misleading way. Leading to other people getting upset about the mislead. Those people should go to the original provider, who is doing exactly what they asked for!

matt-p · 2024-11-29T19:12:55 1732907575

kind of true except if you've got 1000's of customers it all evens out and your traffic profile is actually quite smooth if you've got say 6 X 100G transits, 4 X 100G IXP ports and 5 X 100G PNIs then the impact of an individual 1G customer is not even noticeable, honestly. We can work to 1Mbps 95%ile being about 250GB of total transfer at scale.

oarsinsync · 2024-11-30T17:39:12 1732988352

I completely agree. The product being sold has only a loose connection to the cost incurred by the provider. This is why the product being sold is being sold in a vague / loose-ish way, because for the overwhelming majority of customers, the product being sold can be sold for a profit.

saurik · 2024-11-29T02:54:54 1732848894

Doing billing like this is awkward as you just don't know what you are going to get. And, in a world where most people are in fact using small amounts of bandwidth, what you have done is caused people who use small amounts of bandwidth to pay MORE than they should, as they are effectively paying the price of the bandwidth for the average user: if you use less than the average, you are subsidizing the people who use more than the average.

Meanwhile, in an ecosystem where everyone isn't already being ripped off with overly-expensive bandwidth, if an ecosystem-level event happens that causes the average user to suddenly use more bandwidth, the service either has to raise rates for everyone or they have to start claiming some uses of bandwidth are "egregious".

The result is then that, to defend the small-scale user from paying even more than the too much you are already charging them (as they are subsiding the larger users), you suddenly start doing traffic analysis with price discrimination by use case, and network neutrality goes out the window :/.

The real reason any of this works is just that people in fact aren't being charged fair prices most of the time, and these unlimited plans let the provider hide that from all involved. If everyone were charged a fair price, not only would heavy users pay a lot and light users pay LESS than they often do today, but everyone would be paying little enough that this idea that it is a big customer "concern" goes away, the same as it is for electricity or water: except in extreme circumstances, no one frets over sudden utility overages.

RandomThoughts3 · 2024-11-29T06:24:11 1732861451

This is a for profit business not a limited common ressource being shared.

What you are missing here is that the adjustment is not low usage users subsidising high usage users, it’s OVH margins. Nobody is being subsidised. Low usage users just make OVH more money than high usage users. OVH doesn’t mind because per user costs are actually low and they are already competitive at that price without adding more complexity to their product mix. Users which would lead to an actual loss are rate limited.

saurik · 2024-11-29T13:57:18 1732888638

> What you are missing here is that the adjustment is not low usage users subsidising high usage users, it’s OVH margins.

I did not miss this, and it was part of my point: the only reason this makes any sense at all is because these providers are ripping people off on bandwidth, which is how they have a margin so large that they feel a need to hide it from people under this kind of ridiculous pricing abnormality.

What is awkward is just accepting that and helping to make it worse by advocating for making it easier to kind of hide that fact: bandwidth is a commodity product, and these pricing games aren't pro-consumer because they somehow help people not have to worry about one month getting ripped off too much... they are anti-consumer because they enable the perpetuation of the state of affairs wherein people get ripped off in the first place.

The bandwidth providers know this, but they--of course ;P--like their excessive margins... but, if you just stopped claiming this was pro-consumer and realized what was actually going on here, the idea that a margin so excessive as to be able to essentially make the usage for the median user irrelevant should indicate a nigh-unto-ridiculous level of market distortion.

Like, we shouldn't sit around and just tolerate these margins. And that this particular pricing trick helps make these margins a bit more stomached by people really sucks! And in some sense I get it that it does make it easier to stomach... but... only because I think people are just buying into the idea that this must be a reasonable price :(.

And--even then--it doesn't fix the other problem I talked about (which I explicitly hedged as being in the world where the price wasn't set up to gouge everyone): when Facetime came out, it overnight was going to cause everyone with an iPhone to suddenly need more bandwidth, and so network providers temporarily needed to ban it or charge more for it; we see the same thing with the step up to video streaming services from basic web browsing, leading to providers feeling a need to zero-rate.

The reality is that bandwidth IS a limited common resource being shared at that provider--the same as any other product where the price isn't being distorted: this is the whole reason we use markets for this stuff in the first place--and the pricing of it at different providers should encounter market forces to drive it down closer to cost... except we are trapped in a local minimum here by people who refuse to understand that unlimited schemes cost more, not less.

RandomThoughts3 · 2024-11-29T15:36:08 1732894568

You don't actually tolerate high seller margins. Hosting is a competitive market.

If there was significant gains to be made by being more aggressive on the low end of the market, providers would already be doing it (and they are - OVH 5$ offer is quite aggressive). There is a reason nobody actually offers a better deal.

immibis · 2024-11-29T10:52:56 1732877576

If you pay $5 and use $100 of bandwidth costs, you are in fact being subsidized by other users, not by margins. We don't know what OVH pays for bandwidth though.

RandomThoughts3 · 2024-11-29T15:33:19 1732894399

But nobody can do that. OVH doesn't let you be a large net negative.

What happens is roughly that:

- You are costing 1$ (bandwidth, etc.). You make OVH 4$. They are happy. Nobody offers you a cheaper alternative so you are stuck paying 5$ anyway.

- You are costing 4$ (bandwidth, etc.). You make OVH 1$. They are happy as marginal costs are low anyway.

- You are costing more than 5$. OVH severely rate limit your bandwidth to cut their costs and wait for you to leave because the service is now useless to you.

michaelt · 2024-11-29T12:45:44 1732884344

If I order some shoes from Amazon, I find them uncomfortable, and I return them for a full refund causing Amazon to incur a loss - have I been "subsidised" by other customers?

Personally I would say if Amazon makes a profit selling you a book and makes a loss shipping me some shoes which I return, the loss was paid by Amazon, not by you.

Aachen · 2024-11-30T08:20:54 1732954854

The comparison is more apt if you gained something (because the bandwidth user gets a product out of it), say by having worn the shoes for a day and doing this every day so you get free shoes for life. Then, yes, it's pretty clear the paying customers are the ones footing your bill

wang_li · 2024-11-29T14:51:45 1732891905

Yet book prices will be adjusted so, on net, Amazon does not lose money.

ghssds · 2024-11-29T06:02:08 1732860128

View this as an insurance and it suddenly all makes perfect sense. You pay a little more so if this month your usage pikes, you won't get a surprise invoice you didn't budget for, and you won't get cut either. At worst, you'll get rate-limited. This price stability is valuable and paying extra to get it isn't being charged an unfair price. Of course if you don't find it beneficial, you should choose another offering.

saurik · 2024-11-29T13:58:59 1732888739

This is only relevant because the cost of bandwidth is excessively high--much higher than it should be--and so people essentially need to pay for this gouging-insurance.

matt-p · 2024-11-29T19:26:59 1732908419

So, the "fair price" for internet bandwidth in Europe/NA is typically between a tenth and a quarter of a single cent per GB transfered in the heaviest direction.

So you prefer to pay $4.50 for your vm + 47.12126 cents for 460Gb data transfer , rather than $5 for your VM with unmetered data transfer?

I think by the way that the sensible answer is what DO/Linode etc do which is allocate some included data transfer per VM and pool it across your account. That's honestly a very sensible balance from my viewpoint, but they then charge you quite alot for overage around 1-2c per GB which is ~10X the "fair price".

Aachen · 2024-11-29T10:37:09 1732876629

> > is a meter attached

> Yes, of course.

So it's not unmetered as advertised or am I misunderstanding that word?

> this stance also risks degrading the experience for the 99.5% of "normal" customers—those who don’t exploit this simplification of "free bandwidth"

How so? If they want to be relaxed about it, the terms can say that you can burst more (e.g. "you can use 500GB/month, and burst to 5TB for two months of every two-year period; we'll send you a notification email whenever this happens so you're not caught by surprise"). If they don't want to be flexible, they can mention the hard limit that they are going to enforce regardless of whether they call it unlimited without asterisk. Either way, the buyer would know what they can actually use and doesn't have to guess

whatio · 2024-11-29T12:02:28 1732881748

“Unmetered” means “You will not be charged under normal circumstances based on the measurement of the data you use.” It does not mean that your traffic is literally not measured.

They don’t put a specific hard limit because doing so both limits their own flexibility as a service provider and creates a target for abuse by users.

Dylan16807 · 2024-11-29T19:22:55 1732908175

For some definition of "normal circumstances". Being a bigger user should fall within it or that's not accurate advertising.

Some places will offer a choice between faster metered and slower unmetered. That seems like a good compromise to me. A nice big link should cost the host a single digit number of dollars per 100Mbps, so it's not hard to find an option where everyone is happy with the speed and pricing.

whatio · 2024-11-29T19:30:29 1732908629

If you want a contract that has every term and circumstance negotiated up front, you’re going to need to speak with Hetzner’s business development team. You’ll also need to be a bigger fish than a single hobby developer.

Dylan16807 · 2024-11-29T23:09:00 1732921740

Hetzner has terms up front just fine. $1/TB.

The unmetered issue involves other providers, and plenty of them are up front about it too.

chipdart · 2024-11-29T06:56:22 1732863382

> However, on the other hand, is it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth?

Irrelevant. If you sell a vCPU with enough bandwidth to feed your 1GBps 24/7/365 needs, and you charge $5/month for it, then it matters nothing what's your personal notion of reasonable. What matters is the service plan offered by the cloud provider and the performance indicators they are contractually obligated to meet.

oarsinsync · 2024-11-29T15:17:41 1732893461

> What matters is the service plan offered by the cloud provider and the performance indicators they are contractually obligated to meet

Indeed, and those indicators are specified in the contract, not in the headline product description. There are a lot of people unhappy that those indicators in this contract are not specific enough. Those people shouldn’t buy these contracts.

(Also, if you use your 1Gbps port at full speed at the most peak time for bandwidth utilisation, for 37 hours in a month, and not at all outside of that, assuming 20 cents a megabit with 95th percentile billing, the costs you’ve incurred to your provider are $200. Also it doesn’t matter at all what you do after those 37 hours, the costs to the provider are the same. You doing 300TB in a month costs the same as you doing 16TB, if you do the 16TB the ‘wrong’ way.)

matt-p · 2024-11-29T19:39:57 1732909197

Thats only true if all your customers choose the exact same 37 Hours (and the same/similar destinations) back in the real world that's very very unlikely and so the 95%ile "issue" is a bit misleading unless an individual customer has the ability to use more than a couple of percent of your overall capacity (rare-ish at scale).

oarsinsync · 2024-11-30T17:36:43 1732988203

I completely agree. The product being sold has only a loose connection to the cost incurred by the provider. This is why the product being sold is being sold in a vague / loose-ish way, because for the overwhelming majority of customers, the product being sold can be sold for a profit.

derefr · 2024-11-29T01:28:38 1732843718

> is a meter attached

Probably not by default, but if your usage starts to saturate their network switches they’ll add one, to figure out who’s disrupting everyone else’s QoS.

derefr · 2024-11-29T16:07:32 1732896452

Forgot to mention, OVH makes the current topology and traffic saturation of their network switches public: http://weathermap.ovh.net/

(I think they mostly do this so that customers can see and verify that any DC-level peering relationships, or per-customer site peering contracts [a.k.a. "OVHCloud Connect"] are being taken advantage of to flow the customer's traffic. But it's convenient for other things, too.)

silisili · 2024-11-28T23:57:04 1732838224

OVH is the most set and forget experience I've ever had. They email me maintenance notices 3 or 4x a year, but I don't think I've ever had any downtime. It just stays happily humming along for years. I think I pay something like 60 a year for it.

markvdb · 2024-11-29T00:21:49 1732839709

OVH and reliable, in the same sentence? They're cheap, so suitable for projects you don't mind going poof.

Personal anecdote. A few years ago, I lost a lot of sleep on a domain renewal at OVH. Their incompetence was mind-boggling. A less common tld was the only slightly challenging bit. After a week of calling and emailing, and on the verge of the domain lapsing, I gave up and sent someone to the tld registry with cash.

Also, do search for OVH SBG2 should you have missed that.

tredre3 · 2024-11-29T00:55:06 1732841706

OVH's infrastructure is absolutely very reliable.

The pain begins when you need support. Just like you, I have lost a lot of sleep over domains held hostage by their incompetence (for almost a year in one instance). Lesson learned, never use OVH for domains.

The support for their dedicated servers is just as bad, mind you, but short of a hardware failure you really don't need them. I have several years of uptime on all my current services.

So for personal projects their vps/dedicated is still a fantastic value.

jsheard · 2024-11-29T01:01:40 1732842100

> OVH's infrastructure is absolutely very reliable.

Well except that time one of their datacenters burned down, likely due to insufficient fire suppression, and the data backups were also lost because they kept them in the same building as the originals.

https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-...

These reports criticize OVHcloud for having no fire prevention system and no power cut-off on the site, for using wooden floors, and for a free-cooling design that created airflows that spread the fire. The reports also say that water was detected near electrical systems before the fire broke out.

It takes quite a while to regain trust after shitting the bed that badly.

ggeorgovassilis · 2024-11-29T01:30:18 1732843818

> data backups were also lost because they kept them in the same building as the originals.

I asked that question Hetzner support and they admitted doing the same

jsheard · 2024-11-29T01:33:36 1732844016

Better to be honest about having weak backups so users can plan accordingly, than to lie about how safe the data is and lull users into a false sense of security...

the OVH contract relating to automatic backup stipulates that a backup of the VPS server is scheduled daily, exported, and then replicated three times before being available in the customer space, and that the storage space allocated to the 'back-up option is physically isolated from the infrastructure in which the VPS server is set up.'

tyingq · 2024-11-29T00:40:02 1732840802

They can be terrible. But it is dirt cheap. So if you use their stuff in a way that you can recover somewhere else if they have issues, you save a lot.

As you mentioned I would stay away from them for things like domain hosting. Just use them for cheap compute, etc.

mst · 2024-11-29T19:14:02 1732907642

I keep my whois and NS entries elsewhere and my nameservers sufficiently distributed that I find the risk acceptable, but both hetzner and ovh are firmly in the "I have always felt I got a very cost effective 'exactly what I paid for' - and in the case of my rare interactions with their support more than I'd hoped for" category for me.

Neither has ever caused me a problem that didn't feel like "potentially having this level of problem occasionally is entirely in keeping with how little I'm paying" basically.

rapind · 2024-11-29T04:58:45 1732856325

Would you recommend vultr for dedicated metal? I’m lazily shopping around for a decent dedicated metal setup, but I need something reasonably reliable.

rat9988 · 2024-11-29T14:47:53 1732891673

Hetzner and Ovh are very reliable unless you need to reach to support.

dabedee · 2024-11-29T07:48:08 1732866488

From personal experience a few years back, OVH support was one of the worst I have ever experienced in my career. Technical incompetence at multiple levels of the chain (e.g lack of understanding of how DNS works). I would never recommend it to anyone, not even my worst enemy.

codexon · 2024-11-29T22:00:08 1732917608

I can confirm this.

I had packet loss on my server. They asked me several times to reboot my server into rescue mode and leave it there for 10+ hours until their senior technician could look into it at an unspecified time of day.

After a month of doing this 3-4 times, they finally admitted that their switch is overprovisioned and there was no ETA. This problem happened in 2 locations.

Also had a problem with the failover ip failing to move. Again they told me to reboot into rescue mode and leave it like that for hours. No fix.

I've left OVH entirely after being a customer of theirs for over 10 years.

fl0id · 2024-11-29T03:13:01 1732849981

Yeah,just hope you never have to interact with support

BonoboIO · 2024-11-29T04:20:58 1732854058

As long as their their wood floor datacenter and your backup in the same location does not burn down

„… data center had wooden ceilings, no extinguisher, and no power cut-out“

https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...

mst · 2024-11-29T19:16:59 1732907819

The systems *I* currently have at hetzner are, so far as I'm aware, on a "we don't charge for bandwidth but if you use a shitload you'll get throttled for the rest of the month" plan just like my ovh boxen.

But I only pull dedis from hetzner; my VPSen are all ovh based. So please nobody expect my experience to generalise without triple checking the terms just like I did in the process of signing up for those systems.

jitbit · 2024-11-29T11:50:42 1732881042

I wouldn't say "affordable". Looking at their pricing page right now. A US-based 64GB RAM cloud server in OVH - $539

AWS is $220 (us-east, r6a.2xlarge instance, 1yr reserved)

dist-epoch · 2024-11-29T12:16:52 1732882612

If you reserve in AWS for a year, you might as well get a dedicated Hetzner server:

70 EUR for 64 GB with 8c/16t, 1 TB local NVMe

123 EUR for 128 GB with 16c/32t, 2 TB local NVMe

tokinonagare · 2024-11-28T23:37:08 1732837028

I've only ever been on OVH and was surprised to discover a few years ago that bandwidth is not only unlimited but also costly at most other hosting companies (including cloud ones).

9cb14c1ec0 · 2024-11-29T00:21:15 1732839675

Shout-out to Microtronixdc, my colocation provider. For their $55/month/U colocation package, they provide unmetered gigabit for no additional cost.

https://microtronixdc.com/

mrbluecoat · 2024-11-29T15:29:51 1732894191

Their option to colocate a Raspberry Pi (or similar form factor) for $5/month is also interesting as a VPN exit node.

diggan · 2024-11-30T19:32:05 1732995125

Based on their broad "Acceptable Use Policy" (https://microtronixdc.com/acceptable-use-policy), I'd wager a guess they'll find a reason to shut that down.

jfreak53 · 2024-12-02T21:27:44 1733174864

We have a few exit nodes already on the network.

mrbluecoat · 2024-12-01T13:33:14 1733059994

Yeah, I also later noticed they charge $46/month for 4 amps whereas the Raspberry Pi 5 requires a 5V, 5A power supply to take full advantage of its processing capabilities and features.

daemonologist · 2024-12-01T14:32:27 1733063547

I assume the 4 amps they charge for are at 120 V (480 W), while the Pi only needs 25 W plus some rectifier inefficiency.

jfreak53 · 2024-12-02T21:29:00 1733174940

Its based on wattage, 4A yes, but 4A multiplied by the voltage. So Pi 5 is 5A, but multiple that 5A times 24V dc and you get 120watts :) so yes, you can definitely colo a pi 5 my friend :D

jfreak53 · 2024-12-02T21:27:24 1733174844

Thanks for the shoutout!! :)

Aachen · 2024-11-29T00:43:41 1732841021

I see terms for hosting and domain registration and managed databases, but not colocation. Do you know if they really not care if you actually use that connection to the stated allowance?

9cb14c1ec0 · 2024-11-29T00:55:43 1732841743

https://microtronixdc.com/products/colocation

Their big selling point is that they truly provide unmetered connections.

jfreak53 · 2024-12-02T21:29:42 1733174982

Yep, we do!! :)

shlomo_z · 2024-11-29T02:50:58 1732848658

Bandwidth for these providers is super cheap. The main cost is power consumption.

jfreak53 · 2024-12-02T21:30:10 1733175010

This is true, power is our largest expense.

arcastroe · 2024-11-29T02:19:31 1732846771

Exactly! I had a three node k3s cluster hosted on OVH. When I decided to switch to Contabo, it was as easy as adding the three Contabo nodes to the cluster, and then removing the three OVH nodes (plus updating some DNS rules). It was the easiest and simplest migration I'd ever done. All my services and data just moved automatically and mostly with zero downtime. The only service that experienced a little downtime was Plex, which as I understand does not support high availability. If I ever find a cheaper host, I'll simply switch over. No hassle and no vendor lock in.

lysp · 2024-11-29T07:44:20 1732866260

This is what I also want to do.

How are you handling storage? That is the only issue I'm struggling with for a small 3-node deployment.

arcastroe · 2024-12-01T07:18:39 1733037519

Longhorn volumes automatically replicate to multiple nodes (configurable) and automatically move to the nodes whose pods need it.

A postgres database running on a single node will experience some downtime during (re)deployments or when moving across nodes, but should be pretty quick depending on the size of the database. For true HA database, CockroachDB is supposed to be compatible with postgres, but I haven't had a chance to play with it.

5Qn8mNbc2FNCiVV · 2024-12-03T15:21:01 1733239261

How's the performance with Longhorn with databases?

arend321 · 2024-11-29T09:55:34 1732874134

That's indeed the pain point. Distributing a stateless app is relatively easy. Distributing the shared file system and database over a remote, higher latency, cross-cloud setup is hard.

WildGreenLeave · 2024-11-29T14:00:44 1732888844

I run everything on Longhorn, including databases. Not the highest iops but definitely worth the hit for the easy of migrating.

rmbyrro · 2024-11-29T11:12:46 1732878766

The OVH Eco-line (Kimsufi, So You Start..) are incredibly good and affordable. I have a few beefy servers, which I use to create my own VPS service on top of Proxmox. I'm not a sysadm and it's still simple to setup and maintain.

Eco is clever, because they reuse good hardware pieces to assemble new servers, instead of throwing out as garbage...

Have been a very happy user with several servers for quite some time.

princevegeta89 · 2024-11-28T22:51:27 1732834287

As someone who's already using the dedicated server for a ton of things, I have been really grateful. But now, I have a new question, are they going to do this to their dedicated servers as well?

dantillberg · 2024-11-28T23:02:26 1732834946

This change only affects VPS services in the US, and for what it's worth, Hetzner does not offer dedicated servers in the US.

consumer451 · 2024-11-28T23:04:40 1732835080

I am very ignorant here, so my apologies, but...

When someone runs a dedicated server these days, does this mean a one-off linux install? Or is this more likely to be a docker install so that it's portable?

princevegeta89 · 2024-11-29T00:19:37 1732839577

It's an actual entire machine given to you. I remember there were a few options for me from Ubuntu, Debian to Red Hat to choose from, but all of them would also have preconfigured system users and some level of administration done by the provider.

But other than that, it's an actual bare metal machine and I installed Ubuntu on it and threw in a giant heap of services that have been running on it for more than a year now.

consumer451 · 2024-11-29T00:56:45 1732841805

If you could rewind the clock, would you have started setting it up any differently, like in a container?

I am just curious what your options would be now if you wanted to migrate. Would you just copy your bash history to a local text file for reference, and then repeat the steps on a new server?

princevegeta89 · 2024-11-29T03:55:00 1732852500

No, I wouldn't have started differently and I like the performance and the dedicated hardware I get for the money I spend. I have a custom backup solution that will upload daily backups of all my data to remote drives and I should be able to restore the setup on another machine without much problem.

sneak · 2024-11-29T02:22:29 1732846949

Generally even in containerized deployments, you run one container per service/process. You wouldn't run everything you’d run on one box in one container.

I definitely recommend using docker compose or similar even in a one node deployment versus just installing and running things on the host linux system like it’s still 1998. Having a single directory to back up and a single file defining all of the services that can easily be redeployed is very convenient.

RandomThoughts3 · 2024-11-29T06:36:14 1732862174

The performance impact seems completely unjustified for most of the things people do with a personal machine however.

consumer451 · 2024-11-29T07:16:45 1732864605

What is the performance impact? Going one page into Google results, I found this paper. Is there a better reference?

> At light workload levels, the native host performs better than Docker. However, as the workload increases, both Docker and the native host show similar performance, with the difference getting smaller

https://www.researchgate.net/publication/376557310_Comparati...

RandomThoughts3 · 2024-11-29T09:22:20 1732872140

> What is the performance impact?

DNAT, layered file system access, likely duplication of libraries if you don't pay much attention to your containers.

sneak · 2024-11-29T09:33:57 1732872837

The performance impact is approximately zero.

ajb · 2024-11-28T23:17:31 1732835851

Replying here as your other question is at max thread depth:

A non virtualized Linux install isn't more locked in than a docker install, as for a bare metal server you are choosing your own OS. I have done the docker thing on a bare metal server, but that's because I wanted to run multiple services on it and isolate them operationally.

consumer451 · 2024-11-28T23:23:29 1732836209

> A non virtualized Linux install isn't more locked in than a docker install

Again, sorry for my ignorance here, but if not virtualized, how does one move hosting providers otherwise? My experience is limited to either running all the bash commands in an install readme, or installing a docker image.

So there must be something in-between, to recreate a linux install elsewhere?

> Replying here as your other question is at max thread depth:

btw, you can click on the time of the post, and reply there when there is no reply link in the main thread.

0xFEE1DEAD · 2024-11-29T02:32:14 1732847534

Using dedicated servers doesn't mean you're not using virtualization - it just means you're the one managing it. You control the hypervisor and the vms running on top of it. Because of that, you're actually less tied to a specific hosting provider since you're not reliant on their APIs to set up and manage your infrastructure.

Even if you're not using virtualization there are still plenty of ways to migrate your servers.

One of the most common approaches (which was the thing before docker took over) is managing servers with an IaC approach using tools like chef, puppet, ansible, saltstack etc.

With IaC you define your entire infrastructure in configuration files and deploy those configs to your host. It's a bit like docker swarm but for managing physical and/or virtual servers instead of containers.

Another popular option, often paired with IaC, is to create your own pre-configured *nix images tailored to your needs. For example, you might have specific images set up for your load balancers, db servers, file hosts, or other roles in your stack.

I've worked at a company where we handled migrations using dd. Technically that's also an option. Wouldn't recommend it tho.

ajb · 2024-11-28T23:43:09 1732837389

>So there must be something in-between, to recreate a linux install elsewhere?

There's a class of tools like chef, ansible, puppet.

Or you can just package your services into debs and run a Debian repo to install them from, same as the base OS

Possibly there's something closer to dockerfiles out there by now as well

>btw, you can click on the time of the post, and reply there when there is no reply link in the main thread.

Cool, thanks!

Makes me puzzled about the point of concealing the reply links, I guess it just adds a bit of friction?

djbusby · 2024-11-29T00:09:33 1732838973

It's a fire extinguisher feature; for when the thread has too many, too fast replies.

Phil_Latio · 2024-11-29T00:23:10 1732839790

If the server hoster supports it (Hetzner apparently does), you can enable KVM and install a previously prepared image. If the server hoster & hardware supports it, you can login remotely to the server management interface (like HP iLO) and install an image this way.

If you don't have above options or simply don't want to do it this way, you can also bootstrap via SSH. But instead of manually typing in shell commands, you will automate it in some way with custom scripts and/or tools like Ansible.

lupire · 2024-11-28T23:36:24 1732836984

You can use a configuration manager like Puppet.

https://en.m.wikipedia.org/wiki/Puppet_(software)

thrw42A8N · 2024-11-28T23:33:27 1732836807

Talos Linux, Flatcar Container linux...

andai · 2024-11-28T23:32:08 1732836728

Isn't a Docker image an OS image and a bunch of shell commands?

kuschku · 2024-11-29T15:05:00 1732892700

Imagine you'd set up your own server at home.

You'd buy a computer, plug an install USB drive in, and install ubuntu.

Then you'd connect to it via SSH, configure it, maybe install docker and set up your docker containers, etc.

A dedicated server is very similar.

The server is sitting in a datacenter at hetzner, and you usually install an OS with a button in the management UI, sure.

But everything afterwards is the same. You just connect via SSH, install docker or k8s and your services, maybe an nginx, etc.

You also have an option to request KVM access. That allows you to control the server as if you had connected a keyboard and monitor to it. You can even enter the BIOS to diagnose issues, if you'd like.

Personally I've got an install script that automates everything and sets up kubernetes and automated encrypted backups. Then I just deploy everything else with k8s.

fabian2k · 2024-11-28T23:07:35 1732835255

For Hetzner you rent actual physical servers. There is nothing virtualized there, it's real hardware without abstractions.

consumer451 · 2024-11-28T23:09:45 1732835385

Thanks, I guess what I meant to ask is that is it normal for people to virtualize their own dedicated server these days by default?

Would this be a best practice to avoid hosting vendor lock-in?

Filligree · 2024-11-28T23:17:58 1732835878

There's no lock-in possible. It's a bare metal Linux machine, you do whatever you want with it; you can replace it with the PC under you desk if you want.

If you want to run k3s, k8s or docker, you can, but personally I find those too complicated. NixOS is much easier to deal with, and achieves the same result.

conradfr · 2024-11-29T11:19:25 1732879165

Sure, I use CapRover on mine.

BrutalCoding · 2024-11-29T00:58:49 1732841929

Yeah, it's a one-off install. In my case, I did Proxmox[1] for a while with VM's and LXC's, with some of my VM's and unprivileged LXC's running Docker (compose) too because it made the installation of said software easier. It's great, but I'm moving over to switch to Debian with Incus[2] instead of Proxmox. Just for fun mostly.

[1] https://www.proxmox.com/en/ [2] https://linuxcontainers.org/incus/introduction/

srockets · 2024-11-29T05:49:44 1732859384

CSPs aren't your cellular provider. You don't get better pricing by telling them you'd switch, because both the AE and yourself (assuming you ever did the math), know it's not a viable option.

Trying to be multicloud by choice, unless you have a very unique use case, which you probably don't, is simply admitting you are incapable of calculating the cost of being multicloud. This would get you horrible pricing, as you just showed your hand.

ndjdjddjsjj · 2024-11-28T22:54:50 1732834490

It does feel like a case of the Costco hotdog going up to $2 followed by "grrrr. Thats it! I'm..... going to keep buying it because it is still damn cheap!"

BrutalCoding · 2024-11-29T01:13:38 1732842818

I remember reading something about the Costco hot dog story, quite funny IMO, here's what I just found from 2018: "I came to (Jim Sinegal) once and I said, ‘Jim, we can’t sell this hot dog for a buck fifty. We are losing our rear ends.’ And he said, ‘If you raise the effing hot dog, I will kill you. Figure it out.’ That’s all I really needed. By the way, if you raised (the price) to $1.75, it would not be that big of a deal. People would still buy (it). But it’s the mindset that when you think of Costco, you think of the $1.50 hot dog (and soda)." [1].

Turns out Costco has a new CEO this year, and again the hot dog topic came to light apparently, lol. This article is from 2024: "'To clear up some recent media speculation, I also want to confirm the $1.50 hot dog price is safe,' Millerchip said." [2].

Sources:

[1] https://www.425business.com/news/costco-ceo-craig-jelinek-on...

[2] https://www.usatoday.com/story/money/shopping/2024/05/31/cos...

Aurornis · 2024-11-29T01:43:13 1732844593

These stories are marketing mythology.

Think about. Imagine they sell 100 hot dogs per hour. A $0.25 difference means $25/hr in a store doing many orders of magnitude more revenue each hour.

It’s nothing. The way they play it up in the media gets a lot of attention and builds goodwill, but it’s entirely meaningless to their bottom line.

It’s amazing that people eat these stories up, though. I’ve heard so many people repeating this story as if it’s some amazing secret.

immibis · 2024-11-29T11:04:48 1732878288

When the price does increase, you'll know that there's a new CEO who's lost all connection to reality (in the same way that always happens when you put a person in front of an abstraction without obvious leaks).

Moru · 2024-11-29T09:05:04 1732871104

It's also Free Marketing worldwide. Can't compete with that price.

8n4vidtmkvmk · 2024-11-29T00:27:54 1732840074

Wait what? Did they actually raise the hotdog price?

I never buy the hotdog but a price raise is indicative of bad times to come.

kadoban · 2024-11-29T00:45:59 1732841159

Costco hotdogs have been underpriced for decades. It's an intentional choice, a loss leader. It raising would be mostly meaningless.

But no, last I looked a week ago or so it was still the same price.

gamblor956 · 2024-11-29T04:25:45 1732854345

It's not a loss leader. They still make a profit on the combo because the ingredients are cheap and it takes almost no labor to prepare on a per-dog basis.

And 1.50 is cheap now but was relatively expensive when the combo launched in 1985 (for comparison, a Big Mac combo with fries and a drink was 2.59, a KFC combo was around $3).

Aurornis · 2024-11-29T01:45:47 1732844747

It’s a warmed hot dog, a bun, some self-serve condiments, and it’s served by people who can do hundreds of them an hour.

$1.50 might very well cover the cost, but even if it’s a loss leader then it’s entirely negligible.

The mythology around the hot dog price makes people think they’re getting a screaming deal, though. It’s perfect marketing material.

andix · 2024-11-29T00:21:51 1732839711

Is it an option for you to just move the servers to one of Hetzner's Europe location? I guess a lot of high bandwidth applications don't require low latency.

0xblinq · 2024-11-29T05:26:51 1732858011

> this is why (HN cough) multi-cloud/dedi k3s

As long as you don't factor in the cost of the effort to make that work as seamless as it sounds.

api · 2024-11-29T00:15:49 1732839349

DataPacket, ReliableSite, HiVelocity, and FDCServers among others have great bandwidth prices.

jdhendrickson · 2024-11-29T00:18:22 1732839502

Can vouch for HiVelocity under the previous owners anyway, not sure what's going on over there now. My companies launch was bandwidth and compute intensive and they handled it well.

j45 · 2024-11-29T21:42:04 1732916524

Being cloud-agnostic is highly valuable. It's also possible to make determinations on where your needs are served best pro-actively, independent of service plan like this.

cchance · 2024-11-29T23:24:40 1732922680

Thats not a bad overage price, but then again cutting from 20 to 1 while raising prices seems fucking insane

S0y · 2024-11-29T01:34:45 1732844085

>multi-cloud/dedi k3s is great

As someone with the exact same setup, thank you for strengthening my confirmation bias.

bhouston · 2024-11-28T13:56:55 1732802215

> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced.

I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.

justinclift · 2024-11-28T14:23:12 1732803792

Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?

sdwr · 2024-11-28T15:49:55 1732808995

Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.

asyx · 2024-11-29T08:01:03 1732867263

Hetzner has been in this business for a very long time and I’m sure half the German small to medium sized companies use Hetzner. They have costs as well and with virtual servers they just assumed the usage pattern was similar to their eu data center and that didn’t work out so they increase the prices a bit. So that on average their calculation works out again.

Like, this is not Netflix tripling the prices over a few years.

gchamonlive · 2024-11-28T21:51:12 1732830672

That or they just didn't dimension well their prices and now it's biting them back.

I prefer to assume naivety over malice/negligence.

baq · 2024-11-28T15:54:21 1732809261

MBA value extraction 101.

yurishimo · 2024-11-28T14:40:06 1732804806

Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering.

The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.

bmicraft · 2024-11-28T22:31:38 1732833098

It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".

hyperpape · 2024-11-28T15:04:17 1732806257

As an engineer, if you don't qualify whether your answer is average or max, you've messed up.

drpossum · 2024-11-28T15:27:17 1732807637

I'm confident the marketing folks would just say something akin to "shut up nerd", cash their bonuses, and leave you with the problem.

cluckindan · 2024-11-28T15:36:03 1732808163

As an engineer, NEVER give out averages without checking that they are consistent with the distribution of the actual data.

phil21 · 2024-11-28T17:17:44 1732814264

Averages tend to skew once exposed to adverse selection.

Moru · 2024-11-29T09:09:13 1732871353

Don't think you find much engineers in the marketing department though.

tssva · 2024-11-28T14:29:17 1732804157

I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.

KomoD · 2024-11-28T15:58:52 1732809532

> I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan.

I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".

Dylan16807 · 2024-11-29T07:16:43 1732864603

Well, I doubt an extra $1/TB is going to stop Hetzner from being very price effective.

Aeolun · 2024-11-28T14:57:21 1732805841

Then why does this only apply in the US? Are they saying EU customers are well behaved?

citrin_ru · 2024-11-28T19:34:18 1732822458

It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.

freefaler · 2024-11-28T19:56:50 1732823810

- in EU there are large peering exchanges to swap traffic

- in USA no peering exchanges exists and you need to pay for your traffic most of the time. Few big operators in US and they enforce this.

Looks like some deal wasn't renewed and they lost a big chunk of cheap pipe or/and some of their upstream providers decided to do something with routing.

Slartie · 2024-11-29T09:11:57 1732871517

Also, Hetzner is way bigger in the EU than in the US. Good access to services hosted by Hetzner is thus more relevant to EU ISPs, because customers in the EU will probably use more services hosted on Hetzner infra. This gives Hetzner more leverage in the EU to negotiate beneficial conditions with regard to its uplinks and peering agreements.

eatery1234 · 2024-11-28T21:18:50 1732828730

Hetzner colo's for their US servers/locations. Perhaps their US bandwidth rates are much higher than their own DCs?

1oooqooq · 2024-11-28T15:18:15 1732807095

probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.

Hilift · 2024-11-28T15:16:00 1732806960

Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".

mrweasel · 2024-11-28T15:32:28 1732807948

Hetzner did pay somewhere around 20% of the cost of C-Lion1, so it wouldn't be surprising if it has hurt hem.

Hamuko · 2024-11-28T15:34:30 1732808070

Last I heard from Cinia was that the cable should be fixed by the end of November, so sometime within the next 55 hours.

master-lincoln · 2024-11-29T14:07:36 1732889256

Seems done https://www.cinia.fi/en/news/cinias-c-lion1-submarine-cable-...

BonoboIO · 2024-11-28T18:09:30 1732817370

8 years ago, hetzner had no cloud offerings

The dedicated servers still have 20 tb traffic included

lostmsu · 2024-11-28T18:11:32 1732817492

Do they have dedicated servers in US?

no_wizard · 2024-11-28T22:13:47 1732832027

In Hillsboro, OR they been building up their data centers but this hasn’t converted to dedicated servers in the US yet

nik736 · 2024-11-29T11:59:47 1732881587

Do you have a source for this? My last info was they rented space in an existing data center.

dumbledoren · 2024-11-30T02:57:43 1732935463

What? Afaik they are unmetered. And real unmetered - there are people who are saturating their connections 24/7 while using them.

paxys · 2024-11-28T14:47:01 1732805221

“Culprits” because they used the service they paid for within its advertised limits?

It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with?

They are simply deflecting blame for their own enshittification.

macspoofing · 2024-11-28T14:50:35 1732805435

>“Culprits” because they used the service they paid for within its advertised limits?

"Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.

paxys · 2024-11-28T14:52:59 1732805579

So then the culprits are the company’s own engineering and marketing departments for not correctly anticipating user demand.

macspoofing · 2024-11-28T15:44:16 1732808656

Sure .. also them.

Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses.

It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.

immibis · 2024-11-29T11:08:48 1732878528

Looked like? Sounds like it was a late fee. Look at us, we've got the cheapest plane tickets by $25 ($50 booking fee applies).

altairprime · 2024-11-28T18:00:46 1732816846

It’s not any more possible to correctly anticipate all pricing structure vulnerabilities, than it is to correctly anticipate all program and API security vulnerabilities. There is always a statistical chance of novel outcomes when humans are involved.

ghaff · 2024-11-28T15:02:54 1732806174

Or you take your best shot and then adjust as needed.

learnedbytes · 2024-11-28T14:17:56 1732803476

Want to give a clue on who the culprit is?

bhouston · 2024-11-28T14:24:26 1732803866

I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.

dpeckett · 2024-11-28T15:06:06 1732806366

This is typical of Hetzner, if a product SKU is losing money they very quickly make changes, even going as far as to discontinue the product entirely (eg. GPU servers). They definitely don't seem to be a fan of loss leaders.

I'm guessing somehow the traffic usage patterns of their USA customers was very different to their EU counterparts, or the cost of expanding network capacity was a lot higher than anticipated.

It's a bit of a shock for sure but it seems this model is a big part of how they can maintain their slim margins.

gnfargbl · 2024-11-28T16:01:51 1732809711

I have no complaints at all about this model. They work out the cost of providing a service, then they charge that cost plus a markup. They keep doing things that make them money. They stop doing things that don't make them money.

It seems like a straightforward way to run a business.

dpeckett · 2024-11-28T16:22:41 1732810961

Yep they're the technology equivalent of a discount supermarket. Everything is commoditized to the extreme.

Breath of fresh air in the modern cloud era tbh.

josephcsible · 2024-11-28T22:37:34 1732833454

I have one big complaint and one little one. The big complaint is that they didn't even give one business day's notice, and the little complaint is that they raised prices at the same time they cut what they were offering by 20x, instead of doing one at a time.

naniwaduni · 2024-11-29T05:19:52 1732857592

They're giving two business days' notice for new product and three months for existing product?

josephcsible · 2024-11-29T06:07:39 1732860459

December 1st's change isn't just for new customers. It's for newly-created or rescaled servers belonging to existing customers too, and it's plausible that those operations might happen a lot for some customers. And Thanksgiving and Black Friday are holidays for almost all American tech workers, so I'm not counting them as business days.

jjeaff · 2024-12-01T22:30:43 1733092243

Normally that would be ok, but considering the way many systems are setup to load balance and quickly spin up new servers and spin down un-needed servers on the fly, one business day would not give you enough time to revamp your system to work with a different provider.

Moru · 2024-11-29T09:24:13 1732872253

The bandwidth market is very different between EU and USA, maybe they weren't prepared for the much higher prices in USA? I'm pretty used to having a 100 Mbps connection to our servers that we can use without any strings attached. Even on the lowest tier. (Not Hetzner customer but been thinking about it)

tayiorrobinson · 2024-11-28T17:40:32 1732815632

To be fair, given how cheap a lot of Hetzners products (especially Server Auction, my beloved) are compared to the competition, not wanting to have loss leaders seems reasonable to me

dumbledoren · 2024-11-30T02:59:00 1732935540

Rather, the backbone providers dont do peering agreements and the traffic is very expensive, especially in the post-zirp inflation period. Europe is different - everybody peers with everybody so traffic is dirt cheap.

sigio · 2024-11-28T13:39:49 1732801189

Traffic over-usage is $1 per TB, so this is still quite fair, only in singapore is traffic really expensive at $8/TB.

jsheard · 2024-11-28T14:14:48 1732803288

> only in singapore is traffic really expensive at $8/TB.

Expensive by Hetzner standards but still cheap by cloud standards, egress from Singapore EC2 instances is between $80 and $120 per TB for example.

tr33house · 2024-11-28T14:48:43 1732805323

$80!! EC2 is a scam

jsheard · 2024-11-28T14:57:35 1732805855

It's $120/TB for the first 10TB in a month, so you need to be spending >$1200 just on bandwidth every month before you even get the "discounted" rates.

eatery1234 · 2024-11-28T21:21:38 1732828898

I know serverless is really expensive, but Vercel is about $400 for 1TB

leerob · 2024-11-28T21:35:50 1732829750

Vercel is $150: https://getdeploying.com/reference/data-egress

ea016 · 2024-11-29T10:21:24 1732875684

Vercel heavy user here. They have a very misleading pricing. It's "starting at $150", it varies depending on the region. I end up paying $400 / TB as we have a very international website.

grepfru_it · 2024-11-28T15:06:03 1732806363

The real savings only come in bulk

binarymax · 2024-11-28T14:33:09 1732804389

That’s very fair. I wish they had put that in the email!

geek_at · 2024-11-28T14:40:32 1732804832

That is right, it would have maybe reduced the public outcry. It just makes it 10$ more expensive per month which is totally OK in my book

jgalt212 · 2024-11-28T14:00:04 1732802404

so still order or orders of magnitude cheaper than the the big 3 hyperscalers.

tecleandor · 2024-11-28T14:24:44 1732803884

  AWS EC2: 100GB included and then $90 per TB.
  GCP Premium Tier: $120 per TB
  GCP Standard Tier: $85 per TB

That's a bunch of money...

glzone1 · 2024-11-28T14:39:10 1732804750

AWS drops to $50/TB - still 2x to 10x maybe? A lot of CDNs cost a surprising amount or hide pricing.

freefaler · 2024-11-28T19:12:06 1732821126

In Singapore and Asia in general traffic is very expensive, because there are no peering exchanges. So you need to pay for each Mbps you use. Big ISPs there are oligopolies and aren't too fast or keen to work with you.

jpalomaki · 2024-11-28T12:49:39 1732798179

Am I calculating right that 20TB per month means around ~60Mbits per second for 24/7? Not a network expert, but it is hard to see how this could be sustainable for less than €5/month.

Sounds a bit like the usual case where company is able to give a generous offering because most customers utilize just a small portion of it. Maybe with the attention they have been getting, they have attracted more bandwidth hungry customers.

summarity · 2024-11-28T13:13:50 1732799630

That is what they (explicitly) said in the email: users using next to no bandwidth were offsetting the costs for the heaviest users. That was no longer sustainable.

drpossum · 2024-11-28T13:27:51 1732800471

I think I'm missing something then. If that were true, wouldn't you lower the network allowance and then make the product cheaper or at least priced the same? This would have the no-traffic users paying "their fair share" and the overage costs (or higher traffic addon) would make up for the heavy users.

The current plan makes everything more expensive for everyone. They would do this if a) they never had a sustainable model in the first place or b) they were just being greedy

pas · 2024-11-28T22:38:36 1732833516

They recently entered the US market, so almost surely there was some honeymoon period (ie. investment to gain market share), but likely there's a significant difference between how the IXP/bandwidth/peering market works in the US compared to their home market (EU).

matt-p · 2024-11-29T23:53:51 1732924431

Pricing is fairly similar, particularly in ashburn. Europe can be a little cheaper, but usually not a massive amount in it.

pas · 2024-11-30T13:44:19 1732974259

Hm, interesting! Then probably lots of "bandwidth conscious" folks pounced on on their fancy new PoP.

Whatever the reason, rather strange to do this without even a few days of notice.

shubhamjain · 2024-11-28T13:01:16 1732798876

They already offer it on their European servers and it's still unchanged. Also, just because it's included doesn't mean everyone is using it to their full capacity.

immibis · 2024-11-28T14:39:18 1732804758

Internet is much cheaper in Europe than in most other regions and there's more of it. Europe is effectively the center of the Internet. Most of the time (in regions like Asia-Pacific and Africa) this is simply due to having more time and money to build it, but when comparing Europe to the USA, it's probably because of regulatory structure - more competition, less monopolization.

mardifoufs · 2024-11-28T15:04:09 1732806249

What do you mean? Is peering/transit actually more expensive in the US? Residential internet isn't really relevant here, and if it was, it would still not really make sense considering that Germany (where they have their main datacenter) has much worse internet than the US according to most speed/bandwidth averages. Because if you are not referring to residential ISP, I don't think there's any peering/transit provider monopoly in the US

immibis · 2024-11-29T04:38:58 1732855138

Yes. I suspect that because of the low number of residential ISPs there's less internet infrastructure in general and less competition. Who needs peering if there's only Comcast? You point out that networks can exist that don't serve residential customers, but who do they serve - mostly other networks which don't exist because they can't serve residential customers. The entire US is just the access region for a few networks to backhaul to NYC, Chicago and SF, not a full fledged network itself.

atwrk · 2024-11-28T13:47:17 1732801637

Is that much? I use a German competitor, Contabo, and they offer 32TB even with their entry-level VPS for 4.5€ plus taxes.

rsynnott · 2024-11-28T13:51:04 1732801864

IIRC a while back one of the CDNs had a post about their transit costs; they worked out _way_ lower in Europe than in the US. I don’t think Hetzner Germany is increasing prices here, just the US one?

shubhamjain · 2024-11-28T13:27:19 1732800439

Possible theory why they did it: To my understanding, you don't pay for traffic, you pay for network capacity. Maybe US instances aren't getting the uptake they had hoped for, and they are looking lower some costs by reducing their network capacity (they are a very German company and care a lot about efficiency).

Hetzner is very cheap and still profitable because classic "economy of scale" and vertical integration. They own, build, and operate all their data centers. This comment goes into more details[1], but it's possible this doesn't really work out in a foreign location like US.

[1]: https://forumweb.hosting/13663-why-are-hetzners-dedicated-ho...

esskay · 2024-11-28T13:47:08 1732801628

Counterpoint: They could've lowered their capacity without changing plans if they were indeed seeing lower uptake, and increase capacity as/when it's needed if it did indeed ever pick up. They wouldn't be the first provider to put products out of stock whilst they scale back up.

crest · 2024-11-28T14:24:48 1732803888

That doesn't explain a 20x regional pricing difference for the same service (outgoing bandwidth). It's either a malicious change to extract exorbitant rent from customers (hostages at that point) or buying enough bandwidth is whole lot more expensive than they calculated and it reflects costs to provide the product (a virtual machine a specific traffic cap).

If its mostly the later case they really fucked up their customer communication. They should care enough to provide their customers with time to respond and transparency to (re-)earn the trust a hosting provider requires.

mdasen · 2024-11-29T05:26:31 1732857991

If it were malicious, they would have increased prices in Europe as well. They could try and extract money from European customers just as much as American ones.

And the bandwidth pricing is still quite cheap, $0.001/GB. Major cloud providers usually charge nearly 100x that (AWS 90x, GCP at 85x or 120x depending on whether you want standard or premium, Azure at 80x or 87x depending on standard or premium).

You used to get 20TB for free and now you get 1-8TB free. If you had to pay for 19TB, that's another $19. If you had to pay for 19TB from a major cloud provider, that'd be $1,556-2,335. Even if you had DigitalOcean, they'd be charging you $0.01/GB (10x) and you'd be hit with a $190 bill.

I think the issue is that in the US, they don't have their own network. They peer with a single company in Virginia for 200Gbps and then pay for transit on 1Tbps. That's a lot of transit to be paying for - and 5x more transit than peering. In Europe, it looks like they have their own network between Finland, Germany, France, Amsterdam, UK, Austria, and Czech Republic. They also have a much better ratio of peering to transit. So they can use their network to carry the traffic to a lot of Europe and then maybe they have peering arrangements to handle most of their traffic when they need to hand it off.

In Europe, they're more likely to face owner economics while in the US, they're effectively renters - and the more their customers use, the more it's going to cost them.

If this were really a cash grab, it's a pretty terrible cash grab. It will certainly impact some users, but the maximum it will impact any user is $19/mo (if they're on a server with only 1TB of included traffic). But most people don't use many TB of traffic. Consuming all Bluesky posts in zstd compressed JSON is 30GB/mo. These weren't servers that had unlimited traffic on a 1Gbps port. They had a 20TB limit. If you were hitting that and intend to continue hitting that, it's another $12-19/mo.

If I had to guess, I'd say that they probably thought they'd expand more in the US and build out a fiber network here (as OVH has done), but that didn't happen and now they're looking at continuing to pay transit for the foreseeable future. Though I feel like the optics of this are pretty bad given that most people probably use less than 1TB and those users will still feel like something is being taken from them (even if they were never using that much anyway).

dumbledoren · 2024-11-30T03:19:45 1732936785

American traffic is expensive and they scarcely do peering agreements. And the backbone providers are probably jacking up the prices to boost revenue in these inflationary times. Eu users are not affected because the Eu invested heavily in the internet backbone and everybody peers with everybody there. In the US the isps cripple any effort to improve the internet infrastructure for profit.

ptero · 2024-11-28T13:35:34 1732800934

This seems to be a huge PR blunder. As a single data point, I have to say that my first reaction is illogical.

I have two hetzner shared instances and I am royally pissed by the 20x reduction in traffic allowance. It is also irrelevant to me: over the last 12 months I never exceeded 1TB. My unhappiness on the traffic reduction is purely of a "what if I start using more" type. For which two rational answers is "well you can explore alternatives then" and "d'oh, your average is way under 100GB, it's not going over 1TB". But I still started looking at alternatives.

My feeling is that the reduction is aimed at a small group, but upsets a much larger set of customers who now will start looking for alternatives. Which indicates a typical marketing screwup. My 2c.

axelthegerman · 2024-11-28T13:43:53 1732801433

Fair I felt similar about it.

But then, I have looked at alternatives before they even offered their US locations (and also for a CA one) and couldn't find anything decent for even nearly the price.

So we can pay a little more and get a little less, or move somewhere else and either pay a lot more or get a lot less

christophilus · 2024-11-28T15:28:55 1732807735

OVH is comparable and has locations in North America.

SixThreeOne · 2024-11-28T19:31:23 1732822283

Very well put! I had the exact same reaction as you: even though I don't even use 1TB/month, I found myself looking at alternatives simply b/c of the cognitive jolt of seeing the 20x reduction. Sometimes I wonder if companies don't consider the psychological impact of changes enough. No, logically, this change means a meager price increase for me, but if it caused me (and at least one other) to think about competing offerings, then like you said, indicates a marketing screwup.

That's not to say they have to keep offering that much traffic if they're losing money (of course they don't), but the way you make changes matters.

danielovichdk · 2024-11-29T07:49:36 1732866576

While all the guesswork ? They explain clearly why this is happening. And it seems fair.

"With the new tariff structure, we want to make conditions for our customers around the world as fair as possible. To do that, we will calculate our prices based on local conditions in Europe, Singapore, and the USA. Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. The new prices will give our customers the best possible price for the resources they use."