I'm a bit surprised by the replies talking about serving millions of requests a second, or needing ultra-consistent packet latency. Perhaps most of the HN demo is working for companies that operate at that scale.
But the OP is talking about running dozens of servers, not hundreds or thousands. My own modest apps have held up to the traffic from being #1 on reddit and not fallen over, never been null routed or nerfed by DO, and served hundreds of millions of requests and hundreds of TBs of data. I have nothing but good things to say about DO.
I have no experience running apps at the "millions of requests a second" scale, and I'm willing to accept DO might be a bad choice at that level. But what percentage of apps will ever reach that scale?
>>> I'm a bit surprised by the replies talking about serving millions of requests a second, or needing ultra-consistent packet latency. Perhaps most of the HN demo is working for companies that operate at that scale.
There is a high concentration of serious professionals on HN running world class operations. There is also a large amount of startup and single man operations who would hate to write a $100 check.
The thing is, one can only be on one side. Then the other side has nothing valuable to say to you, all their advice is misplaced.
So, to answer your question. It's stupid to ask what apps will reach a large scale. You're either at large scale already or you are not. Obviously, the article is only intended for small scale operations (actually, the entire article is just a generic ad, digitalocean has a nice affiliate program).
I think this is called maturity. GCP finally knows who the target customer is, and they're building features for them. DO has really nice and simple UI, but try to spin 5,000 servers and you will see what a hot mess it is.
GCP shines at a very large scale. For us, what makes the difference, is the spending predictability and massive discounts coming from running preemptible servers or simply committing to certain usage.
GCP is not as service rich as AWS, but most of the offered services are capable of withstanding incredible loads. In our case, while running on tens of thousands of mostly preemptible servers[0], we couldn't pull this off anywhere else (trust me, we've tried).
It's great to know which service works for your loads and make the best of it.
+1 you said what I was going to say. GCP probably doesn’t want the business of small web sites and portals. I think that under Diane Green they have a clear vision of who their target (large/huge) customer base should be and what they need to do to attract those customers.
For small projects, especially when I am paying the bills, it is difficult to beat OVH and DigitalOcean on costs.
That said, GCP is still my favorite provider, largely because using GCP makes me a little nostalgic for the time I worked as a contractor at Google - really enjoyed their infrastructure.
I hear you. Miss most of the tools and infrastructure every day since I left. I was ecstatic when they made PubSub available to everyone. Wish they would expose more of the internal services, especially Colossus, Dremel, Borg, ...
Yes, those are using the technologies underneath the hood, but are not them itself. I would like to have access to them directly as I would like to build other things at top of them
not really. BigQuery is for all intents and purposes an instance of Dremel running for cloud users. And GKE is effectively a clone of borg, rather than an instance of it- which is great. It sounds like what you want is a job at Google...
+1 The real money in the cloud business is in the Fortune 1000, not the small fry. 5 years ago it was only startups and Netflix using the cloud. Now the battle is on between AWS, Azure and Google to be the “new data center”.
There's time you need more flexibility. I'm serving 1.2MM requests per second from 3 GCP regions, managing instances and GKE clusters with terraform, and I cannot see how could I possibly set that up in a resilient fashion with DigitalOcean.
I think DO is perfect for certain scale apps. You usually care about UI things mostly when you spin up couple servers; but when you operate hundreds of machines you need automation.
GCP has it's quirks, e.g.
* 130k connections/core limit due to conntrack,
* lower networking throughput compared to AWS (16Gbps on GCP vs 25Gbps on AWS),
* no support for enhanced networking (haven't tested recent Andromeda 2.1 yet, though)
* no way to attach more than 8 local SSDs (arguably a good thing)
* etc
So does AWS, so does DO and you have to pick what's best for your project. One thing I like in general here is competition that makes all of those services better.
>"GCP has it's quirks, e.g. * 130k connections/core limit due to conntrack, * lower networking throughput compared to AWS (16Gbps on GCP vs 25Gbps on AWS),"
Can add to that, I worked for a few months in a small mobile ad network and the throughput of that was insane for the size of the company. Also a shitshow of a company on many levels.
Ads are a super intensive workload with super low margin. It'd be more correct to say that google built a very efficient platform that can run anything. Well, anything easier than ad, so almost anything.
You do get additional 2-3TB per instance which alone is a couple hundred $$ in bandwidth savings. Will it cover the price of these extra instances? Obviously if you need 8 SSDs DO isn't for you. But the real question is - could GCP be more lean back and still provide the same level of service? I think so.
Anyone who count in a unit less than kilo dollar is a waste of time for AWS/Google. A big business doesn't care about "hundreds of dollars" of savings, it's like cents.
It's interesting that 130k connections/core is limiting to you. Care to share some more details? Are those websocket connections? If yes what do you use as your backend? Something custom?
Those mostly are slow user connections. Most of the direct user connections to our backends performed using slow mobile connectivity so request taking 4 seconds considered normal and that exhausts connection limit.
Ended up serving most of the traffic from n1-standard-16 or lower.
Personally, not professionally, I like GCP more. Feedback cycle is (used to be) faster and it feels (used to feel) less enterprisey. I also liked the idea of a new player on the market, and wanted to make sure I vote for this diversity with dollars.
I think these days GCP and AWS are more or less on par. One thing I learned the hard way: invest time into calculating your expected cloud spend. Your use case, partners you integrate with, your audience. Those impact significantly on cloud pricing.
Yep, pretty regularly I see "Disclaimer: I work for Google Cloud" on HN, with AWS and Azure I see a lot less, no solid metrics here. I also like I can get staff on a Google Groups for many of their products like StackDriver to share feedback and get a PM to respond eventually.
I'm sure there is some internal rule (contract/NDS) at AWS that you are not allowed to talk in any way outside of Amazon about AWS without supervision ;)
Even if you're only operating at 1x $5/month droplet (their lowest instance type), you'll still get top grade email support included with your plan.
Getting a response in a few hours is almost guaranteed and often times they'll even help you debug problems that go beyond the scope of their infrastructure.
Compare that to something like AWS which charge you an extra $29/month minimum (or 3% of your AWS bill) to speak to an entry level support person through email.
If Google’s own premium speedy network, anycast IP, global load balancing and security features bring no benefit to your web operations, then your math might be right.
If you need your LB to handle 10s of millions of HTTP/S requests per minute, then what other choices do you have other than Google?
F/W default “Everything within a project should allowed”? Sorry, but no way.
Not familiar with other DO capabilities, but cloud platform is not just about a VPS available throu REST API, it is way far beyond.
>Not familiar with other DO capabilities, but cloud platform is not just about a VPS available throu REST API
DO is slowly transforming into a "light" cloud service. They have load balancers, a cloud firewall that's partially VPC like, object storage, block storage, etc.
It is still far from being AWS, but they seem to be headed, at least partially...that way.
I think there's plenty of room for a budget cloud provider that doesn't kill you with network egress charges.
> If you need your LB to handle 10s of millions of HTTP/S requests per minute, then what other choices do you have other than Google?
The same options every business that needed them has done for decades? Renting dedicated servers én masse, colocating, or running own datacenters? And doing this in multiple countries?
What I'd suggest for a start is renting dedicated servers or colocating in a few places (EU-Central, US east, US west), and running on that.
Dedicated servers, even with having to maintain them, are always cheaper than GCP. Even more so is the network traffic.
I've started with dedicated servers with a few European hosters myself where I'd be able to get servers and traffic for 40€ a month that would have cost me upwards of 14'000€ a month with GCP, AWS or Azure (180TB+ traffic per month, I don't have the exact numbers again, but I used each service's cost calculator for this). Sure, if I want high quality networking, I'd have to pay another 360€ per month extra with these hosters, but that's still cheap enough that it means I can basically run a 180TB/month serving service on a hobbyists budget.
Dedicated servers == constanct cost.
Many cases, you reach peak times for several hours per week/day. With GCP scale up and super fast, and when done, tearing down and saving money.
Also, from my experience, when you reach a certain level of operations, you discuss with Google sales team and you get a significant discount for bandwidth.
We're well past that point (with AWS) and we get a meaningful discount, but it's in no way competitive on a per-TB or per-Gbps basis with a dedicated colo.
It's also higher quality of service IMO. When AWS gets their daily DDoS traffic, they manage it and we never notice nor care. With smaller colo providers, they are much cheaper, but if your neighbor is getting DDoS'd, you notice it. I assume Azure and GCP are similar.
With regards to DDoS: that's something where OVH has surprised me well in the past, I've been a target once or twice, but they automatically handled all the upstream issues, and thanks to their hardware firewalls, I could filter the traffic before it hit my server.
On Twitter, the OVH CEO claimed that they've previously had servers targeted with 1.1Tbps DDoS, and still managed to keep that user's site working.
I've used Online (the company that also ownes scaleway, but their business line), Hetzner and OVH before.
Hetzner and OVH have much better peering than online (ofc still worse than GCP), especially noticeable when users are in Asia Pacific and you have only servers in the west.
Online is significantly cheaper, but well, the peering issue.
Generally I prefer Hetzner for compute-heavy workloads (especially their ML server offer with an i7-6700 and GTX1080 for 99€ is awesome) or networking, for most of the rest, I'd just go with Online.
Important for me is that I can just rent a handful servers, try out building something, fail, and try again, always knowing that it won't cost the world, and allowing me to experiment. If it fails, I can just try something else, if it works, I can just scale up. Doesn't have to be perfect, just has to be enough to let me fail early and often.
>What has gone wrong with GCP? It was trying to be AWS. Too complex and feature-rich instead of simple and feature-complete.
Yeah no. GCP is a direct competitor to AWS, so of course Google needs (at least partly) to match Amazon's product lineup. The argument of this post boils down to "GCP should be like DO", i.e. less services and a slicker UI, but Google won't win over AWS customers with that.
Indeed. Further to that, there are already a lot of DO-like hosting platforms out there but fewer AWS-like solutions targeting enterprise. So why would Google target a crowded market when they have the resources to target enterprise.
Also to offer my own anecdotal experiences using DO. I find it highly inflexible when I need to do anything more granular. Sure it's easy for deploying a standard build, but anything that falls slightly outside of DO's remit quickly becomes more trouble than it's worth. Where as AWS (and similar) might intimidate new comers with it's scale and complexity but that flexibility can also be a godsend.
I wasn't focusing on product lineup. GCP is clearly superior to DO and AWS even more. I was just offering an alternative to someone primarily using compute. It seems AWS LightSail understood this too: https://amazonlightsail.com/
Good luck with DO, the last time I checked they still shut down your instances as you hit sustained 300Mbps incoming traffic [0]. Could never work with a service that weak on the policy side. Also remember the disaster with surge [1].
AWS, Azure and GCP instances are completely different to Linode, DO, Vultr, etc... With the first three you get dependable performance characteristics with minimal variation (excluding t* EC2), this is the exact thing you need to do reliable horizontal scaling. Linode and Vultr and DigitalOcean etc.. are great for the price point but if I'm CPU bound and about to scale the identical service to N boxes then it's a poor choice.
As someone pointed out you have Lightsail if you want Linode/DO from AWS. 3.2.1... lets have cliche AWS/GCP/Azure is ridiculously expensive based on some undefined VPS memory/cpu details, hey virtual core!, and let's not forget the usual HE.net / Cogent / Level3 quote of 10Gig for $2000 and work out real bandwidth from there.. :rolleyes:
You're getting 2 (or X) virtual cores - which means absolutely nothing and can vary from minute to mintue / hour to hour - Linode was even giving 8 virtual cores with the cheapest plan 2/3 years ago. Virtual cores on Linode/DO/Vultr are shared/contended, basically undefined in terms of lower bound performance - you are at the mercy of other tenants - and there's a lot more tenants in the building too (where number of tenants is exponentially more than physical cores), so I maintain if your CPU bound and horizontal scaling these are poor choices. The cost of provisioning high performance networking is very real, Linode/DO/Vultr don't expect 98% of their customers to use anything close to 1TB and this nicely fit with the idea that they are poor choice for horizontal scaling - hence why AWS Lightsail isn't a threat to main cloud offerings. If you're just getting start and no where near the limit of single node then fine, I even use Linode for a few clients myself "Bandwidth:
39GB Used, 13961GB Remaining, 14000GB Quota ".
Disclosure: PM of High-CPU Droplets @ DigitalOcean
I'd highly recommend people check out the performance of the High-CPU Droplets for any compute-bound tasks like CI/CD, batch processing or active front-end web servers.
Someone from our team put together a comparison of compute series machines which includes GCE and you can see the price-to-performance ratio here is pretty solid for DigitalOcean: https://imgur.com/a/o16t2
Interesting, I missed this announcement. This is definitely a step in the right direction to competing for CPU bound workloads. "Achieve up to 4x more CPU performance on High CPU vs Standard Droplets." I guess we can take this to mean that standard droplets virtual CPUs are contended at 4:1 - I would of thought it be higher.
You should also check out Vultr's dedicated core plans [1]. I have ~50 instances running on Vultr and CPU performance / unixbench scores are significantly better than DO and Lightsail's similar sized instances.
We were also impressed with DO's GUI and easy of use. We setup a few monitor servers on their network to measure uptime, latency, packet loss and similar metrics since we deal with audio streaming over UDP. Suffice it to say, it was a mess. So many micro-timeouts and tons of packet loss in small bursts.
For prototypes and small projects dealing mainly in TCP and if uptime is not really important then DO is great. For us, we stick to AWS / GCP for the bulk of our load including dev/testing etc as we want to have the environment as similar to prod as possible.
It's not private. It's internal and shared with all other customers at the same data center location. I wouldn't be surprise if this misleading marketing resulted in lots of accidentally exposes services.
Huh. I have zero experience with using cloud services, but I'm a bit surprised by the complexity of both of those UIs. I always assumed it was way simpler than that. Like, I thought it was as simple as using a command-line tool, like "docker run <myprogram>" but "cloud run.." As there any cloud services like that? Where you just specify your environment and CPU features you need etc from a single console command?
You could spin up a server with just one AWS CLI command (and most likely google console, but I have no experience with it). As you said, just specify the server spec in the command option. Though, AWS CLI is very verbose with their commands, imo.
Anything more complex, such as weaving several server under one VPC with a load balancer and routing configuration, I'd recommend checking out terraform. It also supports multiple cloud providers.
What you are probably looking for is a PaaS (App Engine, Heroku, Zeit Now, etc) not IaaS
On GCP you can run:
gcloud app deploy
And it will deploy your app on App Engine, with auto-scaling built in. The thing is, that is just the start. Pretty soon you will need databases, caching, versions, etc, etc.
The high bandwidth is because with cloud services like AWS and Google Cloud you'd be serving most traffic from either object storage (like S3) or their CDN. Both of which are in the same region as the DO bandwidth figure you quoted. This not only reduces your compute bandwidth costs but also drastically improves your application performance when under load (and as an added bonus you also reduce your compute CPU costs since they're not serving up static content).
I've found even some relatively simple infrastructures (2x web servers, 1x database server + some extra caching services), AWS actually worked out cheaper than Digital Ocean. The problem is you'd have to approach hosting on platforms like AWS slightly differently than you'd approach hosting on the likes of Digital Ocean. But once you start scaling your application, you'll soon find that you'll want to adopt the aforementioned topology anyway regardless of arguments about AWS/Google/etc vs Digital Ocean.
S3 is $0.023/GB (still x4 higher), plus $0.004 per 10K requests
CloudFront is $0.085/GB, plus $0.0075-$0.01 per 10K requests)
GCP is $0.08-0.12/GB (same for all services), plus $5 per 1M requests)
And who says most of the bandwidth goes towards static assets? 99% of our requests are dynamic (thus the NodeJS frontends)
> And who says most of the bandwidth goes towards static assets?
It obviously depends on the application but generally you'd expect that to be the case for web sites (ie not services that act only as an API backend).
> 99% of our requests are dynamic (thus the NodeJS frontends)
I don't know your business so you might be transferring more dynamic data rather than static but your argument here doesn't help explain your business as you're mixing metaphors. eg most of our requests are dynamic as well but JSON APIs will generally return smaller chunks of data compared to an image. So in real bandwidth terms on our application, more data is transferred for static content rather than dynamic. None of the static content touches our web servers though. In fact a fair amount of the dynamic content doesn't either since a lot of that isn't user specific so is served from caching services.
If you don't mind me asking, how much bandwidth you transfer each month? Even taking the above into account, our bandwidth costs (which is nothing compared to when we self-hosted) gets dwarfed by our computing costs (VMs and DB).
By the way, how are you finding node.js for your workload? I find it a weird choice if 99% of your requests are dynamic based on bottlenecks I've experienced when deploying test systems on it. But I ask about your experiences because I've not used node.js for anything at scale (ie just a few thousand requests a second).
I’m not the one you responded to, but, I wanted to share my data:
I’ve had months where I’ve transferred above 180TB of data, at an overall cost of below 40€ in that month, with dedicated servers at a European hoster.
But the OP is talking about running dozens of servers, not hundreds or thousands. My own modest apps have held up to the traffic from being #1 on reddit and not fallen over, never been null routed or nerfed by DO, and served hundreds of millions of requests and hundreds of TBs of data. I have nothing but good things to say about DO.
I have no experience running apps at the "millions of requests a second" scale, and I'm willing to accept DO might be a bad choice at that level. But what percentage of apps will ever reach that scale?