It's not much better if you end up being a GCP customer.
My company has gone through 4 reps in 3 years. Every time we get a new GCP rep they just want to talk to us about "expanding our use of GCP offerings". The only thing they want to talk about is starting to use BigQuery - not my business at all.
I signed up for a Google Cloud Security summit, and afterwards a sales rep reached out me. It was obvious from the start they had no idea I was a gsuite or GCP customer. They then directed me to a NEW account manager (#4) even though I had been working with a different one. I had worked with the prior account rep going over our architecture to make sure everything was kosher (sustained use discounts etc). I even made them a schematic of our architecture on GCP at their request. Once I provided that to them I was met with radio silence.
It's really insulting, and to me obvious they don't care about my company at all. We're looking at other options.
I wouldn't be surprised if somehow this post leads to me getting an email from another rep "Wanting to start over and doing things right", which will inevitably devolve into the discussion of how can I use BigQuery.
I've had the same experience with GCP account reps. They always go missing and someone new emails us about how they are taking over 6 months later. Every call we have had with them has not resulted in anything meaningful. Our biggest issue is how their "highly available" Cloud SQL goes down every couple months for maintenance, not how we can use BigQuery.
> "highly available" Cloud SQL goes down every couple months for maintenance
Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance.
If anyone from Cloud SQL is reading this, please, please stop doing this.
We’re currently migrating to Spanner for a variety of reasons - but the mandatory downtime on their Postgres CloudSQL offering will be the part I miss the least.
It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!
Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?
> Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?
I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.
> that'll be really really hard and take a long time
It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives).
Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.
Sounds familiar.
Product and sales areas of the business look at immediate revenues - and hopefully profit - but are always more keen to do that at the cost of increasing technical debt within the engineering side of the business. Unless sales see a real impact to technical debt, they will always choose the short-term approach.
Agreed, and I don't think that's even necessarily the worst thing. It just means that engineering and product/sales have to have a conversation, mutual trust, and a shared vision that extends beyond the next quarter. These are hard things to cultivate, however.
I agree. In whichever case, blaming individual engineers seems weird. Rarely are individual engineers to blame, and even when they are the organization should be robust enough to route around the occasional deficient engineer.
It's a full-company culture problem where ~everyone has personal responsibility.
Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc.
It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.
I tried explaining this to some Azure product teams, and they gave me a blank stare in return.
Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service!
You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps.
It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform.
It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.
My experience with MSFT is that the account reps are clueless by design. Even the sales assistants with fancy technical titles are short of clues. Easier for them to oversell when they don’t know what’s really going on underneath. We have to go 3 or 4 deep into their product org to find someone who will fess up to product issues that we are already aware of.
Also they now require you to become Microsoft Partner, if you want to use the oauth2 login not only for personal Microsoft accounts but also other Microsoft accounts such as those provided by a business running Office 365. They changed it ~ 3 months ago and are now not able to verify people in time. Even if they just want to check your name, email and maybe a profile picture and nothing else. The process is much less obvious than for the same thing with Google or Facebook. Even the ZIP-Code has to include spaces as if it was written on a letter, otherwise you will not be able to send the form, aaaand there is no hint about this requirement.
In general, Microsoft, Google and others are behaving really enterprise-y in that they are slow, you cannot reach anybody of importance to solve massive issues with their products and everything is actually quite expensive for what it does.
I use Aurora just for this reason. RDS is just a great product. Doesn't even need the cloud sql proxy equivalent since you can just install the authenticator into the database.
NO, RDS includes non-Aurora versions of many DBs, plus MySQL and Postgres-compatible Aurora servers, plus MySQL and Postgres compatible Aurora Serverless.
Thanks for clarifying. I'd associated RDS with the non-Aurora offerings of Mysql and Postgres. Judging by upstream version compatibility it appears Aurora is more heavily forked than their non-Aurora siblings.
I don't think Aurora MySQL/Postgres are just forks, I think they are a completely custom datastore behind a MySQL or Postgres-compatible interface (which probably uses a lot of non-engine code from the open-source base database.)
Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility. Seems like only one major version bump has happened since they launched the first one (Mysql 5.6-to-5.7).
Which is fine. It can just be a little confusing as they drift and the caveats grow.
> Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility.
Current Aurora Postgres is compatible with pg 12.4; pg 10+ support has been around so long that several versions that support 10+ have already been EOL’d by Amazon. Even Serverless, which lags behind, is on 10.x.
WAIT. Be careful. That is a super expensive product with a high likelyhood of lockin. It doesn't support all SQL features. Also! I run hundreds of GCP databases and never ran into: "but the mandatory downtime on their Postgres CloudSQL", maybe it is only Postgres?
Same here. I have sent my GCP rep 5-10 follow up emails by now and still no response. It really feels like I have no one to talk to over there and I'm spending thousands per month.
That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.
I think you’ll find me surprising :) It’s surfacing and chasing down root cause of these misses that make things better for all customers, and that’s my goal :)
I helped a major user of GCP migrate off the platform to AWS for this exact reason. Totally insane that they still do this when AWS has had a rock solid offering in the form of RDS for like 10 years now.
Absolutely not. I'm sure it's happened but I have never personally seen an RDS instance go down, in over 10 years consulting and building in AWS. Google Cloud SQL was going down for multiple minutes every month for one of my clients, and their support just said there was nothing they could do about it. That cost Google cloud at least a million dollars in revenue over the next few years as this was a fast growing startup.
I’ve been a long time Azure user, their SQL Azure product was very glitchy and had constant outages (“just add retries!”), it’s gotten better in the last year or so, but it was totally unacceptable.
AFAIK that's a "feature", not a bug. They throttle concurrent requests and return a different error code compared to the on-premises SQL Server.
I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries.
Compared to just locking up or timing out, it's probably better.
E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!
Do y'all just like, ignore SLO's? Cloud SQL has a 99.95% SLO, going down for multiple minutes every month is within that. No smoke and mirrors here, there are ways to mitigate it but it's not Google messing around with expectations. HA doesn't mean 100% uptime.
Yet AWS manages to give near 100% uptime with RDS at a similar price point. CloudSQL downtime in the cases I've seen was caused by them rebooting both master instances at the same time. This is amateurish and totally unnecessary, the whole point of having multiple masters is to the ability to do maintenance reboots in a staggered schedule. This should be a trivial problem for Google to solve and would result in much better reliability for their users, yet it's been years with no change.
I had some instances that needed to be rebooted because they were running a vulnerable MySQL version. AWS gave six months warning (before forced reboot) and ended up extending it to 12.
When we first launched on GCP, there was no question that it was the way to go (frankly, because of BigQuery). Working with AWS, when we launched, was going to cost us significantly more up-front, before we had even brought on our first customer.
Fast forward 5 years... AWS has closed the gap in every way that matters. I still, frankly, trust Google's Security more than Amazon's, but I don't encourage folks to use GCP the way that I used to.
Just the opposite. In 2021, no questions asked, it's either AWS for general compute, or something more targeted if your business doesn't need it.
Until you get to the point where your bill is larger than a dozen engineering salaries, you won't get any respect from these people.
> I still, frankly, trust Google's Security more than Amazon's
If you have the time, could you expand on this? While I'm not directly involved in security at AWS, I'd be down to forward your thoughts to people who do.
I used to work at Google on Cloud, and am now an AWS customer. Have used both clouds extensively.
My comments are mostly backed up by my experience at startups and are not colored by my experience at Google (too different a beast).
GCP is great for teams that are also using GSuite because you can set permissions at the level of a Google Group and have them propagate to individual members. You can, of course, also create groups in AWS but they don't have the same semantics of Google Groups and don't cover the wide range of use cases that Google Groups does.
The AWS scopes -> policies -> roles -> resources chain of abstractions is less natural conceptually than GCP's GSuite accounts + service accounts with attached scopes per project.
Also the fact that each managed service (GCE, GKE, Cloud Builder) has its own service account that you can attach scopes to is really nice. GCP service accounts just feel more discoverable than AWS IAM roles - I think it's because the number of AWS pre-built roles is so overwhelming.
I think all of these replies so far capture my thinking. However, I think the simplicity of the GCP IAM model is what I will miss most going back to AWS.
I’m sure they exist, but over the last dozen or so years I’ve worked with public cloud offerings across 5 or 6 industries and domains, I haven’t found a use case that can’t be easily implemented in the simpler GCP model.
Gone deep on IAM in AWS, attempted a similar thing on GCP later for another project, was very surprised how weak GCP IAM was.
Top of mind things I found weak/weird:
- Basically can’t do least privilege, only want a role to read messages from one pubsub queue? Nope not possible
- IAM policies seem bolted on, legacy roles seem much better fit in the ecosystem, but they suck obvs.
- different gcp resources have somewhere between very coarse grain to just acceptable iam operations. Might just be GCS that lets you do proper least privileges policies like you would do in AWS.
I was very surprised how bad it was, AWS IAM is some black magic shit that is deeply impressive and often taken for granted, even GCP can’t replicate.
One thing that GCP is far better at is account setup. Having everything nested under a single gsuite organization with folders and projects and IAM flowing through is incredibly easy to work with and makes permissions simpler to understand. AWS has a long ways to go in this regard.
I came from an AWS background to my current company's GCP setup and was very confused at how IAM worked on GCP for a long time. Now that I know the system, though, I agree with you. It really makes a ton of sense and works really well.
The biggest problem I have with GCP is that something will say "you need the foo.bar.baz permission", and when I go to the IAM page to give that to myself... there is nothing in the search results for "foo", "bar", or "baz". Instead, I have to guess the "friendly name" for the permission.
I can totally relate. The amount of times I've spent scouring the docs for the "machine name" to put into TerraForm, or vice versa, to do through the UI...
I disagree. Once you learn IAM and able to segregate users into groups each with its own layer of security, then it is good enough.
Often the UI, and docs make it seem like everything is all over the place but AWS feels like lego with some pieces tucked away. That is where I think AWS can be improved upon with a better documentation UI and discoverability.
I do have to commend Google on Flutter + Firebase + Firebase Functions. I think if Amplify focused on serving Flutter users more it could pull me away from Google altogether.
Unfortunately, Google has done a fantastic job with making Flutter integrate with Firebase through Android Studio and there really is no product from AWS that matches its developer friendliness and low learning curve. This makes it very easy to switch.
I guess it is somewhat of a threat because the Firebase Cloud Functions also offer something of a counter to AWS Lambda as much as I love using it with API Gateway.
IAM shouldn't be a thing to learn. It's account management, default and easy to access options should be sane enough for most people to use. At big companies, sure someone has it as a dedicated part of their job description. But if you're in the majority of smaller companies, ones maybe that's just doing e-commerce and tech isn't their core skill set, account settings should be near invisible and still be trustworthy. It's not the Slacks of the world that have an issue with this, but the long tail of the world we now live in that software has eaten and companies are just scrambling to exist in it. Flutter integration is not in the list of concerns of this long tail.
And telling them to "just learn it" isn't the customer focused mindset, it's the engineering one.
I created all of the IAM roles that our 100+ person company uses. It was and is important from a security standpoint that we do not just blindly give too many permissions to employees. I had to do some research to understand what the bare minimum was and it wasn't too difficult to do.
Custom roles created through Terraform helped a lot.
It absolutely is required beyond small 3 person startups. Google is great to get started but when you are dealing with a dozen or more developers, especially at large organizations IAM offers that granular control and overview.
Yes its a bit of a pain having to add policies sometimes when you are first getting started but once it's up and running you can rest easy.
Learning it isn't that much more time consuming or difficult, its just a bit of effort that is all (we are talking a few hours at most).
What exactly are you disagreeing with? Segregating users into groups is possible with either platform.
The discussion is about GCP having much better functionality around it where projects and permissions are naturally integrated with gsuite organizations and users. This is objectively true. AWS has an archaic project system with a dozen different attempts at uniting it all but nothing comes close to GCP's smooth and easy manageability.
Thoughtful features like supporting UEFI Secure Boot with vTPM attestation. This allows building setups where even a full GCP account compromise can be mitigated.
Integration with our org GSuite (this alone is a massive plus).
> I still, frankly, trust Google's Security more than Amazon's, but I don't encourage folks to use GCP the way that I used to.
I trust virtually anyone's security over Google's. I've never had issues with AWS. I've consistently run into serious Google security failures. Google has airtight security for its own data, but not for its customers.
Examples range from Chromebook and Android security update policies (tons of expired machines on the public internet, in the case of Android, usually without people knowing), to pay-for-security on GSuite, to really difficult-to-audit Google Drive security (there's no convenient way to track and audit what was shared with whom or where data went), to just a ton of other things.
I've never seen Amazon be callous with my data. I've seen Google do things that even nineties "we don't need security" Microsoft wouldn't have imagined....
The only people I know who really trust Google security worked for or are close to people who worked at Google. There's a reality-distortion field based on how much Google invests in its own security that people fail to notice very basic failures, like millions of expired Android devices, or a lack of audit logs if someone physically accesses your machine to rifle through your gmail....
I had a similar experience with their recruiting. Some interviews were cancelled and had to be rescheduled only after I inquired.
Many weeks to generate offer and paper work and then 24 hrs to accept with a "take or leave it" ultimatum.
Just a very poor taste and I know some fine people who work at Google. Its unfortunate they don't seem to value warmth and humaneness when communicating externally
To be fair, Covid likely played a part in that long silence. I believe there was even a hiring freeze for a bit in spring. Not that it excuses bad behavior...
If the recruiter replied and said as much I would have been ok with that. At least they were communicating. Plus covid didn’t effect the US big time until March. That’s all of Jan + Feb they had to let me know something was going on.
They called me eventually were kind on the call. But the whole communication issues leading up to that really ruined it for me. I was initially quite excited too. I have no idea if this is common or if my recruiter was especially bad. Reading the parent comment made me think of the similarity.
Wow, that's a pretty horrible experience. Weird too since AFAIK recruiter/recruiting coordinatiors comp tied in some way to shepherding the successful candidates through the hiring process.
Me and a friend of mine interviewed in Nov and passed HC in Dec 2019 as well (different recruiters between the two of us). Our experience was different from yours: Immediate timely followups from the recruiting coordinators, team-matched and hired in the same month. Hopefully our experience is more common...
I know why. HR received so many resumes which they don't do any additional effort, do they receive compensation for each hire, no. Most of them are not FTE.
Sigh, I remember a similar experience. It was a third-party rep they pushed us to, but we would be asking for ways to re-architect one thing we already had setup on AWS, and all they would do is just try and upsell us on random offerings that clearly did not resolve our specific needs.
Some European-based customer apparently had a requirement if we engaged with them, that our service be offered via an acceptable vendor such as GCP, for some reason AWS apparently wasn't, but it was such a nightmare to even prod about an architecture that would have feature-parity with AWS, it wasn't even worth it. Also as an fyi, I'm no AWS fanboy, I don't use it in any of my own projects to avoid vendor lock-in this company suffered from.
Same terrible experience here with GCP sales and support, but the other options aren't much better. The reality is that unless you are in the 7 figure range, you don't get serious attention. I'm still surprised why sales is so dysfunctional but billions of quarterly profit means there's little need to change.
Since at least when we started spending about $100k/yr we've had a dedicated account rep we can contact at any time. They also get in touch to schedule a check-in every few months.
They've been genuinely helpful in several situations and have scheduled meetings with various teams around AWS (like, actual engineers) to get us answers to questions, support, and guidance. We've been put in touch with team leads and engineers working on beta features when we tried to use them and had issues to report.
Obviously this is all a sales tactic: if we have questions about X, putting us in touch with experts in X makes it more likely we'll successfully implement it and then pay them to use it. But it's the kind of sales we're getting value from, not just blindly pushing us to pay them more money.
This is for $100/MONTH!! That is the deal of the century.
And they are ridiculously helpful.
I don't understand this - AWS must be losing money at least on support side, though they obviously get happy customers (myself included).
And even at $150 this would be great.
I had a client on gsuite with google 8 years ago - we COULD NOT get anyone to help with some weird admin state flow issue - it just was not possible to talk to a human being for ANY amount of money.
I suspect that AWS is 'losing money' on this in the same way that Apple are 'losing money' on their high Street retail shops.
Working at a place with AWS enterprise support by contrast, for the second occasion, I would suggest that many of the places paying $15kpm don't cost that to support.
Yeah. AWS is probably "losing" tens of dollars per month hosting my personal account. They've made a few million dollars in sales as a result. I've personally started several projects on top of AWS which spend that much now. That started with the free tier back when AWS was young.
Google has treated me so badly so many times now on my personal account (as well as on business accounts, for that matter) in so many different ways that they've, conversely, lost MANY million dollars in business sales. It's hard to even count; a lot of people ask me for advise on decisions, and whenever someone even thinks about using Google Cloud in a business setting....
This is not a hole I see Google getting out of, except by eventually shutting down the Google cloud. Too many people have had too many bad experiences, and reputations take a long time to recover.
And the failures just keep on piling up.
Google is great for personal use, but I think they're diversifying in all the wrong directions. They're not structured for success there.
Google has been relying on it for years. It's completely seamless and means you get even better reliability by insulating from the underlying hardware issues.
Looking at support as a cost kind of misses the point. Bugs are bugs, and your big-ticket customers might just drop you rather than helping you figure out what is wrong with your product. Ignoring paying customers (even small-time ones) prevents you from improving your product. If they took the time to reach out (and especially if they're paying $100/month for support) you really need to listen to them and try and figure out how you can fix the issue so it won't affect big-ticket customers.
The price starts to go up steeply once you hit the % of monthly spend.
I'd guess they don't get many resource intensive support queries from the < $10k a month customers (and at that level you probably don't get the A team support)
Another comment later identified this - I actually did get a weirdly A-team support response. That's what made me scratch my head because I was coming in the poor / cheap / don't know what they are doing door (and most support is crap user misconfig issues at least from my experience). I just wanted the thing noted somewhere in case others were hitting it, instead I got a way above standard support specific technical response and some suggestions.
The other poster indicated it is possible for stuff like this that you get bumped, even on the el cheapo plan, to someone actually on product team. While I'd hate to be on the product team having to answer these, it perhaps keeps folks aware of customer issues so they can at least improve docs for corner cases?
That's what it says, but in practice I've asked some really general and technical questions of AWS support and always received a helpful reply without a paid support plan as well. With a paid plan the response time is better.
In general the AWS support has been great. In many cases, they've forwarded our requests to product teams who have even fixed bugs we've run into and contacted us directly.
Our other experience is with paid Azure support, which did little else than direct us to the (not related to the question) docs. They also had a really hard time understanding our technical questions about specific APIs. To their credit, they did eventually escalate to the PM of the service in question.
In general, the team responsible for the service really must be able to help out with support requests. In AWS this is definitely the case, in Azure as well but there's a bit of gatekeeping. Does developers and PMs in GCP participate in support?
Microsoft support is always useless in that way and the TAMs are pretty powerless. I hired an intern just to contest the hours to effectively cut our (large) premier bill 70-80%.
Their model was fault-based, and a “bug” gets billed to the support group. So the game was always for MS to avoid assignment for non Sev-A cases, and our game was to find a product defect for anything.
Or support helpfully suggests that you file a suggestion in the public Azure feedback forums.
Yeah... just like the other 5 dupes of the same suggestion with hundreds of upvotes that the product teams have dutifully ignored because it only matters to customers.
Google Cloud, AWS, MS Auzra are made to lock you into their system. I guess if fine if you are ok with it and you don't have an experienced systemadmin/dev op on hand.
There are plenty of cloud agnostic platform out there that does VPS, load balancers. digital ocean, linode, etc.
If you want to be green there is also the Advania they use geothermal energy. And since it is in iceland better data protection policy than US companies.
I'm not affiliated with any one of them, it is just from year of cloud provider hopping.
Couldn't agree more. I am on both DO and Linode for many years and I have had an amazing experience. Not affiliated with any of them, just a happy customer.
Have you tried deleting an S3 bucket? Go ahead, I'll wait. How about egressing any amount of data? I hope you have a Diamond Platinum AmEx attached to your account because you're about to lose your house with that bill.
Route 53, S3 storage, load balancer, everything is encouraged to work other AWS services! Smells much like what MS was doing. Don't get me wrong, it is more open than Google Cloud and Azura.
Somehow your comment reminded me how Google talks go at GDC and similar game development conferences.
While everyone else is talking about game engines, design and programming techniques, Google's talks are mostly around their cloud offerings, and customer telemetry.
The reason why the account managers change this way is because of so called territory or account realignment. These are sales terms but ultimately it means don't sit on one client for too long. Salesforce does this every single year, last week I received an email from 6th or 7th account manager since I started using them. The story is always the same: they get in touch,try to understand your business,ask questions,and see if they can spot something that previous guy missed. I hate it. However, this seems to be working on their end,as the sales keep growing...go figure.
So why is Google this way? They already have a terrible reputation for their non-existent support. Never thought they'd treat actual companies this way though. It makes no sense...
Inevitable. The entire company is built on a scalable business that required no direct contact with customers. Direct person to person interaction is the antithesis of their successful business model.
Of course, now when they enter a business that needs direct customer interaction, it shouldn't be surprising that they aren't good at it. No supportive processes and probably no internal appreciation for what reps do. Thus the turnover. They will have to build out a sub-org that nurtures and develops this part of the business.
I personally find customer interactions problems to be epidemic with fast-scaling internet businesses. It does however create a space for others to come in and do it better. We can hope...
Even their customer facing engineers who aren’t total a holes act like they are from a superior type of existence and we should be honored to be graced with their specialness without meaning to, but it is ground into them.
This seems to summarize my experience with software sales processes and interactions in general. I’m hopeful that software sales culture is beginning to evolve into something other than the pack of hyenas on a carcass that it is today. I personally have resorted to rejecting any and all conversations with software sales people unless I can dictate the direction of the conversation.
To be fair, BigQuery is probably their best product since Search so I can see why they try to milk it. It capitalises on what Google's best at, distributed systems engineering, while avoiding what Google's worst at, API design (by instead using SQL, which wasn't invented by Google). I say that as a customer who otherwise has little positive to say about Google.
As the guy who designed the architecture diagramming system, mostly to help folks keep straight what they're trying to help folks build, I hate that it's being used as a qualification/filter/etc. Sorry yo.
It's interesting that you've had that experience - it mirrors mine in dealing with Oracle reps: no engagement, no interest, very high turnover, always under pressure to sell, sell, sell. I wonder if appointing someone ex-Oracle to head GCP has carried that culture over.
You need to see GCP as its own brand with its own support team and policies (though they are impacted by some shared technical infrastructure and associated policies). I cannot say whether these are good or bad, only that they are distinct for GCP.
Experiences with support or lack of support concerning other Google product areas and divisions, especially those not designed for businesses really don't apply if you know how Google operates.
I don't use GCP in a personal capacity at this time and I work for a competitor (though am certainly not speaking on behalf of the competitor).
I use a lot of Google products. All of them I pay for, except search.
It is true there are individuals that shine, but across the board, Google sucks at support. It starts at the top with what kind of company they want to be, and goes from there.
Once upon a time I was one of 3 people globally that was the entirety of the GCP support team :)
You should have seen the extent to which the rest of the team and I tried to resolve customer issues.
Random App Engine app getting 500s calling an Open ID endpoint (the service wasn't strictly speaking managed by GCP). I absolutely could not reproduce this and had no other reports, but I knew it was happening for the customer. I examined everything that could be different... After 2 weeks I finally got it:
The customer app was running in a different data center compared to where I was testing. Calls to the API endpoint were timing out internally (pretty aggressive timeout interval). It turns out the Open ID service wasn't running in the same data center region. That should not have been the case but occurred because the internal BigTable service had to be upgraded in a particular region and hence all App Engine apps in that region were moved to another one - a fairly seamless process. The Open ID service unfortunately wasn't provisioned in that region (it's not a dependency of App Engine).
Once I identified the issue it was a matter of telling the on call SRE about it who then spun up Open ID in that region and thereby resolved the issues in a matter of 5 minutes.
Often a new support case meant discovering platform limitations - for example running into BigTable anti patterns because App Engine Datastore allocated IDs sequentially by default. That could cause poor read / write performance, in part due to automatic sharding behavior of BigTable. We later switched to snowflake IDs.
And then there were lots of Java folks used to Classpath scanning (default behavior of Spring Framework?) which for larger projects caused App Engine container startup to exceed the allowed time limit. That became a user education issue for which we then had to write an in depth article.
Good times :)
I absolutely hated dealing with billing inquiries though... so difficult to map someone's code to billable operations. Some SDK methods were of course utilizing multiple billable operations. Determining SLA violation refunds was much easier.
One thing that AWS Premium Support does very well is make situations like this seem unremarkable. This is the default way you do work. Going above and beyond is solving the issue, the next issue the customer hasn't asked you about, and giving practical guidance on things like cost optimization or availability concerns after weeks of deep diving.
I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.
What I am saying is that this obsession was normal to us also. I didn't know any Googlers (whether in support or engineering) who didn't care about customers.
We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in.
I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :)
These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now.
And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer)
Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.
I really appreciate that. Both as a customer, but also as a fellow practitioner.
At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer.
As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me.
As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that.
I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for.
I don't hold this against the support folks at Google.
I hold this against the leadership at Google.
This is their strategy to contain costs or to juice sales.
If it were working, I think their numbers would be very different in the market. So the proof is in the pudding.
Something isn't working at GCP.
I don't think it's the engineers or the teams. It's the leadership and the strategy.
What you are describing unfortunately is due to the external vendor teams managing support -- a business decision as you rightly mentioned.
A lot of support became outsourced and the original support team became more of a tier 3 backline support with the first 2 tiers being handled by vendors. This started for GCP just after I left in mid 2013.
The vendors responsible for support have contracts to meet certain customer satisfaction metrics and time to first response metrics (based on customer tier and ticket priority). The majority of inbound support cases would get auto assigned to the vendor team. Only the top customers would reach Google employees directly.
The vendor and their staff simply have no incentive to troubleshoot an unusual issue - they also cannot have access to the tools that may expedite such troubleshooting. It's all about pattern recognition to quickly resolve a ticket. As many vendors were overeager to escalate support cases to the Googlers it became necessary to impose more requirements on the vendor team to be able to escalate an issue - invalid escalations were being tracked and the vendor would be penalized for the total of these.
I built the internal tool via which GSuite and GCP outages or significant issues would be managed from a support perspective. This had the advantage that vendors didn't need to waste their time (poorly) parroting (with a delay) the status of an outage as related incidents could be (manually) grouped together and then all be updated at once by the current incident manager on the support team - freeing up lots of support folks to work on other support cases.
I definitely agree this vendor-based support model is a frustrating experience for everyone involved. I witnessed that myself after leaving Google. By the way, I'm very surprised GCP tries to upsell you on consulting services based on your support interactions.
Running a support organization for a very technical product with so many unknowns and variables at scale is hard. I wonder who does that well and how they accomplish it (from an operational perspective).
I have no idea how AWS manages support internally. I work at Azure - until last week as a Developer Advocate, now as a software engineer in Azure CTO incubations - I have no idea how our support organization is run or what their quality is. However, from what I hear Azure support is eager to help anyone and everyone. I remember being very confused I got a phone call to inquire whether I needed help within a day or two of signing up for an Azure free trial.
Is paying extra 3-10% of your already expensive invoice worth the premium support that you _might_ need? Not sure I would consider that customer obsession... more like insurance.
Honestly given how good BigQuery is, I don't blame the sales rep for keeping their eye on their wallet. I know a lot of companies who are primarily AWS but use Google Cloud exclusively for BigQuery analysis.
(no, I'm not and have never been a Google employee)
I've used both GCP and AWS at multiple companies and personally, and my take is that a handful of important GCP products are significantly better than their AWS counterparts (Bigquery, Bigtable, Spanner, GKE), and a lot are just OK, usually slightly behind AWS in terms of features. If one of those products that's significantly better could be a differentiator for you, GCP is the better choice.
I'm not a rep, and I don't know anything about your use, but I'm 100% ready to learn about it and see if any of the stuff I know can be helpful to you and your team. Sorry this has been your ride to-date :( miles@sada.com
I'd say your experience isn't unique to google. I think at some point we are going to hit a place where people realize the convenience of the cloud doesn't outweigh the additional cost of having a mostly predatory business partner.
the convenience of the cloud doesn't outweigh the additional cost of having a mostly predatory business partner
Back in the dotcom days companies would spend a fortune on Sun kit but I bet when averaged out over time a comparable company would be spending a LOT more on cloud billing.
> Back in the dotcom days companies would spend a fortune on Sun kit but I bet when averaged out over time a comparable company would be spending a LOT more on cloud billing.
I would like to learn more about this. I'd have thought costs should go down over time. Are we doing more or is the cost per unit (not sure what that means) is truly going up?
I would like to learn more about this. I'd have thought costs should go down over time. Are we doing more or is the cost per unit (not sure what that means) is truly going up?
I think a number of factors add up over a 3-5 year timeline (obviously this will be more or less true for different organisations). There is the way the cost scales for a given instance in the cloud - in the old days, for example, doubling the memory or doubling the CPUs didn't double the cost of the kit, but it does for clouds VMs.
Another example is that the cloud bills you for everything, in the old days I could have a database server on a network and query it as much as I liked, the cost was fixed upfront for the lifetime of the hardware. Whereas it's very cheap to get started with a managed offering but e.g. BigQuery charges you for every query, Cloud Functions charge you per invocation, bandwidth is chargeable etc.
Speaking of hardware, in the old days you could look at your hardware and say, actually, it's fine, we don't need to upgrade/replace it this year after all, and it will just keep running. Whereas in the cloud the payment is continuous (perhaps offset by the fact that it's easier to "give back" excess capacity).
There will come a point at which the cost of DIY vs cloud will cross over, the question is whether you will reach that point, and if so, what you will do about it, since you may be well and truly locked in at that point.
When was the last time a landlord reduced your rent?
You always can drive cost concessions from sales, especially for base workloads where you have time flexibility.
For a big company, cloud rarely saves money for many categories of expense. In a normal market, it is almost always faster time to market to rent, and always cheaper TCO to own.
> When was the last time a landlord reduced your rent?
This is a different market and one which is ultimately constrained by the availability of land. Notably, cloud prices do fall especially relative to the compute power. Specifically I remember when Fargate moved to firecracker and prices fell by like 40% or something similarly considerable.
Maybe managing your own internal cloud is indeed cheaper (especially if you don't account for support or maintenance!), but arguing that cloud prices don't decrease or making some housing analogy seems like poor reasoning.
Maybe cars or trucks are a better example. The ROI of buying, leasing or renting a vehicle varies and the optimal answer depends on the scenario!
It’s always better for you as a person to rent box truck to move. If you’re a company that needs a truck 3-5 times a month, there’s a probability that leasing may make more sense.
I’d say that businesses that suck at managing on-prem will not magically get competent in a public cloud.
> I’d say that businesses that suck at managing on-prem will not magically get competent in a public cloud.
It takes time to build a competency. If you have to figure out how to operate everything from the hardware and networking all the way up to application, then you might very well fail as a business. If you can pay Amazon to manage your networking, hardware, databases, managed services, etc while you work on your application-level competencies, you stand a much better chance of succeeding ("I don't know if I could build everything from the ground up, but I can probably fumble my way to a container image"). As you become very proficient, you can cut costs, and if you're super proficient you might get off a public cloud altogether or more likely you'll just use that as leverage to negotiate a lower cloud bill.
Interestingly, you don't hear about many businesses that are so competent that they move from the public cloud to their own private clouds. Some people will shout "Lock in!", but I don't buy that--I've migrated between public cloud providers before and it's work, but if you're competent enough to run your own private data centers then you're plenty competent enough to migrate to them.
I haven't looked, so the number after the dollar sign might be going up or down.
But if it is not going down by more than inflation then the cost is basically going up.
Title should maybe be switched to "Google Cloud vs AWS Onboarding Comparison for YC companies".
Not to say that isn't a useful article on its own, but it's hard to draw too many meaningful conclusions for the rest of us when the first line of the AWS bullet points is "reach out to dedicated YC email".
I was able to get 130k free GCP credits (over 3 years) just by filling out the startup forms... as a complete nobody. You don't even need an affiliate to sponsor you.
I'm not saying anything for or against AWS or GCP. All I'm saying is most startups experience with AWS won't begin with "Send email to dedicated YC email address" so it's hard to draw conclusions from this article about what AWS support is like for "regular" start ups.
While I agree that this is a confound and as someone who also went through the same process, GCP also knows that you're applying as a YC company, and is a PITA regardless - in that way, there's a normalization factor of applying to both entities' startup credit programs with similar social factors. I've heard the same pains with GCP from non-YC founders about the free credits process and from non-YC technologists about bad customer service.
The grants start out small (I think the first one was $3k) -- but then once you spend 75% of them you can apply for the next round, which for me they gave me $17k out of a possible $30k based on previous usage. After I spent around $15k for that over the next year I applied again and got $100k.
Just one note though that I already had an MVP that I was running on GCP at the time I applied (but I was within the $300 free credits that I started with so I didn't pay out of pocket).
If you hit it big, you’ll spend on their platform for a long time. You don’t have to explicitly promise anything because you’ve built everything around their platform for months/years.
vendor lock-in. Good luck switching away from GCP once you're tied in to their cloud services. Might as well spend the VC money and not worry about moving.
Maybe things have changed since I applied - but I have not talked to a single sales person and only communicated via email with the google cloud for startups team. The process was so quick and painless that I at some point felt like I was being scammed and had to make sure I was not going to end up with a huge bill after they pulled the rug out from under me.
It's not too bad if you use the firebase store. But yeah using the firebase realtime db for anything that requires relationships/indexing can be kinda cumbersome.
I've built a prototype on top of it and it's fairly rudimentary but you can make it work. My biggest concern is that there are no case studies I can find of people using it past the prototype stage and how the pricing/scalability works out at that point.
We've spent a lot of time working around the limitations of FireStore, but it does work reliably. Pricing is VERY hard to extrapolate from early use; all it takes is one feature request and your assumptions are blown.
Love to hear from anyone who has gone beyond the prototype phase.
In 2015 I worked at a start-up and was in charge of applying for various cloud start-up benefits at AWS, GCP, etc. At the time getting GCP credits was the hardest and required meeting with a program coordinator after getting a referral from an industry recognized VC. It may have changed since then, bottom line YMMV.
Are there similar forms for AWS? I got 100k credits for my prior YC startup but I'm exploring new projects for a possible future startup and if I could even get 10k credits it would be a big help.
If you're planning on building your business on the cloud, the availability of free credits seems like the worst way to evaluate a cloud provider. Generous "free" credits might even be a warning that they have to pay people to use the platform.
If you truly need cloud hosting then that is $130k of free money no? That's more than many pre-seed rounds. If you use that for a bunch of Linux machines running some kinda containers and a non-proprietary database then I don't see any downsides.
Doesn't the same logic apply -- that new upstart is unproven and is essentially paying people to use their platform. The last thing your burgeoning new business needs is trying to debug issues on a brand new platform, you can't even hire industry experts to help because there are no industry experts.
So if you want to work on building your business and not debug cloud provider issues, avoid the new upstart.
This is also pretty specific about needing more free credits than GCP provides out of the box.
If you just want to get started with GCP, you just sign in with your Google Account for $300 in credit. When you run out you can just start paying with a credit card.
Exactly. Google: gives me $300 of credit no questions asked and tells me what things will cost upfront.
AWS: makes me apply to a program to get credits and if I want to price out anything it's almost a full-blown research effort where I have to dig through documents and cross reference tables between different services and I still probably miss something important. Free tier is opaque enough to frustrate even "use it at small scale and see," because you still have to do the research to see if they are pulling a "your first hit is free but try to scale and WHAM." Oh, but they're customer obsessed, so after much begging and pleading they'll refund half of WHAM, one time.
I don't really agree with that comment. I think all of the megaclouds have undesirably complex pricing structures, but they also all provide pricing calculators. AWS has one here: https://calculator.aws/
If I punch in the services I want to use, I can quickly see how much they will cost, and it's easy. I recommend clicking "Advanced" inside services on the calculator if you want to be sure you're not going to run into an edge case that costs lots of money unexpectedly.
If you use any of the megaclouds without first understanding the price structure, good luck.
Personally, I think a lot of companies would be better off using DigitalOcean or some other medium-size cloud. The pricing is much simpler, and pretty much all medium-size clouds charge $0.02/GB for egress bandwidth, either globally or in the US, depending on the provider.
In contrast, the megaclouds make shocking amounts of money off of their extremely pricey egress bandwidth, among other highly profitable aspects of their business.
Even still, AWS is a solid cloud platform, and they really do seem customer obsessed compared to what I've seen happen on GCP.
I don't really agree with this comment, because we've gone through year after year of new AWS cost analysis and pricing tools that always seem to miss the mark in a major way that's only obvious in hindsight and always seem to be consistently worse than GCP and Azure. They never seem to fix the flaws in old tools, they just tack on new tools with new blindspots. Hence my comment about cross-referencing. If you're willing to put in a lot of effort across tools, you can get a complete picture, but it really does take a lot of effort and foresight into the exact structure that your solution is going to take (which involves cross-referencing documentation). Amazon doesn't make any effort to quote you a price once they have enough information, they always make you work for it. If a price transparency tool is gated behind enough effort, is it really a price transparency tool?
That said, I'll grant you that AWS is not the only megacloud leveraging opaque cost structures. They just leverage them more.
My experience with AWS customer support was not so good. At the time I had very little knowledge about what I was doing so I can't recount all the details here, but the short version is I had a wildcard cert tied up with a resource that wasn't mine (something on Amazon's side was referencing it) so I could not delete the cert which prevented me from doing something with the associated domain name. AWS has no mechanism AFAIK to report a bug in their system, so I had to pay for developer support (which was expensive to me at the time). Even then, the issue was never actually resolved for me. I think I actually switched to a totally different domain name just to get around it. Here's a support thread where people are having the same issue:
> pricing tools that always seem to miss the mark in a major way that's only obvious in hindsight and seems to be consistently worse than other clouds.
Can you provide an example of a service that the AWS calculator doesn't compute the correct price for?
I honestly can't remember ever being surprised by what things on AWS cost, and I don't think I'm that shockingly good at detecting hidden costs.
Cloudwatch costs have been the ones my coworkers complain about. The one that got me was sagemaker -- the console had a bug where if you went to a region without endpoints, it would pop up a tutorial screen and the tutorial screen would "stick," hiding endpoints in other regions. Which led to hanging endpoints, which were covered by free tier for a few days before costs exploded. We had alerts active, but they alerted when we span up the resources, which was expected, and didn't make it obvious that paging through all the regions ensuring there were no running resources (itself painful) and watching daily costs for a few days was insufficient to ensure that we weren't going to have a $700 bill at the end of the month (or maybe $1400 -- I forget if $700 was the half refunded cost).
When I shared this anecdote at Re:Invent, the sentiment at the table was "lol that's cute, here's my story with an order of magnitude higher price tag." There were 5 or 6 of us, and my story was the smallest surprise cost except for one other person who was even greener than I was.
> Can you provide an example of a service that the AWS calculator doesn't compute the correct price for?
Can you tell me how I should have used AWS calculator to prevent my surprise charge? You can't, because AWS calculator assumes you know exactly what you're asking for, and the problem with opaque pricing structures is that you sometimes don't.
Other clouds tend to be much more upfront about "this will cost X," "this is costing X," "you're out of free tier," etc.
Yeah, that was weird to me also. It just sounds like YC companies collectively use enough AWS to get some premium tier support. I assume GCP has something similar, but YC companies don't spend enough there to get it.
AWS has, for a few years now, had a team dedicated to ensuring that AWS wins all of the successful startup business, which they know may turn into huge amounts of revenue for the few startups that succeed. The team was headed up originally by Paul Zimmerman (https://www.linkedin.com/in/paulsloanzimmerman/); he may be a good person to reach out to if you're hoping for some credits.
I definitely raised my eyebrows at "access to AWS YC slack"
It's really nice to hear from high-prestige companies about their experience because it gives outsiders a sense of what prestige might fix. If you're a small startup looking for help using AWS, finding someone to introduce you might really help! Not so much with GCP.
For what it's worth, as a non-YC company, I found the GCP onboarding process to be far superior in every way, and my experiences were essentially reversed. For context, we started with GCP for Startups 1.5 years ago when we were just getting beyond a demo, and with AWS Activate 1 year ago, when we had almost no revenue and just about 1,000 users. We were based in the Santa Fe Business Incubator (NM, USA).
All in all, it had taken our incubator at least 1.5 years to get enrolled in AWS Activate, and 4 days in GCP for Startups.
When we applied for GCP for Startups, we were approved in a week, and were contacted right away by an account manager and a support engineer who offered us help with anything we needed. When we applied for Activate, it took a month with lots of followups, and being shuffled between 4 support team members who were communicating with the Activate team (there was no way to contact them directly).
It definitely felt like our business was important for GCP, and for AWS, it wasn't. It was an easy choice.
Yeah, I think having a YC dedicated mail address at AWS helps. As I am still waiting for a response from AWS regarding joining one of their programs while GCP response was within the day. All access sorted the next day.
Based on other stories though, the customer experiance between the 2 companies appears to be simliar even if not a YC Company.
Google has never had a good reputation for Customer service, they believe they can solve all Customer Service with bots and Automation, this will ALWAYS lead to a lower level of customer service.
Amazon started in retail sales where customer service is king, so it naturally has batter customer service philosophy than Google.
Google has shown zero signs it even desires to have a customer service philosophy that is remotely similar to Amazon, or even Microsoft Software which is somewhere between Amazon and Google on the customer service scale
Of course Google could offer the same thing. My proposed title change would be even more recommended in that case.
My point is that obviously AWS thinks that on average being in YC is going to result in more revenue for them and therefore they prioritize support for YC companies. As a non-YC company I won't get the same treatment which makes any conclusions from this article less useful.
I used to be a huge fan of GCP and bet on it to power my startup, and have come to greatly regret it.
Recently, I needed to increase a CPU-limit quota from a small number (like 16 vCPUs to 64 vCPUs) - nothing crazy. In the past, the quota increase system was more or less automated and would only take a few minutes to process.
This time, however, GCP denied my quota increase and forced me to schedule a call with a sales rep in order to process the quota increase. It was the biggest waste of time and kind of goes against the entire point of instant cloud resizing.
It also feels like the velocity of new features, instance types, etc has slowed down dramatically in the past year. Also, while I'm ranting, Google Cloud SQL is probably the worst cloud service I've ever used (and it costs an arm and a leg for the pleasure!)
CloudSQL was slow for us until we do the following:
1) Increase the disk size to 100GB as this increases the IOPs
2) Switch to using private IP addresses. Huge speed increase
3) get rid of cloudsql-proxy. Another huge speed increase
These 3 things have kept our database instances very small and costs low.
This causes some other problems... Kubernetes doesn't do a very good job of treating the sidecar as part of the main container so you will get random disconnects in your app when the proxy is restarted unexpectedly. This is actually why we abandoned it, just to deal with the random hiccups. Hadn't even benchmarked it.
Yeah, have hugely over provisioned disks for IOPS. Am still using public ip + cloudsql-proxy because the alternative didn't exist when I first deployed, but I'll try switching.
I went through this during last summer. The nice thing is that you can switch to private ip and cloudsql-proxy will still work. At least you can isolate your changes.
> Switch to using private IP addresses. Huge speed increase.
Interesting. I'm looking Cloud SQL right now and the advice seems to lean in the opposite direction: use public IPs for ease of connecting. Can you quantify the decrease in latency? All I can find is bits about reduced network hops.
I honestly don't know what the difference is but the number of hops is probably a contributing factor to the decrease in speed. There could be some translation layer happening when going from the CloudSQL instances private IP to public IP with some security checks that slow it down.
I didn't have to worry about the 'ease of connecting' when using CloudSQL as all my GCP services are in the same VPC.
Also, private ip is one less security problem to worry about.
Is it speed of establishing a connection or does it affect pooled connections? What database engine are you using?
We abandoned cloudsqlproxy partially because the sidecar in Kubernetes caused random disconnects, but also because it just had bizarre network errors sometimes. (That was mostly when connecting from outside GCP though.)
We are using PHP and thus no connection pools at all. This definitely doesn't help us with speed. With public IP there was some mystery point that just slowed everything to a crawl.
I got the same treatment but I think I came off as annoyed enough in my message to sales (with an implied threat of just moving to AWS) that I didn't get an upsell conversation.
- No way to upgrade major postgres version without full export and import into new cluster.
- Incredible delay between postgres versions. IIRC, it took nearly 2 years for them to add postgres 11 after it was released.
- HA is basically useless. Costs double, still has 4-5 minute window of downtime as it fails over, doesn't avoid maintenance window downtime (both primary/standby have same maintenance window) and you can't use it as a read replica. Honestly, feels like a borderline scam since I'd imagine a new instance could be spun up in the same amount of time a failover takes (but I haven't tested)
- With default settings, we experience overly aggressive OOM-killer related crashes on a ~monthly basis during periods of high utilization. On a 32GB instance, OOM killer seems to kick in around 27-28GB and it's incredibly annoying.
- Markup over raw instances is almost 100%, with no sustained use discount outside of a yearly commit.
It's just a lot of money to pay for a crashy, outdated version of Postgres.
To be fair, it looks like GCP supported Postgres 13 (Nov 5, 2020) before AWS did (Nov 27, 2020) and AWS currently marks Postgres 13 as a preview. Maybe GCP had a large initial engineer-cost to support multiple versions of Postgres and now the incremental cost to add new versions is small?
> It's just a lot of money to pay for a crashy, outdated version of Postgres.
Have you looked at other options? I'm evaluating GCP SQL and the comments in this thread are scary. Seems like Aiven might be a good way to go. I've also briefly looked at CrunchyData's Postgres Operator [1] for Kubernetes but it's a lot of complexity I don't really want.
Amazon RDS waits until the "dot-one" release of a new PostgreSQL major version before releasing to general availability. This ensures that all supported extensions and modules (71 in Amazon RDS for PostgreSQL 13) have been updated to work with the new release and with in-place major version upgrades. Amazon RDS releases beta and "dot-zero" versions of PostgreSQL in the Preview Environment, so that customers can start testing and developing against new features in the latest major version.
I’ve only looked at CrunchyData which does seem like more complexity than I want - I was willing to suck it up pay the premium but the monthly OOM crashes have forced my hand - but to where, I don’t know yet
To clarify, it’s a lot more work than bringing up a snapshot. You need to do a full export as SQL and reimport as SQL. Super annoying, slow, and requires hard downtime.
Am using SQL proxy but doesn’t do much re: HA.
I don’t know, I’ll probably just run my own Postgres at some point. The only peace of mind that I get from Cloud SQL is the automatic backups.
Why don't they make the upgrade seamless? If it's truly an export/import process, then it should be dead simple for them to do that on their end. Especially after they've snatched your db from serving requests
Of course there's an API for creating and resizing instances. The issue is there are quotas that cap how many CPUs you can have, and increasing that is a manual process.
I am founder of another YC backed company. We based our startup on GCP infra. They have great tech (for the most part) but I regret it so deeply for two reasons:
Support or desire to help customers is non-existent. For any questions, they want us to upgrade to paid support and pay them at least 10% more every month for that (we ask like 1-2 questions a year). What? We are already paying you thousands of dollars every month! I get included support for all my software subscriptions - so this is my biggest beef. Also, when they do help, they keep passing you around from team to team and dont resolve issues as well I’d like. It is just not a company I can love as a customer.
Their status dashboard is a joke. They dont even report minor outages, when they do, they start after a huge delay and update very slowly. And worst of all - when it only affects a single zone or a single region, they remove it from historic reports so everything looks green/great.
Basic business hours support is $29/month or 3% of your service spend, whichever is greater. 24/7 is $100/mo or 10%, which also includes outage assistance.
I’ve also worked for places with enterprise support ($15,000/mo or 10%) but of you’re bringing in millions per month it’s definitely worth it.
The AWS personal health dashboard is also pretty reliable. The public status page is the source of many jokes.
While you do pay for AWS support, I must say that in my experience AWS support is pretty top notch. I don't particularly like the (somewhat) recent changes where the priority of your ticket is based on your support plan but I'm guessing it's because everyone always chose "critical" when making small support ticket.
In my experience with AWS support, it's a major difference on whether you are asking EC 2 questions or some of the lesser used service questions.
For Media services, the supporter will almost always need to coordinate with an internal team, which there is no visibility over, and then it becomes a game of telephone to make the supporter relay the information in a way the internal team understands. I've had the same thing happen with peering/networking related questions.
For EC 2, VPC, DynamoDB kinda questions, they are indeed pretty good.
I guess that makes sense. The quality of their support is probably directly correlated to the level of internal tooling to help diagnose issues. For more popular/older services that tooling is probably better.
I don't know about Google's support offerings. But when I worked for a startup that was on AWS, here's what we'd do:
1.) Try to figure things out ourselves
2.) If we can't figure it out, subscribe to AWS Support.
3.) Get question answered and then turn off the support plan.
You'll have your support plan for the rest of the month and pay a prorated amount for the days during which you had support. It's quick and cheap.
Last I checked GCP has a minimum one month billing for support. Then again, we're talking about $100/mo here, saving half of that pays for how many minutes of engineer time?
Lol no. AWS NEVER updates its status page unless it is a massive incident that everyone notices.
As for support, if you pay for the most basic plan you will get a basic answer within 48 hours usually. Its decent. The more you pay, the better the support.
This feels like a somewhat narrow and biased comparison. It is focused around getting startup credits, and while that's important, I think it's a very limited view of onboarding.
Better would be to think about what the general initial usability of these services are. How easy is it to spin up the compute load? Create reasonable IAM policies? Debug problems?
My own experience (and bias) is that while AWS has vastly more features, GCP is much more usable. The latter feels like a coherent setup with projects and IAM. AWS always has a surprisingly amount of work around org accounts and IAM setups.
So maybe it's faster to get AWS credits, but it's much harder to make use of them.
I agree, post is very focused on customer support and getting the credits, not at all on product usability. I didn't start building on either AWS or GCP with the expectation I'd have a human to talk to, and from that perspective, GCP was much, much more usable. I found (and find) AWS's interfaces and documentation to be a maze.
>The latter feels like a coherent setup with projects and IAM. AWS always has a surprisingly amount of work around org accounts and IAM setups.
AWS Organizations is atrocious compared to GCP projects. I truly do not understand how/why AWS is doubling down on the awful user experience of Organizations, cross-account access, creating multiple accounts, etc. It's truly the antithesis of customer obsession and yet they show no signs of moving away from that model. And not only is the user experience bad, it also just feels hacky. AWS Organizations and Control Tower feel like poorly applied band-aids rather than real solutions.
I actually prefer AWS to GCP in most respects, but the AWS Organizations shitshow is something I can't personally get over.
Google's business model is to have no customer service and have customers discard-able without notice when Google chooses. Why would a business want to do business to business with Google and actually pay them at this point? They do 'free' sort of well but they are a terrible option for dependable partner whether you pay them or not.
Effect of ToS violations
Google-wide disabled account
In some cases a Google-wide account (which covers access to a variety of Google products like Google Photos, Google Play, Google Drive, and GCP) will be disabled for violations of a Google ToS, egregious policy violations, or as required by law. Owners of disabled Google accounts will not be able to access their Google Cloud resources until the account is reinstated. If an account is disabled, a notification is sent to the secondary email address provided during the signup process, if available. If a phone number is available, the user is notified via text message. The notification includes a link for appeal and recovery, where applicable.
In order to regain access to their GCP resources, owners of disabled Google accounts will need to contact Google support and have their account re-enabled.
To minimize the effect of an account being disabled on Google Cloud resources, we recommend that you add more than one owner to all resources. As long as there is at least one active owner, GCP resources will not be suspended due to the one of the owners being disabled.
Given the Google account horror stories that pop up every few months, seems risky if you're solo/only have one GCP owner.
Google will suspend entire accounts even if you have more than one owner.
A few years back, our business credit card was somehow stolen and used to buy Google Adwords. We disputed the charge with our bank. A day or two later, at 4am local time, our GCP account was suspended for fraud (presumably because the same, stolen, card was attached to that account). All instances were stopped and our service was brought to a halt.
We couldn’t contact Google Cloud support because our account was suspended. We had to go through our network to get our account re-instated. Pretty awful way to start our morning, to say the least.
On the link above there is a separate section for billing account suspensions and it seems to work how you described.
You can fairly easily swap billing accounts a project is using if you still have an organisation admin account that isn't suspended. I saw this risk coming at a previous company and tried to get them to treat billing accounts like any other important resource and go for redundancy, but the director could not see the value and our Google account manager denied the risk existed, luckily they have been more fortunate and this has not happened to them yet, although it did come close once with some issues with the payments.
Credit card companies tell you to work with a company before issuing a chargeback, as chargebacks are a last resort. The above form helps you keep you account active without being swept up in the chargeback process.
It's something we all want to keep improving. I'm hoping as SCA (strong customer authentication) rolls out across europe and maybe other countries pick it up, it should cut down on issues like this.
I believe that the "work with a company" guidance is for a dispute (e.g. you ordered something, but didn't get it or are otherwise unhappy) -- not an outright unauthorized purchase by someone other than the card holder. I don't see what most companies would do if you approached them in this case: you have no order number, no information about the purchase, nothing to return, etc.
The only logical thing to do in the case of a stolen or copied card is to contact the bank directly, so that the card can be canceled and fraudulent charges reversed.
What happened once the account was reinstated and everything was up again? The customer service obviously sucked; did they acknowledge that and try to make it right?
AWS does not suspend your AWS account if there's an issue with you Amazon account. AWS reps are contactable and dare I say it capable of escalating and resolving issues.
The idea that someone could use a stolen credit card associated with an account, buy something on some Google service, have a chargeback processed as fraud and that would trigger Google's suspension of services on GCP is absurd.
> To minimize the effect of an account being disabled on Google Cloud resources, we recommend that you add more than one owner to all resources. As long as there is at least one active owner, GCP resources will not be suspended due to the one of the owners being disabled.
When even Google themselves is recommending gaming their system since they can't guarantee it won't screw you over, that should be a warning sign.
I was involved in a mid-sized software company migrating from AWS to GCP. My experience was that the GCP team was very hands on, and that enterprise support was very responsive.
The support isn't perfect, nor is the product -- but I would say the level of customer service for GCP can't be compared to other Google products.
Not the original poster but we migrated to GCP because:
1) AWS was extremely expensive
2) Our GCP bill is about 1/3 of what AWS was
3) The Kubernetes offering is top notch
4) Google giving us credits and offering us consulting were the triggers that started us talking.
Seems like k8s is a big seller for a lot of companies. We don't use it currently on our team and the larger company is whole-sale in on AWS I'm not sure they'd ever be able to make the switch.
Always like to see why people make these big changes though, thanks!
I spent over a year doing a full migration from AWS EC2 + ECS instances to GCP + docker + kubernetes. It was a huge task that has paid off very well.
1) Costs per customer are lower because you can fit more containers per VM due to kubernetes doing the scheduling for you. Customers also include developer environments.
2) The number of deploys is way up because there is a simple and established pattern that everyone follows.
3) The speed of creating new services has increased because of the established patterns with containers, kubernetes resources, and deploys. Thinks days vs weeks to get something running.
4) The number of Ops issues are lower because kubernetes handles so many things for you. For example, if a deploy is incorrect for some reason, the old service is sitting there running. No outage = no escalation = everyone sleeps at night.
Even if I was a tiny startup, I would still recommend using Kubernetes. The patterns, tooling and insight that Kubernetes gives you will save you TIME. The time saved is worth more than the tiny cost of a 3 node Kubernetes cluster. That is time you can use to develop your product and sell it vs time spent ftp'ing binaries to your Digital Ocean instance. :)
I know you're only picking on Digital Ocean incidentally here, but their managed Kubernetes offering is Pretty Okay. I use it for my personal stuff and it's pretty much everything I expect from a managed Kubernetes offering.
Just don't use EKS. That is managed Kubernetes in the checkbox marketing sense only.
- Google has a lot of experience running containerized services because of Borg. Google Research says that they have been running these workloads since 2005.
- The above gives them insight into how to do this well. They would have the internal infrastructure, logging and monitoring already setup.
- They are the creator and still a major contributor to Kubernetes itself which means they can insert their influence on it.
To be fair, my only experience running Kubernetes is on GCP so maybe they are all this easy to use and run. The internet tells me that other offerings are not as good. Compared to running workloads on EC2 instances and ECS, using GKE is amazing and pain free.
Note: I am responsible for 5 clusters, 300+ nodes total and several thousand running services. Not huge by any means.
Amazon also has been running internally on containers for years before ECS. But workloads for massive FAANG companies designed by FAANG engineers turn out to be quite different than most AWS/GCP customers.
Just like EC2 wasn't Amazon selling its "spare capacity during off-peak" but always a purpose built service with completely isolated data centers and network fabric from day 1, the connection between Borg and Kubernetes is tenuous at best.
Honestly, I would encourage you to evaluate ECS and Fargate. Forget Kubernetes and direct instance management. Try a construct like https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws... and see how much simpler it is than all the K8S overhead.
Source: I'm at Amazon but I work on none of these services and opinions are my own. I played with K8S and am eternally confused how it's as popular as it is. I guess it gives you the illusion of not having lock in?
I only have one thing to point out about Borg and that is large numbers of Borg developers started working on Kubernetes. They got rid of Borg mistakes and replaced them with all new mistakes. :)
I don't see K8S being about lock-in at all. Using a cloud provider gives you some sort of lock-in and you should be utilizing that providers strengths.
Using Kubernetes is all about the patterns that it provides and how it removes a certain class of problems for you. Deployments, scaling, logging, etc are some of the patterns it provides and the consistency matters. How many of us have worked at companies where deploying two services have been completely different? One team runs the jenkins pipeline while another ftps the files over. Now multiply that by several services and several tasks (logging, scaling, etc).
1. Is instance management on ECS really annoying enough to justify the higher fargate price?
2. Does ECS have something similar to pods, i.e. colocated containers which can communicate over localhost, share filesystems, etc? A quick search didn't turn anything up.
That's fair. As I pointed out in another reply, most of my experience is with GKE so I may be spoiled in how painless it is. All I can say is that GKE is treating me well. Node upgrades are seemless, defaults are decent, it was very easy to create clusters through the console or terraform. I only run 5 clusters and that is not that huge compared to others.
I've used ECS more than fargate. My biggest problem comes from reliability and having to manage things. We still run things on ECS and we get about 30x as many maintenance things we need to do, like retiring instances or instances just dying on us and we have to move things off of it manually.
I have a little bit of experience in Fargate from a previous job but not enough to make a real conclusion about it.
I've never used EKS but I do have friends that would rather roll their own version of Kubernetes on EC2 instance instead of using EKS.
I think this sorts of makes sense for free services like Search, Gmail and Youtube, where each "client" (user) only gives them a tiny amount of revenue.
This makes very little sense for Cloud Computing where each one your clients gives you large amounts of cash. Maybe they are too used to the first scenario.
That is not even remotely true. In the latest earnings, Google Cloud revenue grew by 46% year on year, and they said GCP specifically grew faster than Cloud as a whole.
I've used GCP to great success (with some speedbumps) at 3 companies I've worked for (small and large) over the last 5-6 years. I have a lot of learnings from the experiences to pass on, and one of the key ones is this: work with a reseller.
If you've lurked on HN over the past decade you'll have seen tons of stories about how bad Google's customer support is. I don't think GCP's support is anywhere near as terrible as it is for Google's consumer products, but it does have a lot of room for improvement. If you work with a reseller, you'll get much much better results.
There are several GCP resellers that are really good and knowledgeable, and often are staffed by former Googlers that worked on GCP. The very first thing you should do if you've chosen GCP is to find one.
This feels like working really hard to solve a problem that isn't yours. Why not just get your services from a company that will support you? Why is GCP worth doing this for?
I'll echo this reasoning as one for choosing GCP. Many of the offerings are fantastic on their own - BigQuery, Pub/Sub, Dataflow, Firestore, Cloud SQL, GKE - they're all (IMO) great products that my teams have used to work wonders. On top of that, they're all well integrated and more seamlessly work together in my experience vs AWS services.
I've also found integrating access controls via IAM to be really powerful and also pretty user friendly. In one experience, this helped us scale the sales process and allowed us to easily address sales push-back on data security and compliance.
Also, engineers really seem to enjoy using the cloud console, even at later stages when it was mostly just read-only and had a lot of access controls. The UX is just way better than AWS.
As mentioned, I've experienced issues with GCP and there are reasons to choose AWS instead. I can expand on these if anyone is curious, but wanted to list out the reasons to use GCP, as I feel like most of the posts you see about GCP on HN skew more negative than my own experiences.
Thanks for the tip. What I don’t understand is why GCP can’t get customer support right but resellers can. You’d think Google would have more resources than a reseller to provide better service. Is it the size of the company? Company culture?
Google is squarely a product company. They make products, they manage them as products, and users are a function of marketing, product performance, and support. Professional services firms are customer companies. They have no product other than solving business problems for their customers. Its likely that a VP at a reseller could name their top 15 clients and their relative spend from memory, whereas a Google VP could spout off 15 metrics that drive their division, but not have much awareness about customer spend or what’s driving it within their business.
The overly commercial approach by the GCP rep is so short sighted but not uncommon.
I recently worked on a project which had a potential vendor spend in the millions if not tens of millions when it went to production and scaled.
Without exception, all of the vendors we spoke with early doors were horrible. Only interested in qualifying the size of the opportunity and when it would sign, not giving us access to the right people and playing horrible politics with our client.
I won't list them all, but the worst of the bunch was Snowflake. They were a nightmare and completely shot themselves in the foot.
If these vendors would have just helped like a partner with even a medium term focus, any one of them could have signed an enormous deal within a year.
Instead, we ended up going open source and AWS native, just because they are so much easier to deal with than many other vendors.
Having ran an AWS partner and seen them up close hundreds of times, I agree they are generally very nice and easy to deal with. I did see a recent cultural change when I had a problem in my own startup, but suspect they are still nice to the big boys.
The only reason that I can think of as to why you would want to talk to a sales rep (because you don’t actually need to to use cloud services) is to try and wrangle some sort of discount.
But the sales rep knows that their job isn’t strictly needed, because the customer can sign up and use services without talking to anyone. They only exist to upsell services.
There’s almost no intersection between what a customer wants and what the salesperson provides, so why wouldn’t it be a bad experience for everyone?
We've been using GCP for a couple of years now and have nothing but good things to say about it. We also used Azure and AWS before migrating to Google, and the whole GCP platform feels a lot more intuitive. Lacking in some areas, sure, but more than makes up for it in other areas like Kubernetes. We had to use support a handful of times for various issues ranging from technical to general billing questions to long term commitments, and all incidents were handled quickly and professionally.
From personal experience in multiple startups I can confirm that with regards to customer service and sales processes AWS is light years ahead of GCP.
That said I have heard even better stories about Azure actually - apparently it is yet in another league of its own in terms of perks and actual service game.
The Azure service game is indeed incredible. Every experience I've had with them has been excellent.
A client once requested we file a support ticket with Azure to help deal with a performance issue my team was working through. It wasn't really all that urgent, but the client requested we use the highest urgency level anyway. So I filed our ticket.
In less than a minute I felt like my phone was being blown up by every engineer at Microsoft. And the messages they left made it clear that the fate of humanity hinged on resolving our issue within the next five minutes.
On top of that, the depth and intelligence of the support was downright humbling to this fellow engineer.
(At the end of the day, it turns out we'd screwed up and left debug logging on.)
AWS has had many years to build and polish their sales process, and it shows.
GCP felt like the engineers built a good platform and then tossed it over the wall to some old school VP of sales type people to pitch it in whatever way maximizes their commission checks.
Azure feels like a finely honed enterprise sales org that understands what they need to do as underdogs in this market.
Most people on here dislike Azure because they don't buy into the Microsoft ecosystem, which is absolutely understandable, but if you do buy into the ecosystem, Azures really pretty great.
AAD ties everything together (this is a pro or a con depending on whether you use it), meaning you get secure SSO (including biometric security) for everything from device provisioning to machine identity (through service principals), and then Microsoft 365 E3 and E5 licenses offer every internal business tool you'll probably ever need in one place.
Microsoft is not without their faults and I know a few sprinkles of comments in this thread have definitely highlighted issues I know people have had with them, but in the whole I have to say this:
- my previous company switched from AWS to Azure and had significant savings, their sticker price is not the price you pay, and we were not spending millions to get deep discounts either (they were more interested in us being in a contract instead which makers some sense)
- the support we dealt with was really good for the most part, I felt it was pretty comparable to AWS
- they have really good uptime for the services we used (blob storage, cosmos DB, mssql and postgres, container hosting)
Biggest downside though is they seemed to always be going through SDK changes. I think this had a lot more to do with migrating everything to .NET core though, still was very annoying. Their non .NET sdks were a bit more stable though.
The only people who like Azure are managers that get bonuses for their stupid contracts. Azure is not built for engineers. All of their offerings, especially the automation and tooling, are vastly inferior to AWS and GCP
As I've said elsewhere, the $29 and $99 support plans for AWS have to lose AWS money. At least a (fairly long) while ago the support was nuts through these. Total expert level / out of scope. Frankly I'd encourage them to hold scope a bit more but its obviously working for them.
AWS employee here. When I was an AWS customer I was also often surprised by the quality of the support answers. Now from the other side I can explain why: even on the cheaper support plans it isn't uncommon for difficult questions to make their way back to the relevant team and the answer you are getting was often written by an engineering manager or engineer on the team that built and operates the thing you were asking about.
Obviously there is a balancing act here to avoid slamming the engineers with too much load answering support questions, but it is not uncommon for customers to be getting answers from the people who built the thing. And on my team at least we always try to use support questions to know where we need to improve documentation with more troubleshooting steps, etc.
> And on my team at least we always try to use support questions to know where we need to improve documentation with more troubleshooting steps, etc.
Absolutely. In my ex-team at AWS, you could literally see visceral pain on the on-call's face when a customer tickets-in with a totally avoidable issue. The feedback from such customer contacts did inform most of the product roadmap.
And most certainly, the most heavily prioritized and celebrated feature launches were the ones improving operation excellence including fixing things that a lot of customers had complained about.
That said, cloud support engineers, often times, in my experience, were more knowledgeable than software engineers owing to their interactions with customers which lead them to internalise a tonne of troubleshooting patterns. Only a novel issue would stump them where a software engineer would have to work in-tandem to sort it out.
The detailed internal knowledge-base that these support/software engineers write for issues impacting customers probably also plays an important role, because then even semi-technical folks like TAMs can more or less help the customer out pronto by searching through the knowledge-base, without requiring to escalate further.
Agreed. AWS technical support is exactly that. It's been a pleasure every time, that when I open a ticket, I actually have a real technically minded human behind it.
As a converse, I've also had the extreme displeasure in dealing with Oracle and Tenable/Nessus support...
With Oracle, its either a continual feature-push to go to professional for only starting $50k/yr more (NO), or troubleshooting ends up asking 100 questions for your question.. And if you answer them, they give you 100 more. Effectively its a technical DOS in the hopes you abandon the ticket.
Nessus/Tenable is similar. They want you to use their terrible tenable.io (which isn't fedramped), and will badger you incessantly. And service tickets demand enhanced logs be turned on and provided to them. Their tool can censor some passwords, but have caught passwords in there along with services, addresses, and exploit data about them. And even if you censor the logs prior to shipping to them, they will put their foot down and demand unedited logs.
This would explain the response I got (backstory - I am a contributor to a number of open source packages / not totally clueless, but was coming in from a micro personal account). I was like, how the heck do they afford this response for $99! (or whatever it cost back then - this was a long time ago).
I kept my support plan active for a year as a courtesy though I never had another question aside from the first two I put in.
That said, as a programmer I like time to focus so being asked customer questions would drive me nuts, hopefully they filter out the idiots who just can't setup things right (80% of issues are not bugs but customer setup issues).
For most services, this is how the AWS escalation path works:
1.) Support engineer receives the case. In most cases, for most services, the run-of-the-mill support engineer has gotten enough training on the service that they'd considered a SME at any AWS partner/enterprise.
2.) If front-line support engineer can't solve the case, they talk to their more tenured friends.
3.) If they still can't solve the case, they escalate to Premium Support SMEs in the service. These are support engineers that have proven they have solved complex enterprise-level cases and get specialized training from the service teams on the internals of the service.
4.) If the SME can't figure it out, it's escalated to the service team (i.e. the actual software engineers that write the code).
Steps 1-3 deflect a lot of the annoying, unnecessary escalations. But sometimes escalation to the teams is unavoidable. For example, some service limits are hardcoded into the service and updates require a code push. Or (somewhat rarely) there's a service bug.
That being said, Premium Support engineer have access to the backend service code for most everything. While not every PS engineer knows how to code, on more than one occasion, I've dug through a code repo to figure out why something was behaving oddly.
Not if they sell the rest of their services profitably.
I keep throwing large sums of money at AWS because they insist on making their products useable. AWS wants to me adopt their products, they want to remove barriers, they want to help me make money. I have no problem continuing to invest in their services.
"Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers."
Out of interest, here are the other values:
Ownership
Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job."
Invent and Simplify
Leaders expect and require innovation and invention from their teams and always find ways to simplify. They are externally aware, look for new ideas from everywhere, and are not limited by “not invented here." As we do new things, we accept that we may be misunderstood for long periods of time.
Are Right, A Lot
Leaders are right a lot. They have strong judgment and good instincts. They seek diverse perspectives and work to disconfirm their beliefs.
Learn and Be Curious
Leaders are never done learning and always seek to improve themselves. They are curious about new possibilities and act to explore them.
Hire and Develop the Best
Leaders raise the performance bar with every hire and promotion. They recognize exceptional talent, and willingly move them throughout the organization. Leaders develop leaders and take seriously their role in coaching others. We work on behalf of our people to invent mechanisms for development like Career Choice.
Insist on the Highest Standards
Leaders have relentlessly high standards — many people may think these standards are unreasonably high. Leaders are continually raising the bar and drive their teams to deliver high quality products, services, and processes. Leaders ensure that defects do not get sent down the line and that problems are fixed so they stay fixed.
Think Big
Thinking small is a self-fulfilling prophecy. Leaders create and communicate a bold direction that inspires results. They think differently and look around corners for ways to serve customers.
Bias for Action
Speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk taking.
Frugality
Accomplish more with less. Constraints breed resourcefulness, self-sufficiency, and invention. There are no extra points for growing headcount, budget size, or fixed expense.
Earn Trust
Leaders listen attentively, speak candidly, and treat others respectfully. They are vocally self-critical, even when doing so is awkward or embarrassing. Leaders do not believe their or their team’s body odor smells of perfume. They benchmark themselves and their teams against the best.
Dive Deep
Leaders operate at all levels, stay connected to the details, audit frequently, and are skeptical when metrics and anecdote differ. No task is beneath them.
Have Backbone; Disagree and Commit
Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.
Deliver Results
Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never settle.
I've been working for startups for 20 years and I was never able to get an AWS Account Manager, much less a "Startup Rep". They must only care about spending, your email domain, the fact that you're an ex employee or likely to write about the experience. Cause it's always been "you're on your own" for me.
Then I would advise you try again. Our AWS bill is between 4 and 5 figures and we got dedicated startup account manager while on development within a week of asking.
No internal contacts, literally pinged support team and got it.
I literally just started a new sass on AWS and only have spend about 80$, today I got a call from a rep from their primary headquarters (unlike Google) , and he applied a 300$ initial credit and I can email him for a questions/support. No pressure just all around nice.
I've had similar calls from google in the past that were disasters, just someone hammering the up sales button, I actually had to block a google rep on my phone.
We spend 100s of 1000’s of dollars a month on AWS. Our rep tried getting me to upgrade to paid support, and I said there was no way I was going to do that. I’m already paying AWS millions a year and will be giving them more and more business as Loom continues to skyrocket in usage. No way I’m paying for support at that scale. It’s a bogus model to charge your customers more when they’re continuing to accelerate in growth and pay you more money. I expect to get more support and attention over time automatically because I’m simply paying much more and my need for support hasn’t proportionally gone up in the least.
That was it. Other than that one conversation, my experience with AWS has been _absolutely stellar_ every step of the way. We’ve struck deals with them for several products and have felt like they were fair for both parties. Instead of continuing to focus on getting me to pay for support, they focused on areas where we’d want to rely on other AWS products and have come back with some amazing suggestions. I am extremely impressed with the teams I’ve worked with at AWS - it seems like they really understand the customer, and that incentivizes me to stay and use even more managed services when the unit economics work out for us.
For GCP I’ve had reps reach out. I tell them there’s no way we’ll go multi-cloud because it doesn’t make sense for our business goals and detracts from them. They respond as if they never even heard what I said. I get that you have to be persistent as a sales person, but the response wasn’t even talking about how _part_ of our infra would be worth hosting with them. There’s no conversation - it’s just what GCP wants me to do for them.
The way that support is paid for within AWS is roughly like the way that tipping works in the US at a restaurant. The cost of the 'food' doesn't include the cost of your server/support person. Everybody knows that and so you pay the server separately in the form of a tip - usually as a % of the bill.
So I don't know for sure but my understanding is that what you pay for each service does not include any support cost. It just goes to the operations of the service. The support team/division is entirely funded by the support line-item in the bill.
Google Cloud needs to hit the numbers. And the only way to do this is to hit the big boys. Dealing with 1000 small customers who are petty and spending small vs dealing with 1 big customer that moves the needle. That is what is going on here.
Once Google Cloud manages to hit the numbers, you would expect things to be much better for the smaller players (this remains to be seen).
Not excusing their support operations, but I did manage to get on chat with someone about a Billing issue (to understand my billing better), and I was pleasantly surprised at the response time (immediate).
The slow and free support route is to use the Google Cloud issue tracker: https://cloud.google.com/support/docs/issue-trackers
You tend to use this if you are not critically impacted and you are also a developer.
By and large, things works as advertised. You can keep spamming that Feedback form in the Cloud Console. Product Managers do read it and address it.
AWS seems to realize that good service focusing on product needs will lead to more revenue in the long term.
GCP wants to front load revenue concerns, making product usage secondary.
Even apart from AWS's first-mover advantage, GCP should not be lagging as far behind as they are. Azure for example started a little later than GCP yet still appears to have more of the market.
We use Google for oauth, map and geolocation APIs for which we pay thousands of dollars every month and all my interactions with them have been very painful.
I remember when we asked for a meeting to discuss price raise on Google Maps APIs; The raise was ~100x for us.
They came with the head of the local market and 4 people. Basically told us to fuck ourselves and that from now on we'd have to be billed by a reseller instead of being billed by Google.
And then they spent an hour trying to convince us to move from AWS to GCP. I had the impression I was at a car dealership.
This is just an example out of many so even if I don't have direct experience with GCP I'm not keen to try.
I've been managing the AWS company account and the collaboration with AWS for the past 7 years.
And while not everything is perfect, my experience was great overall and I would do it again.
GC CEO, Kurian has enterprise sales genetics. Entire exec leadership he put in place, comes from SAP Oracle-esque sphere. They speak enterprise, even with startups.
GCP‘s primary target and starting point is enterprise.
And yet their engagement teams are remarkably ignorant of how enterprise works. Even after weeks of conversations I still hear a voice in my head saying 'oh you sweet summer child'.
On AWS: fast, close to our needs and all setup in days. After our one year program cam to an end we still had around 50k in credits. One email asking if they could extend for one or two months, they extended for 6. We also messed up the last month as we did not setup any limits and spend more than 17k in cloud computing - they offered us an 80% discount!
On GCP: We got accepted very fast (2 days). But we struggled for 2 months to get GPU quota. The communication is not fluid and we were pointed from sales rep to support to sales rep.
Also from the management perspective (but this is purely my opinion), GCP is a labyrinth. You need a phd in GCP to setup your users with permissions. And i still could not figure out how to create good usage reports out of it.
AWS has been amazing, even for someone who doesn't spend beyond 1,000 EUR. You always get a human on the other side of the line. GCP / Google I have no time for.
> told rep about infrastructure we were thinking of using but they needed a dollar monthly amount since they were in sales and didn't have an understanding of the infrastructure
I am in computational genomics and this was exactly my experience with Google as well.
Think where the 3 major cloud operators come from:
AWS: Eat-your-own-dogfood, consumer company
Google: Eyeballs are our product, cloud,apps,services is a side-effect of our underlying ad business
Microsoft: Developer first, keep people in our ecosystem, make it easy to adopt, hard to get out of.
Google has good technical product inside GCP, nice fancy stuff like Spanner, etc. All spun out of their own requirements and needs, which is a scalable, invisible, processing engine. Their ad business has two sets of people, ad buyers that bid for their business and don't have alternatives, and anonymous viewers that google sells back to the ad buyers. They're rent-seekers that clip the ticket coming and going, using their search as the lock-in.
No wonder they have trouble setting up a true B2B or B2C business. They're not built for it.
Both AWS and Microsoft are at least either B2B or B2C and understand that it's not clipping the ticket, it's making margin over the cost of supply of goods and services.
Microsoft has even learnt the lesson of losing your lock-in advantage of the desktop.
Google has lost that fundamental trust to be able to both operate as a rent-seeker in the ad business while also delivering actual goods and services as a B2B/B2C business.
We've had pushy account reps trying to upsell from both vendors (and not knowing that we already talked to another rep). We've also had reps get actual engineers on the calls early who advised us fairly on which of their products to avoid and how to exploit various savings options or soon to be released features. We are a consultancy and so these reps were sometimes interacting with us as a direct customer/potential customer and sometimes via our clients (both larger existing customers of AWS/GCP and totally noob unknown startups)
I'm my experience, there is no pattern other than some CS reps are good and some are aren't. Getting credits in both cases has always been a PITA at the start and than easy when the right person to make the call was reached.
I recently joined a new company to start a Data Science team. They had an existing GCP account for years that they had small usage on and had a long term valid form of payment. I request a tiny increase in our GPU quota (ie: 8 T4s in one region) on GCP and was denied and told to talk to a sales person. I literally said in my message to sales that AWS doesn't make it this hard to give them money. Every quota increase still seems to require escalation to our sales person and all they could offer as advice was to switch to invoice billing.
I literally had the opposite experience - a couple of years ago I was leading a data team designing a cloud-based solution and migrating the company from bare metal.
*AWS* - Lots of sales calls, but eventually redirected us to a "partner", who came up with basically a lift-and-shift.
*GCP* - The Regional Business Head was in direct contact. We were provided almost limitless access to a very helpful and experienced Customer Engineer - who spent _weeks_ with us, designing architecture, doing presentations, giving us learning material and eventually was instrumental in my team's increased capability and skill level to do the new infra design. Both came to visit the office several times. Eventually more senior engineers got involved.
We were also repeatedly given generous credit extensions. And best of all - there was a GCP summit in the country, during which I literally had a sit-down with PMs from a lot of GCP's product offerings - including Spanner, BigTable. 1:1 video calls with PMs from FileStore, Dataflow and BigQuery as well as early access to some key features.
Worth noting that we were hardly a "big" customer, or a huge money spender. Perhaps it was the technical challenge that interested them, or perhaps the reputation of the company/founder in the region. Whatever it was, I've had great experience with GCP.
I still don't understand why startups go through the hassle of using cloud, trying to get customer support for unknown and half-baked cloud products and spending enormous money - rather than focusing on building the ACTUAL product?
just go rent some VMs/bare-metal and focus on building your product, everything non-product related is just a waste of time, money, and effort.
it looks almost as if engineers forgot how to scale bare-metal applications (hint: still doable, pretty easy and damn cheap!)
they are free like when a heroine dealer will give your first two needles for free.
cloud offerings are massively overpriced and their goal is to hook you up to their ecosystem and then eat into your margins once your startup lifts off
I've been extremely happy with GCP's GKE offering as a solo dev. They've regularly upgraded it and improved it.
That said, the risks around customer support and occasional account termination worry me substantially. I'd be less worried if I had a corporate G Suite account with admin.corp@example.com with an invoice system and a corporate counsel on retainer to write grumpy letters as needed.
I've based my startup (www.groupsapp.online) on GCP and so far so good.. their free tier is really good. If you design your arch so that you only need lower-end instances your monthly bill becomes a joke. Datastore is ridiculously cheap too (but stay away from SQL, that can be expensive). 8 months ago I migrated to GCP from another cloud provider and found it more stable
Just another anecdote, we are a startup that got GCP credits last year which fully funded our first year, and it was an extremely simple process. So far we've been extremely happy with GCP (and Firebase), but in fairness haven't needed much tech support.
This is a good example of why I’m long Amazon and short Google.
Though Jeff Bezos stepping down worries me a bit.
Google doesn’t care that much about their customers. They mint money from their web ad monopoly. They don’t have aligned incentives, they don’t have a customer focused culture.
Just another Google Cloud vs AWS anecdote, but I have been trying to use Google’s cloud services to translate snippets of text here and there for personal use. I specifically went with Google because their translations are a cut above other translation services.
I recently switched over to translating with AWS because Google. Keeps. Breaking.
There’s a decided lack of focus on user experience in Google Cloud services, from invalidating my credentials, upgrading the tool chain (and invalidating my credentials), to the usability of the gcloud CLI tool itself. With AWS I was up and running in literal minutes.
Microsoft SQL Server on Cloud SQL is quite laughable in terms of feature sets, it currently doesn't support exporting your tables as JSON or CSV, you can only export as a .BAK file which is some MSFT proprietary trash and this bug https://issuetracker.google.com/u/0/issues/170026117 was open for months with no ETA for resolution.
I can confirm the same experience too (my company uses both AWS and Google Cloud) where most of the human touch points related to startup credits are with the sales team for Google Cloud instead of an account manager (we still don't have one after about a year now) if you manage to get pass their usual generic responses.
More anecdata - I did 500 startups in 2016 and used GCP credits, and did not have any problems getting them applied in a timely fashion. To be fair I was already a paying customer at that point (had a small VM or two running) but I did not get a sales up-sell or multi-week delay.
After your free credits will run out it will start being pretty expensive. Then it may be too late to migrate to a different provider due to excessive egress costs.
I like these type of posts that are self-admittedly biased but based on real world experience. Together with the comments in this thread they are very useful.
Everyone's needs are different. To me GCP is easier to get started with.
I don't want white glove service where they try to architect my architecture for me. Pushing me into cloud specific features (like everything in Lambdas stitched together with API Gateways). I don't need that level of help.
Also, of the three startups where I have heavily used AWS (personally "owned" the AWS infra), I never had that level of service offered. The only time I've had that "we'll help you architect it" service from AWS was at a startup that used GCP, and there were high level (CEO and CTO) discussions with AWS about partnerships. Then AWS wanted to help us architect (with Lambdas and API Gateways).
For me, GCP is way easier. Their products are more complete products. They work together out of the box. They are optimized for the 90% use case to just work. It is far easier to get a really good complete architecture working with GCP. Using their products push you to industry standard ways of writing and packaging applications (12-factor, Docker containers, etc.).
The startups I've seen build on AWS often have very bad setups. Hand rolled and poorly managed EC2 messes. Bad logging and monitoring because AWS doesn't have good logging and monitoring. Having to manage everything themselves because AWS doesn't/didn't have good managed platforms (Elastic Beanstalk has enough problems that people don't use it, and EKS barely counts as a managed service).
I've seen multiple startup with hand configured EC2 instances, running some generic web service. These systems create huge burdens and risks as the company grows. The app then grows and becomes locked into the special snowflake hand configured mess. One company I joined couldn't deploy for 6 months because their last deployment (a manual process) took 7 hours of downtime. When I see those systems I wish they started on Heroku or GCP App Engine (2nd gen). They would be way better off with a fully automated platform and 12-factor apps. Once you grow and hire people with infrastructure expertise you can take that clean 12-factor app and move it somewhere else. Non-experts trying to use AWS results in scary things.
GCP gives you that easy on-ramp. You can start with App Engine. If you need something a bit more custom, you can use Cloud Run (same basic tech as App Engine v2 but with Docker containers). If you need to move to something even more customizable for growing special needs, you can use GKE for a fully managed Kubernetes. With all those services you can write normal apps that use environment variables and log to stdout/stderr. All three give you logs in an easy to use logging platform. All three give you metrics in an easy to use monitoring/metrics/dashboard platform.
At every level GCP is easier. Cloud Pub/Sub is easier than SNS+SQS, Amazon Kinesis, or Amazon MSK (Kafka). It is multi-region and just works and scales. GCP VPCs and networking are easier because GCP VPCs are global (not regional), and subnets are regional (not zonal). No need to stitch VPCs together. GCP even has Shared VPCs so you can have a single address space shared across multiple projects (GCP projects == AW accounts). GCP authentication is simple and sane, even across multiple projects, no federation and role assumption needed. GCP IAM is far easier than AWS IAM. Cloud Datastore provides a really simple yet powerful NoSQL store, totally simple, no tunables, just works and scales, and even has some multi-region options available. Want a multi-region consistent database that spans large geographic regions? Amazon doesn't have an option, GCP has Spanner. Want a multi-region consistent blob/file store? Amazon doesn't have an option (S3 is single region), GCP offers GCS in multi-region, dual-region options that provide consistent multi-region file storage. Do you have some weird old app that writes to disk, and you would like it to be able to survive a zone failure? AWS EBS is single zone, GCP persistent disks can be regional, and because subnets are regional that means an instance in another zone can take over the disk and the IP.
For me, GCP is way easier to get started with. And will result in a better system in the medium term. In the long term, when you have thousands of engineers and experts in infrastructure engineering, it probably doesn't matter.
How does interacting with reps have anything to do with "onboarding"? This is more or less an evaluation of how easy it is to take advantage of credits.
I'd be more interested in how easy it is to use a credit card and get a service off the ground. What is the relative quality of the products on offer by either vendor?
TLDR >>> The initial `GCP` onboarding call was entirely about how much money I was planning on spending with google (as opposed to the Amazon call where they wanted to help me architect my service)
I had to evaluate the big 3 for a company I worked at a few years ago. We were a G-Suite shop, so GCP was already in the running, but folks were also interested in the Office 365 option (We still used AD for local auth and had file servers etc, so not fully Chromebook style G-Suite), so I decided to start with Azure as it was the one I had the least experience with.
Long story short, on my first day I couldn't get the console to log me out, even after explicitly logging out, etc. I thought I was going crazy or just doing something wrong, but I absolutely could not get Azure to log me out. I had to create a support ticket and it turned into an incident. It was honestly all a bit ridiculous. I wasn't going to veto Azure purely on that, though it obviously was not going in Azure's favor, I did have to explain my experience and it was effectively banned based on that experience, because we were a small/medium business subject to HIPAA and more senior folks didn't like the idea of us going bankrupt due to HIPAA violations from our data getting exfiltrated.
It ended up being a pretty straight forward choice between GCP and AWS. GCP was already easy for us, because of G-Suite, but also our primary product encapsulated a TensorFlow CV ML model and TPU training was very appealing from both a cost and speed perspective.
Expensive and not at all startup friendly. Most startups are doing containers or serverless and Azure makes you pay for straying away from their PaaS business model.
Not to mention the way Azure handles networking and security is atrocious, bolted on to Active Directory.
I don't know GCP at all, but I do know that AWS Serverless is brain-dead simple to implement and very low-cost even when you begin to scale.
personal opinion: absolute pain. everything is very complicated due to being connected to AAD and other services. lots of features but most of them useless. the legacy that they dragged along really is hindering.
Azure has amazing free benefits for startup. I formerly went through their Bizspark program which gave me a huge credit that let me have multiple VMs, relational database servers, etc for free FOR THREE YEARS.
I'm pretty tied to AWS at this point because that's who my employer uses but Azure was really great and their startup benefits were very generous.
I don't know, but I've tried it briefly and it felt pretty dog. In terms of UI, ease of use, quality of software. I don't feel in general MSFT is as strong engineering-wise either. In terms of customer service, the first email you get from signing up is from an AI, so there's that.
Pretty basic question: the author states in the very beginning: "I used to work at AWS". Is it possible that things went smoothly for him, because they still remembered him, and prioritized his requests because of personal relationship?
Given how often Google abandons products I don't know why anyone would build something important with their infrastructure. That alone should be a massive advantage to AWS.
Google has truly lost the ability to innovate in front of Amazon.
Its business is based on online advertising which has many problems: bot clicks, ad blockers, privacy issues and users don't like it.
The cloud business as we see from this article is not the best and will hardly take market share from AWS.
My company has gone through 4 reps in 3 years. Every time we get a new GCP rep they just want to talk to us about "expanding our use of GCP offerings". The only thing they want to talk about is starting to use BigQuery - not my business at all.
I signed up for a Google Cloud Security summit, and afterwards a sales rep reached out me. It was obvious from the start they had no idea I was a gsuite or GCP customer. They then directed me to a NEW account manager (#4) even though I had been working with a different one. I had worked with the prior account rep going over our architecture to make sure everything was kosher (sustained use discounts etc). I even made them a schematic of our architecture on GCP at their request. Once I provided that to them I was met with radio silence.
It's really insulting, and to me obvious they don't care about my company at all. We're looking at other options.
I wouldn't be surprised if somehow this post leads to me getting an email from another rep "Wanting to start over and doing things right", which will inevitably devolve into the discussion of how can I use BigQuery.