> A great example of how working people have more power than they think if they're willing to risk dollars and cents for matters of right and wrong.
I believe this is key. If more folks at more organizations were brave like this and willing to take the risk, a good chunk of the problems our civilization is facing might be greatly improved.
> Worker's unions helped win the majority of our rights in modern democracies. I wish this fact was more widely appreciated.
The problem with modern unions, particularly in tech, is that the legacy structure is inapt for current problems. Tech workers don't need a union to negotiate compensation, they're compensated fine already. They don't need a huge bureaucratic structure for engaging in long-term detailed negotiations. They don't need a contract at all.
What they need is a no-dues no-fulltime-union-reps union that operates through direct democracy. It does nothing unless the employer is doing something bad wrong. Then if the majority of the union members vote to refuse, either the employer concedes or they strike.
Because it's not about a thousand little things here, it's about a small number of big things. It needs to be able to address those and then go back to being invisible instead of succumbing to feature creep and destroying the host with overhead and principal-agent problems as we've seen with the auto makers.
> Tech workers don't need a union to negotiate compensation, they're compensated fine already.
Software one of the highest margins of any industry. They can afford to pay more, especially since they are constantly whining about "shortages" of tech workers.
Which is why they already do pay more than other industries.
The best argument you have against that is the anti-poaching shenanigans they've engaged in -- but that's already illegal, so the answer there is a courtroom rather than a union.
No, it really isn't. The court requires someone to notice the pattern, or be aware of the pattern, and be willing to risk their reputation. With a union, the onus is on the business to act right, or risk labour action where the SRE folks walk out, and all the little blinky lights turn off.
> The court requires someone to notice the pattern, or be aware of the pattern
How is that different with a union?
> and be willing to risk their reputation
Class action suit or submit evidence confidentially to the attorney general.
> With a union, the onus is on the business to act right, or risk labour action where the SRE folks walk out, and all the little blinky lights turn off.
If Apple won't hire Google employees then the Google employees can retaliate against Apple by not working for them?
I disagree, given the massive cash reserves the tech companies have.
Yes, having a higher salary would be ridiculous in a lot of these cases, but we should moderate that through legislation that benefits the most people - not by a public company further lining the coffers of its owners.
Apple and Google particularly have a lot of cash just lying around, and that cash is the result of the employee's efforts, and they deserve it. I think if we think their salaries are too high in that case, we need to talk about better taxation systems.
> I disagree, given the massive cash reserves the tech companies have.
They have massive cash reverses because the tax laws have encouraged that rather than paying it to shareholders as dividends. And that level of return is necessary because of the nature of the industry -- you have to spend millions of dollars trying to create the next tech giant before you know whether you've succeeded or not, and most of the time you haven't. The returns to success have to be enough to overcome the high failure rate.
Most of the employees aren't taking the same level of risk. If you work for a company for five years taking home a six figure salary and that company fails, you don't have to give back your salary and in a few months you're working for another company making the same amount of money.
If you think you can do better on your own, risking your own time and money instead of taking outside investment, go right ahead -- but then shouldn't it be you who gets more of the reward if you succeed rather than the people you hire in after you're already an established success?
On the one hand it sounds like you're saying that software engineers are paid enough already, then on the other hand you're saying you think the compensation given to the software engineers that founded the company - which is much MUCH higher than that of the average company engineer is appropriate.
It feels like what you're saying is that the risk of failing in a startup is massive enough for a founder that they deserve literally billions of dollars.
Could you let me know exactly what risks you think a failing startup founder faces that would entitle them to say, a thousand times more dollars than the average salaried employee? Are you saying that because a founder may go bankrupt they are entitled to thousands of times more money? Does this mean that any individual that takes out a loan larger than their assets to start a business is entitled to thousands of times more money than their average employee? Could you help me understand what makes you think that?
> On the one hand it sounds like you're saying that software engineers are paid enough already, then on the other hand you're saying you think the compensation given to the software engineers that founded the company - which is much MUCH higher than that of the average company engineer is appropriate.
Of course, because the level of risk is different. $100,000 guaranteed is worth more than a <50% chance at $200,000, much less a <1% chance. A very high reward is inherently necessary to offset the very low probability of major success, otherwise people aren't going to do it.
> Could you let me know exactly what risks you think a failing startup founder faces that would entitle them to say, a thousand times more dollars than the average salaried employee?
The less than one in a thousand chance of making that much.
> Does this mean that any individual that takes out a loan larger than their assets to start a business is entitled to thousands of times more money than their average employee?
There are many ways to turn a thousand dollars into a 0.1% chance at a million dollars. Then 99.9% of the time you lose the thousand dollars -- and it's your time/money, not the bank's. Nobody is going to give you an unsecured loan to gamble with.
But if you bet on your own horse at 1000:1 odds and win, how are you not entitled to the proceeds?
I'm assuming that you don't think risking making no money is enough to entitle a founder to their entire employees wage. How much does it entitle them to?
> I'm assuming that you don't think risking making no money is enough to entitle a founder to their entire employees wage. How much does it entitle them to?
The amount they mutually agree upon. The employee wouldn't agree to work indefinitely for no pay.
The high compensation of successful founders is actually one of the things keeping salaries up, because any of the salaried employees has the option to quit and found their own company. The existing company has to pay well enough to compete with that -- because if what they're paying wasn't actually competitive with that alternative given the relative risk between them, why would anybody accept the salary?
1) The collusion has presumably stopped now that they're caught.
2) It is possible for both to be true at the same time, because the industry is much larger than Apple, Google, Intel and Adobe. Even if they didn't compete with each other, they still have to outbid Facebook, Microsoft, Amazon, etc. -- and pay enough to prevent the workers leaving to found their own companies. It's not unreasonable to expect that the effect on wages was marginal even when it was occurring.
The successful entrepreneur makes his money by capital gains in the share or venture capital markets and not by extracting it from his employees. Employees compete with each other for salaries. Employer competes with other employers for both a)market share b) hiring employees -bidding up their prices.
By comparing gains from entrepreneurship with regular salaries, you are comparing a stock variable with a flow variable. Even Marx got this part correct
> The successful entrepreneur makes his money by capital gains in the share or venture capital markets and not by extracting it from his employees.
Incorrect, that value is only sustained and increased by the efforts of company workers.
> Employer competes with other employers for both a)market share b) hiring employees -bidding up their prices.
Incorrect. Companies that don't have significant oversight in the form of government regulation or strong unions tend to collude to keep salaries low - which is exactly what has happened in the valley, and has meant that these companies have gigantic cash reserves that they aren't leveraging to hire the best talent.
> By comparing gains from entrepreneurship with regular salaries, you are comparing a stock variable with a flow variable.
No, I'm merely saying that the differences and risks suffered by investors and founders versus regular salaried employees are not a justification for the sometimes ridiculous difference between the compensation of the two.
If tech workers form a union, I'm sure it will look very different to the industrial worker's unions of the 20th century. And so it should, the needs of today are very different. The thing is, apart from the remaining unions from that time, most unions already look very different to that model so this is not really a good argument against unionising.
The other thing you aren't taking into account is the fact that this boom in the tech industry isn't guaranteed to continue forever. There will come a time, maybe pretty soon, where tech workers will become as precarious as those steel workers and autoworkers eventually became. Big tech companies are already putting a lot of effort and resources into educating the next generation of programmers to provide a more competitive labour market and drive down salaries. There's already talk of a coming recession, where I'm sure the belts will be tightened and people will be laid off. When we have a union, we will be more protected from the inevitable exploitation in such scenarios.
The temporarily embarassed unicorn founders among us need to realise that we are the creators of all the value in these companies and, collectively, we have the power to influence their direction and impact on society. We can help secure not only our own rights as workers but also have the power to change society at large and secure better standards of living for all workers (or non-workers). That's why these recent actions by Google employees have been so important. They can set a precedent for how other companies and even states can safely act in future, without fearing repercussions from their most valuable resource - the workers.
How do you get and pay for the infrastructure of this direct democracy without resources paid for by dues? How do you get the minority in any vote to go along with the result when there isn't any common binding agreement such as a contract that enforces majority rule?
> How do you get and pay for the infrastructure of this direct democracy without resources paid for by dues?
The technology needed to let people submit proposals and let other people vote on them is on the level what individuals do over a weekend as adjunct to a side project.
> How do you get the minority in any vote to go along with the result when there isn't any common binding agreement such as a contract that enforces majority rule?
Why do you need to force them to? By definition the majority will already agree, and then many in the minority would participate out of solidarity because that's the whole point of joining a union to begin with. You don't need 100.0%, a large majority is quite sufficient in general. And anything that actually required 100.0% is already lost, because then they could pay off the cheapest defector or contract it out.
The technology to submit and receive votes on proposals is only trivial until you think about the details, especially those required for security and authentication.
And your picture of humam behavior is all too rose-colored glasses if it's having all members of a minority vote just go along out of solidarity when it's non-binding. I've seen unions vote on issues, and it's often contentious with emotions running high on all sides. If the losing side in any of those could have just said "nope" to accepting the result, they would have. Sometimes they try to anyway.
> The technology to submit and receive votes on proposals is only trivial until you think about the details, especially those required for security and authentication.
This is a major problem for country-level populations. For corporations it typically comes pre-solved by the corporation itself, because each employee would have a company email address or Active Directory account etc. that could be used for authentication. (In theory the corporation could illegally tamper with the results that way, but the tampering would be immediately obvious to the person whose vote was changed.)
> If the losing side in any of those could have just said "nope" to accepting the result, they would have.
Because they're using the union for the wrong stuff. A lot of the votes would be for things like accepting a policy that gives raises to only senior people. No doubt the junior people being screwed over by that policy would strenuously object when they're the 49%, especially when being in the union deprives them of the opportunity to negotiate something else as an individual.
But how many Google employees have that kind of personal stake in a question like whether to censor search results in China?
Unions are rarely formed unless conditions are particularly bad. One upon a time in this country, the national guard with machine guns might have been called out to clear a strike/protest. Most people are very happy being ignorant of their surroundings or influence of.
The most I've ever done is threaten to quit if a project for the RIAA was accepted by my employer when I was invited into the pre-pitch meeting. It just depends on a specific case.
I'm part of a very privileged workforce and our situation, while not great, was a lot better than the average worker. We still managed to form a union. It can be done.
I wish more people knew that workers fought and died for those rights.
Its one thing to say maybe you'll quit, or skip your pay check for change, its another to actually put your life on the line for what you know is right.
4. Boycott silicon valley startups that have accepted chinese investments?
5. Boycott every product by US/Foreign company (Apple, Amazon, Microsoft, Google, Samsung, GE, Disney, Chevron, Exxon, Shell, etc.) doing business in China?
There's a difference between boycotting products made in China and protesting against building technologies that enable dictatorial regimes. They do not deserve to be conflated.
They're not "enabling" a dictatorial regime, they're just following the laws of a foreign government. Making a censored search engine isn't "supporting" the government, it's just abiding by it.
A censored search engine is a direct instrument of oppression.
Contracting to build a censored search engine is the moral equivalent of contracting to build a barbed wire fence around a concentration camp.
You're being an active participant that directly assists in making that oppression happen by building tools required for the actual act of oppression, as opposed to building something that merely done in the same country.
There are over a billion innocent people in China. The products themselves are mostly innocent and the people who build them make their livelihoods manufacturing them. That the government takes a cut is incidental. It's simply not comparable to a product specifically built to support dictatorship.
Over 5 trillion in US tax money has been used to prosecute unjust war in the middle East resulting in 100s of thousands of deaths. Is boycotting Disney movies that support the government through taxes the same as boycotting predator drones?
"You're not allowed to do anything good ever if you're not already a maximally good person. That would make you a hypocrite, which is way worse than someone who doesn't do anything to help in the first place."
I don't understand the context in which you're posting this quote. pompousprick's argument was that we shouldn't morally prosecute these companies as supporting dictatorships just because they do business in China.
Bargaining power. (Organized) working humans have it...but it will erode as robots and algorithms take on more and more work. An argument for striking while the iron is hot. (No pun intended.)
Historically, whenever there's advancement in technology. Powers that control the current will get challenged, and so far, it has been for the better for the majority.
We've gone a long way from being serfs who have almost no rights, even the right to read and write.. to citizens who can communicate via the internet.
Advancement in technology will/have make current powers obsolete. Robots and algorithms have been taking more work since the stone age.
Better tools, means more surplus, means more time to think critically, more time to comment over the internet, or read books.
The only problem is that we probably won't see any observable improvement in one lifetime. It is however, also very possible that the incumbent powers put an end to our civilization. Hopefully enough people like these Google employees, Snowden, Assange will be there to stop fascism.
I think technology has allowed power to become more and more concentrated. Never before has one state had the power to annihilate entire cities, or land entire armies worth of troops at any point on earth within 24 hours. Never before have large and powerful organisations and governments had the capability to read every word that every person sends to each other and track their every motion around the city with cameras.
Yes there is more surplus, but where is that going? People in developed societies are working more hours now than they were 100 years ago. Wealth inequality has risen to higher levels than ever existed in modern society. I wish it gave me more time to read books...
We definitely need to be more engaged in resisting these dangerous tendencies, particularly with recent political developments. I think organised tech workers have immense power. If Google employees could formalise their current actions and then even unite with other groups across other companies, they would be a force to be reckoned with.
I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly:
1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN).
2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.
3) The sum total of information about this incident can be found as a few one or two sentence blurbs on Google's blog. No explanation nor outline of scope for affected regions and services has been provided.
4) Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems.
5) Some users here are reporting that they have received no response from GCP support, even over a time span of 40+ hours since the support request was submitted.
6) Google says they'll provide some information when the next business day rolls around, roughly 4 days after the start of the problem.
I really do want to make sure I'm understanding this situation. Please do correct me if I got something wrong in this summary.
When everything works, GCP is the best. Stable, fast, simple, reliable.
When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution.
They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a hand-off to another team in a different time zone. We have spent days trying to convince them there was an issue before, which just seems unacceptable.
I can understand support costs but there should be a test (with all vendors) where I can officially certify that I know what I'm talking about and don't need to go through the "prove its actually a problem" phase every time.
As someone who works for Government and Enterprise - all I care about sometimes is how a company behaves when everything goes wrong.
The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation.
Being able to tell the Directors/Senior managers that issues have been "escalated" and providing regular updates are critical.
If all I could say was a "support ticket" was logged, and we are waiting on a reply (hours later) - I guarantee the conversation after the outage is going to be about moving to another solution provider with strong SLAs.
Very similar thing at our office. Considering the scale of which we run things, any outage could be a potential loss of millions _every minute_.
Sure, we use support tickets with vendors for small things. Console button bugging out, etc. But for large incidents, every vendor has a representative within an hour driving distance and will be called into a room with our engineers to fix the problem. This kind of outage, with zero communication, means the dropping of a contract.
Communication is critical for trust, especially if we're running a business off it.
Going single cloud on that scale is simply irresponsible though.
You need failovers to different providers and hopefully also have your hardware for general workloads
And suddenly the CEO doesn't care anymore if one of your potential failovers is behaving flaky in specific circumstances
Not saying it's good as it is.. communication as a saas provider is - as you said- one is the most important things... But this specific issue was not as bad as some people insinuate in this thread
You are incorrect about aws. If your pay for business support, and something is happening to your production environment, they are on a call with you in less than an hour.
"Support costs" calculation often doesn't include the costs of not having support.
When I worked at GoDaddy, there were around 2/3 of the company was customer support.
At the current company I'm at, a cryptocurrency exchange, our support agents frequently hear they prefer our service over others because of our fast support response times (crypto exchanges are notorious for really poor support).
All of my interactions with Amazon support have been resolved to my satisfaction within 10 minutes or less.
Companies really ought to do the math on the value that comes from providing fast, timely, and easy (don't have to fight with them) customer support.
They have though; they've just drawn the conclusion that they'd rather put massive amounts of effort in to building services that users can use without needing support. This approach works well once the problems have been ironed out, but it's horrible until that's the case. Google's mature products like Ads, Docs, GMail, etc are amazing. Their new products ... aren't.
I've seen Google projects where the project lead explained (actually responded) that they don't want to provide support, end of story. Google puts folks in charge but does not give them enough in the way accountability objectives.
With Dell you can certify with them so you can get replacement parts and such without the BS back and forth with some guy in india. Saves everyone time and money.
We actually got to a point where we had a couple of spare parts onsite (sticks of RAM, HD, etc) and so we repair immediately and then request the replacement. This was on a large HPC cluster so we had almost daily failures of some kind (most commonly we'd get a stick of RAM that would fail ECC checks repeatedly).
> instead they keep sending tickets back for "more info"
Isn't that the case with basically every support request, no matter the company or severity? The first couple of emails from 1st & even 2nd level support are mostly about answering the same questions about the environment over and over again. We've had this ping-pong situation with production outages (which we eventually analysed and worked around by ourselves) and fairly small issues like requesting more information of an undocumented behavior which didn't even effect us much. No matter how important or urgent the initial issue was, eventually most requests end up being closed unresolved.
I've definitely had interactions with smaller companies where you can effectively bypass first and second line by demonstrating you know what you're doing, mostly just saying the right things for them to accept that you've done basic troubleshooting steps already and really do need to talk to someone beyond that point.
Yes, same experience here, support at smaller companies can be more dedicated when talking to "knowledgeable" customers. It's generally easier to get to their 3rd level, sometimes just because there is no 1st or 2nd level at all. But at "big" enterprises - not so much.
> I've definitely had interactions with smaller companies where you can effectively bypass first and second line by demonstrating you know what you're doing, mostly just saying the right things for them to accept that you've done basic troubleshooting steps already and really do need to talk to someone beyond that point.
Smaller companies or personalized support structures (like named engineers) is very different. You can build up a relationship and usually bypass many questions to get to main issue, and even get it resolved before you can even open a case at larger organizations.
GCP does have role-based support models with a flat-rate plan, which is really great, but the overall quality of the responses leaves much to be desired.
We had an issue a few weeks back where all nodes in west1-a could not pull docker images. Google support was pinballing P1 issue around the globe and across multiple teams for a few days untill I root caused it for them - turned out to be gce service account issues affecting entire zone. 2 days to rollback (no status page update). I know nobody gives a fuck but can’t help but feel vindicated as an ex google sre.
I think a lot of people give a fuck here; I do, at least. Thanks for outlining it, these things are fascinating (to me anyway, who has never worked in IT/ops).
We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform.
So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had rampant impact on other services.
But i find it concerning that they provided very little update on what went wrong. I also think its better to expect nil support out of any big cloud provider if you don't have paid support. Funny how all these big cloud providers think you are not eligible for support de-facto. Sigh.
I use AWS free tier and get customer support through email, but thats not the case with GCP. Do they provide free email support?
If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.
You know there's 1-man "start ups"/companies out there that serve more users than 99.5% of VC startups ever will, which earn their owner a very livable wage, but who still can't afford a 500 bucks business support plan on all of the 20 services they use.
If you've got VC money to blow so you can pretend your SaaS toy can feed 500 people while having money left to throw at things, that's cool. Just remember that other people might be running sustainable businesses.
There are literally millions of small 1-2 people business that make less than 10k in profits.
I mean sure, they could go and probably afford to waste $200 extra on something random that will be useless to them most of the time, but that money is going straight out of their paycheck.
You don't remain profitable though by repeatedly making bad decisions like that. Which was my point.
Running a (small) profitable business is about making the right decisions consistently, and if you're likely to waste money on one thing, you're also likely to waste it on the 19 other similar things.
Maybe speak to literally anyone you know who is running a small businesses if you want to know more. Yes that includes your local small stores on your street.
At the end of the day you probably pissed off quite a few people on here when you called their livelihood a hobby project.
The scenario being discussed was building a company on top of GCP and being unable to afford $200/month for support costs. Tell me what “mom and pop” shops are pulling in $10k/month from GCP-based software but are unwilling/unable to pay $200/month for the very thing that underpins their livelihoods?
This is akin to saying that a mom and pop laundromat can’t afford insurance, or shouldn’t because they won’t frequently need it.
You’re trying to equate small businesses with hobbies. You’ve now resorted to straw men, slippery slopes, and false equivalency. Maybe consider that if you have to distort the situation this much to make your point, you might just be wrong.
> At the end of the day you probably pissed off quite a few people on here when you called their livelihood a hobby project.
I didn’t say anything about anyone’s livelihood. You’re the one pretending that small businesses bringing home $120k/year can’t afford a $200 monthly support bill.
I bet the guy who started this thread about GCP’s support cost has made a sum total of <$1000 from his “startup”. Likely <$10. Hobby.
I don’t care if “quite a few people” got pissed about my comment. People with egos that delicate shouldn’t use social media.
> The scenario being discussed was building a company on top of GCP and being unable to afford $200/month for support costs. Tell me what “mom and pop” shops are pulling in $10k/month from GCP-based software but are unwilling/unable to pay $200/month for the very thing that underpins their livelihoods?
I was trying to tell you that most small businesses can't go around spending hundreds of bucks of things that provide little value, whether that's a business support plans on services they use or something else. It's true regardless of whether you're a brick and mortar store or some online service.
> This is akin to saying that a mom and pop laundromat can’t afford insurance, or shouldn’t because they won’t frequently need it.
Speaking about about false equivalencies...
> You’re the one pretending that small businesses bringing home $120k/year can’t afford a $200 monthly support bill.
First off, I spoke of businesses making generally less than that.
Also (I already said this, good job ignoring that!) paying $200 bucks on a single useless thing is survivable for even a small business - but you know what's better than only making one bad business decision? Making no bad ones at all. Making too many will quickly break the camel's back.
Which was my whole argument and it's also what people generally refer to when they say they can't afford something.
For instance you may say "I can't afford to go to this restaurant", even though you'd have enough money to do it without going immediately bankrupt. But it'd be a bad decision, too many of which quickly add up.
> I was trying to tell you that most small businesses can't go around spending hundreds of bucks of things that provide little value
And I'm telling you that if you built your business on top of GCP, a support contract is probably not "low value". You'd happily pay $200 for support on your critical infrastructure, just as you'd happily pay $200 for a repairman to fix your washing machine if you owned a laundromat.
If you don't need support, then sure, don't pay for the plan. If you do need support, $200 seems pretty reasonable.
> Speaking about about false equivalencies...
Signing up for a monthly recurring support plan in case you need it is literally insuring your business.
> For instance you may say "I can't afford to go to this restaurant", even though you'd have enough money to do it without going immediately bankrupt. But it'd be a bad decision, too many of which quickly add up.
A support plan for your critical infrastructure probably isn't "useless". Which is the point. If your need for support is that low, then either you've built your own redundant systems to protect you or more likely you aren't running a real business.
No, I understand that most hobby projects are just hobbies and Google/Amazon/etc are under no obligation to provide support for hobbies that are literally a net cost to them.
I'm glad AWS's free tier is working for you, but complaining that Google doesn't want to give you free capacity for your business and then also provide you free support for that business is pretty absurd.
I don't understand why someone would choose to deploy anything mission critical without having an support contract with the ISP, the manufacturer of the the software etc.
Simple, the cost of an outage is less than the cost of a support contract. Very few things are really mission critical as in they can never go down. Rather they simply have a cost to going down and you can choose to pay that one way or another.
I transitioned from collocation to self managed remote server farm and then onto self managed remote vms. All these providers provided de-facto support whether we opted for one or not. You can go to their portal and raise a ticket.
I am not saying with vast numbers its feasible but big cloud providers don't even give you the opportunity to raise a ticket if its their fault. There is a price you pay extra when you opt for any one of them but many don't realize. Having said that - almost all the time, our skilled expertise is better than their initial two level of support staff. We realized it early so we handle it better by going over the documentation and making our code resilient since all cloud platforms have some limit or another since overselling in a region is something they can't avoid. Going multiple regions across when you handle these exceptions is the only way through.
You are right about the Google blog content itself not indicating three days of outage. Turns out they just forgot to mark that particular issue as resolved on Friday, as you point out. This is my mistake. I would update my comment to reflect this, but it doesn't seem to allow an edit at this point.
The items I put down in my comment are based largely on user reports, though (there isn't much else to go on). And I mean these items as questions (i.e. "is this accurate?"). Folks here on HN have definitely been reporting ongoing problems and seem to be suggesting that they are not resolved and are actually larger in scope than the Google blog post addressed.
Someone from Google commented here a few hours ago indicating Google was looking into it. And other folks here are reporting that they don't have the same problems. So it's kind of an open question what's going on.
I'm in the evaluation phase too. And I've found a lot to like about GCP. I'm hoping the problems are understandable.
I've been failing all weekend to create nodes in a GKE cluster through either the UI console or gcloud. Even right now I can't get any nodes to spin up.
Edit: I finally got my cluster up and running by removing all nodes, letting it process for a few minutes, then adding new nodes.
As of this morning, I am still unable to reliably start my docker+machine autoscaling instances. In all cases the error is "Error: The zone <my project> does not have enough resources available to fulfill the request"
An instance in us-central1-a has refused to start since last Thursday or Friday.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
I can't comment regarding GKE as we don't use that particular service, however we are very heavy users of many other GCP services, including Compute, Datastore, BigQuery, Pub/Sub, Storage, Functions, Speech, and others. Zero issues this weekend, everything is running 100% as any normal day.
> For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement)
What blog statement are you referring to? I don't see any such statement. Can you provide a link?
The OP incident status issue says "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI". It also says "Affected customers can use gcloud command in order to create new Node Pools."
So it sounds like a web interface problem, not a severely limiting, backend systems problem with global scope.
Also, the report says "The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific". So the whole issue lasted about 10 hours, not three whole days.
> Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems
Right now we don't know. It's one of two possibilities from what I can tell:
a) Google had a global service disruption that impacted Kubernetes node pool creation and possible other services since Friday. They had a largely separate issue for a web UI disruption (what this thread links to) which they forgot to close on Friday. They still have not provided any issue tracker for the service distribution and it's possibly they only learned about it from this hacker news thread.
b) People are having various unrelated issues with services that they're mis-attributing to a global service disruption.
This is why GCP has no hope of ever taking significant market share from AWS. Google thinks they can treat their cloud customers like they treat users of their free services. Customer support and communication are essential.
Yes. When the AWS status page failed to accurately inform their customers for several hours, AWS used Twitter to ensure that there was communication with their customers.
I recently removed my hosting from GCP. The pricing is confusing and unbelievable. Their customer service is a joke. I don't trust Google for longterm consistency due to the way they shut their own apps but I let that slide as I doubt they will do that on their cloud services. I have experience with AWS (rock solid, world class support but also costly), digital ocean (improving fast), heroku (good for beginners but also expensive and not as full featured as AWS) and finally Hetzner (too early to judge).
“2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.”
Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are fucked globally on aws. Feel free to pay double - but it seems* if you are paying that much just pay for an additional cloud provider. Looks like it’s the same deal on GCP.
On 17 October, there was a multi-AZ network failure at us-east-1. It only lasted 3m35s, but it was enough that our customers were calling about our site being down.
Just grabbed first article. Example: In this case capitalone went down. I don’t work at capitalone - but I imagine they had their data copied across every region 30 times.
I have never understood why employees accept the restrictive clauses which assign ownership for any side projects to the employer. I am not a lawyer, but this has always struck me as amounting to a type of serfdom. If you are seen as a 24/7 unit of the company, and anything at all that you creatively produce can be claimed by the company, then your working capacity and creative capacity is essentially owned entirely by the company while you are employed there. You are not being paid just for your time and the work product you produce during that time. Rather you are literally selling an aspect of yourself. Does that sound reasonable at all? (of course, California has some protections against this. but that's just one state)
I would really like to see people rallying against many more things like this.
Maybe for a lot of the higher end HN devs out there, they can walk and be reasonably certain they'll pick up work in under a month. Personally, I've known a LOT of people (some in software too) that need 6-9 months to find any work in their field. Yeah, Uber and pizza delivery make some ends meet, but for a 'real' job with a 401k and benefits, it can take a LONG time. And at the end of that timeline, they can offer you really anything and you know you have to take that offer.
Anecdata: I'm in biotech and got offered 55k on the Peninsula at the end of about 2 months of interviewing for that particular company. They were the only people that would interview me over ~9 months of applying (caveat: biotech isn't doing well right now). The minimum wage of my 'stop-gap' auto mechanic job in a particular city is 60k. The biotech company would not budge at all.
Many people aren't even aware of such draconian policies they are signing up for. And companies are probably counting on it too.
It is also possible that even the owners of the company (especially small, family owned companies) aren't aware of these dumb policies. They just trust their lawyers to write up the contracts. There was a comment recently on HN - employee reads up the contract, goes to the owner to ask about some clause, and the owner himself is surprised and calls the lawyer to sort it out.
Big companies that spend shit ton of money on lawyers have no excuse - they're likely doing all this intentionally.
I had one success simply not signing a non-compete I was handed on orientation day. I returned the packet of docs without that one and never heard a thing about it.
My understanding (from working with lawyers in the past) is that it is enforceable in some states and unenforceable in others. I'm not a lawyer, though, and so don't really know first-hand.
I use DuckDuckGo as my default. The only time I ever have to revert to Google search is when I'm searching for things indexed within the past year. This is a pretty crucial feature when searching for technical items (ensures a good number of search results while also being relatively current). For some reason, DDG only lets you search up to the last month. If they added the ability to search within the past year, I probably wouldn't need to use Google search at all.
I also think there is a need to figure out how to enable community-controlled SaaS platforms in addition to this. LibreOffice, for example, has essentially released an online office suite. But it has decided not to actually operate and offer this suite to the public in ready-to-use fashion. From what I understand, it is just too difficult and resource-intensive to do that. So it's up to companies running paid platforms to do it (or you can spin it up on your own server/instance and run it yourself). There's nothing inherently wrong with this. But it seems to me that there's a "next step" to take by figuring out how to enable fully community-controlled platforms so that a project like an open source G Suite (running at scale with an iron-clad privacy guarantee that is backed by community audits) that you can just go create an account on could become a reality one day. This seems like it would require a non-profit organization akin to Mozilla. How great would it be to have a community-controlled non-profit organization operating a trusted cloud platform, perhaps even audited by a group like the EFF? Very challenging, for sure. But it seems humanly possible.
I still don't quite understand why Docker Swarm is so overlooked. It essentially offers easy-to-manage orchestration while providing most of the core features that many people are looking for. It even makes use of compose files, which many developers are already writing. Given that most of the complaints about Kubernetes relate to its complexity and the need for a Kubernetes "expert" to look after said complexity, I'm somewhat baffled by how many people just totally overlook Swarm as if it wasn't an option.
Docker staff have been saying words to the effect of "we'll make an announcement with info soon!" for at least the last three+ months, across several projects. Swarm, Docker-Machine, & Infrakit come to mind.
With no actual announcement(s) then occurring.
The commit activity graph on InfraKit - which people are supposed to use now that Docker-Machine has been deprecated - seems pretty clear:
I guess you are right. It's unfortunate because judging from the comments here you would think that there's a market for a simpler orchestration software while it will be hard to compete as just another Kubernetes distributor (even more so now that there are several alternatives for the container runtime). But hey, what do I know about product management?! :)
Nomad is that. We run Nomad quite successfully at SeatGeek, and I know of several Very Large Companies™ doing so at scale as well. One of them even has a fruit logo.
We are running the OSS version of Consul, Nomad, and Vault. While the enterprise versions provide some items that could be interesting, they are not necessary given our workflow, and I'd bet that the majority of development shops out there wouldn't need them either (though I am certainly more than happy to play with them!).
I strongly disagree with your second point. There are quite a few very successful entrepreneurs who have started companies while being mired in terrible financial circumstances. In fact, it's very common for these circumstances to be the stimulus that gets people moving in the direction of their own company. There are many, many stories of this, to the point that it's a cliché.
I myself started my first business at a time when I had virtually no money in the bank, and my car was literally repossessed one month before my business became profitable. If I had stopped and said "I can't afford this" and just given up and gone to get a job, I would have never seen the success that came later.
Also, you point out particular entrepreneurs who came from rich families. You completely leave out the wide array of entrepreneurs who did not. John Paul DeJoria, for example, was living on a friend's property because he was homeless and had almost no money. He started his business in those circumstances. He's estimated at somewhere in the ballpark of 3 billion net worth presently. There are quite a few stories of people with virtually no money building empires out of the rubble.
While I agree that there are a number of factors that feed into whether a startup is successful or not, I would kindly submit to you that one's attitude towards whether or not it can be accomplished is a very massive part of the picture. It takes focus and determination to get most businesses off the ground. Once you begin listing the reasons you can't accomplish it, you've begun setting up the circumstances such that you won't.
For me (formerly well-paid executive at IT companies in the US, originally from Italy), #2 (plus, some reserve money) was absolutely necessary to convince myself to leave my last job and start a company, last year.
>>I strongly disagree with your second point. There are quite a few very successful entrepreneurs who have started companies while being mired in terrible financial circumstances
Textbook example of self-selection bias and survivorship bias.
Even if you compiled a list, it would tell you nothing about the number of people who would have started companies had they not been mired in terrible financial circumstances.
If you compiled a list of all successful entrepreneurs and it was overwhelmingly full of ones who were poor when they started, it would suggest that such circumstances raise the chance of being able to succeed as an entrepreneur. More analysis of different stats would have to be done to elevate that beyond just a suggestion. Personally I feel that at least to some degree it could be the case. (Necessity is the mother of invention, and limitation actually helps creativity far more than abundance) I didn't grow up poor necessarily, though there were times growing up that things were limited quite a bit. I do believe these times are disproportionately responsible for my abilities today, as well as happiness.
I believe this is key. If more folks at more organizations were brave like this and willing to take the risk, a good chunk of the problems our civilization is facing might be greatly improved.