The majority of the team is in SF for Google Cloud Next to present this. I'm about to hop into the car to head up myself to watch, so I can do a little AMA throughout the day. Reply to this to make sure I see your question!
Here's a question I've already seen:
> Is this a Google project or a Go project?
It is a project led by the Go team, and we are very aware of our responsibility to be cloud provider neutral. At the same time, while the project is so new and evolving so rapidly, we need to have working examples that we can change atomically in a single repository (atomic changes necessary for rollbacks and such). We don't want to have cloud provider-specific code in the Go project, so the code is currently sitting under the Google GitHub org. When the interfaces to different products stabilize, the intention is to graduate them up to the Go GitHub org. Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured. The cloud provider-specific code will always live outside the Go project.
I'm a little bit concerned because the last package that Google seems to have purpose-built for Go+Google (AppEngine?) was context, that gave us things like context.Context, a generic KVS that needs reflection, heavily relies on convention and doesn't work in linear time and "just make every function take a context and use context.TODO if you don't need it".
It also seems to be one of the biggest points of controversy for Go 2, with calls from "everything should have a context" to "context should go away".
The libraries that support it have now all their function signature way longer, and it is not well understood if it is a "Request for cancellation" or something else
Context is a pet peeve of mine. It's a design mistake as implemented, even if it serves an important function. It infects everything, forcing every method to pass contexts around, and it conflates multiple separate concerns (cancellation, deadlines, and key/value data).
Given that all of a Go program sooner or later will need contexts somewhere, it would be much cleaner to let goroutines have this stuff implicitly. I don't know enough about the intervals to say whether this is feasible with the current runtime, though.
After Google threw half of Android into "Google Play Services" I don't believe it.
> Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured
Of course they can, but you're working on a Google-centric version first right? Otherwise why would it be at Google Cloud Next?
That makes all other cloud providers second tier, and I don't think you should call yourselves "platform neutral". If you were, you would code up multiple platforms from the beginning, like is normally done for support on different operating systems for example
(Eng Manager for Go Cloud) The APIs and HTTP server released today all work on AWS as well as GCP and support for Azure is a top priority but didn't make it for today.
Is this compatible with the Go runtime of Google App Engine[0] currently? I was having difficulty getting the golang Google Cloud Storage client to work on App Engine several months ago.
Hi, Tech Lead for Go Cloud here! Glad that you're excited to contribute! We don't have a tag yet since we're just starting out. The best way you can contribute right now is to use Go Cloud for yourself and file any issues that you find.
> we are very aware of our responsibility to be cloud provider neutral
Yet, Azure support is listed as unplanned.
The language could mean that this is not going to be planned. The fact that y'all have not worked with Azure Go folks to get this moving is something to ponder.
A reminder that Go is a Google language and isn't vendor neutral?
PM for Go Cloud here (and poster of the Azure feature request).
"Unplanned" on the issue tracker just means we haven't assigned it to a sprint milestone for the team yet.
Azure and more clouds are very much in the works. As an open project, there are many approaches and we're open to all of them.
One other point: since Go interfaces are implicitly fulfilled, anyone can write and share implementations of Go Cloud for new providers or custom services without asking anyone for permission. If you take a look at the implementations we've provided, you'll find it's actually rather easy.
Hi PM guy. If project is to remain independently cloud-gnostic and people file a lot of bugs and pull requests it'll definitely help Google or Alphabet or whatever deliver that k8s abstraction y'all've workin' on. That's great for you.
My my question is, what if developers want access to innovations in specific cloud providers, such as reduced redundancy storage on S3? Wire up or own provider? Will there be some sort of community-provided provider registry or will we be grepping awesome lists on GH? Are providers extendable in any way?
The main goal of Go Cloud is that the abstraction is not leaky and that you don't get access to specific features in the way you are describing in the application itself.
Now if the specificity of the thing you want happens at provisioning time but the API is the same (e.g. choosing to use Aurora on AWS vs standard RDS), then there's no problem. Go Cloud doesn't need to know about that.
If the API changes too, then that's a much different thing, and gets back to the leaky abstraction which we want to avoid. If you can't simply wire new providers to your application code and have it Just Work, then we've done our job wrong.
But remember that it's all Just Normal Go, there's no magic here. It's just standard Go. If you want to write an app that ties itself to something specific on some cloud you can; just write the code you'd write now. But that necessarily means you're tied to that product.
Contributors could also add drivers that not only fulfill your Blob interface but add features that might creep into multiple services. For example, someone could contribute a SuperBlob that fulfills Blob and also adds conditional writes, which GCS supports. On the app side, if they want to check if some feature is available that speeds up operations, they could then see if the returned Blob is also a SuperBlob and if so, perform the more optimized approach.
Maybe it's a biased and distorted perception that I have, but Go seems to be the most consistent and unsurprising language I've used, and I mean this positively. I haven't written any Go for over a year since my current job mostly uses Node, but I feel immediately comfortable jumping in to the examples in this blog post. I don't think that I'm explaining what I mean very well, but it's something I don't think is so true for Node/JS and other languages I've used.
Meh, everyone has a preferred type of language they find least surprising. To many Ruby is the least surprising, and Matz used "principle of [his] least surprise" as a guiding principle when designing it, but many other people find it enigmatic and confusing. Some people find Haskell the least surprising and most consistent, others say Java is. Every language has caveats and inconsistencies, even Go. Like how Go has generics, but only for two or three built-in types. It seems to be more about the person trying out the language than the language itself.
Minimalism helps. Ruby has one kind of minimalism, but fails at minimalism in some other ways. The same with Go. People who like Haskell like the kind of reasoning it facilitates. People who like Java want a managed version of C++'s "good parts."
A lot of it has to do with expectations. Going beyond the principle of least surprise, there's also a principle of "no dismay." Go is like the dependable old Corolla. It sets modest expectations, and it delivers a high degree of utility very consistently, and if you're ever let down, it's very rare. C++ can cause you utter dismay due to rather subtle slip-ups. I've only seen this level of dismay in Golang around subtle resource release issues around prepared statements.
Why is := used on the first line but = used on the second line, and then := used again on the third line? I get that you use := when introducing a new variable, and = when just assigning, the third line is not really introducing a new variable, since err is still in scope... or is it?
Not to mention, what is the last argument `nil` supposed to represent? In most high quality Node.js libraries, functions/constructors will take an optional options object so you can omit it, or pass an object like { timeout: 300 } to it.
Third line ':=' is not required, you can reuse globally scoped 'err' with '=' as well; unless you want to preserve the current value of global 'err' for future use (which would be confusing).
agreed that the := that can be used when one new var is introduced but not the others is extremely confusing.
I also hate the inconsistent behavior when you check multiple errors in a row, the first one will get := but not the second one (if you don't put those in a if statement, which I never do).
But now if for some reason, you remove the first := statement, the code will error on the second one, that needs to be changed from = to :=
> It seems to be more about the person trying out the language than the language itself.
This is true, but Go does have a clean and mostly familiar syntax, by design: you can mostly follow along with a piece of Go code even if you haven't learned the language.
I think that what the OP is pointing at is a related issue: readability. This is explicitly a high-priority design goal for Go: lots of things in the language are designed so that developers can easily read and understand the code in the projects that they work on, and that memory use is never hidden.
On the flip side, I personally find go extremely difficult to read.
First, repetitive error handling makes it overly tedious to actually get a quick intuition for what a function is doing. With Rust as a comparison, it’s extremely easy to see what the happy path is trying to accomplish without having the downsides of arbitrary and unexpected exceptions. Go is inarguably worse here.
Second, the inability to express higher-level concepts (due to, yes, lack of genetics) seems like it forces every function to deal with the nitty-gritty details of whatever it’s trying to accomplish, rather than being able to express ideas at a higher level and deal with the specifics in smaller code units. Yes, you don’t get “surprising” behavior, but the cost of this is that you’re forced to keep both the high level goals and the low level details in your head simultaneously, which I find to be a massive hindrance.
As sort of a side effect of both of these, there ends up being a ton of repetitive boilerplate. This actually hides bugs since it’s easy to miss minor differences between overly similar blocks of code dealing with errors or looping. As an example, languages like Rust and Ruby that have proper functional-style iterator methods prevent so many bugs through their inclusion it’s absolutely baffling to me how someone would opt to design a language today without such things.
The problem with inconsistencies and surprises in languages, is that these are often the basis of conundrums that are more expensive to detect, figure out, and fix by a factor of 10X or more. I say this with appreciation and without irony: It turns out that "Blub" is actually a good thing, so long as it's done right.
All languages are going to suck, somewhere. The trick is to actually optimize a language for the difficult, expensive parts of development. Many languages seemingly optimize for writing things from scratch slickly. They're "blog example optimized." Instead, languages should be optimized for debugging, reading code, and deployment.
The fact that Go is not afraid of boilerplate is often actually a good thing. I've had to debug C++ in the confluence of 2 templates, such that there was no source code to display in the debugger. I don't expect to have this problem in Go, ever.
"You can't pay people enough to carefully debug boring boilerplate code. I've tried." (Yaron Minsky of Jane Street)
While C++ goes to one extreme, not having certain expressive means (whatever they may be) has a price, and for certain kinds of problems, this price may be high.
This returns to the idea I very much agree with: "The trick is to actually optimize a language for the difficult, expensive parts of development." The catch is that difficult parts are also problem area-dependent. Go definitely found a sweet spot for certain kinds of problems, though.
"You can't pay people enough to carefully debug boring boilerplate code. I've tried." (Yaron Minsky of Jane Street)
There's boilerplate, and then there's boilerplate. If boilerplate is just a formality, with clean, competently designed underlying concepts, then it should be fine. Someone smart will build a code generator for that part of the project, you have your type safe-whatever and the project goes on its way. On the other hand, if your code base is just numb-nuts and people are just cut & pasting lava-flow code all over the place and rampantly putting business logic into ORM routines, then yes, that's going to impact hiring.
not having certain expressive means (whatever they may be) has a price, and for certain kinds of problems, this price may be high
This is also true. Even fairly innocuous boilerplate can build up to the point where refactoring can become tedious. There is no utopia, only constantly changing cost/benefit tradeoffs.
Go the language now is indeed very similar to Java pre-1.5, but with a better OOP story, and a much better concurrency story.
I still think that JVM + JIT is hugely important, though. Now it's importance is somehow diminished by presence of LLVM, but 20 and even 10 years ago the situation with native compilation was different. (I know that the canonical Go compiler does not use LLVM.)
Go is certainly very easy to reckon with. But I was surprised at how little O cared for it when I tried it, given the hype I'd heard. The two things that immediately jumped out at me were error handling and generics. I understand both were deliberate choices but I'm still surprised they were made - I've been working with Swift a lot lately and it just feels like it makes so much more sense to me.
I do love how opinionated Go is, though. Every language should have a gofmt equivalent.
I feel like a caveman typing out error handling code in go, but on the other hand (even as a beginner in Go) I _never_ run into unexpected conditions. There's tradeoffs of course but every time I find myself thinking about what kinds of error conditions may be happening in my Node code (where I spend most of my time), and whether I covered them all... i think I'd trade.
this. I'm writing a system in Go and Vue at the moment, and while there is a certain tedium to writing "if err != nil { " for the thousandth time, I'm happy that if an error occurs somewhere it's not going to get silently passed up the stack until it pops up somewhere totally out of context.
I keep shooting myself in the foot _the same way_, all the time - and it's incredibly easy to do so:
- not knowing if "Foo" being returned is a struct or an interface, so you're not sure if you should "if foo != nil" (it could not be nil if it's a struct)
- channels freezing (specially unbuffered ones). I still can't figure out how to debug those well.
- (the most common of all) spawning goroutines on a for loop, and all of them closuring over the same value (instead of one per item on the loop)
it's a surprising amount of sharp edges, for such a tiny language...
1. Get IDE support. There is no convention of adding a suffix like "Interface", and I personally am very happy with this. C# ISomething naming conventions have annoyed me quite a bit.
2. Always limit the scope of your channels. I like using unidirectional channels with structs that have more unidirectional channels. (For instance, a chan of workRequest{workData, returnChan} where return chan only receives responses for this request and is closed when no values are left)
3. I think you've been treated to well by JS closures, always hand over arguments you're going to use in the goroutine, this also allows the CG to clean up the stack of the function that started the closure goroutine.
1. Naming conventions don’t solve syntax warts. Modern languages solve this as part of their null safety (eg optional monads or the ? on kotlin and swift)
2. Yes, I know how to “properly” do it. Does not eliminate human error.
3. Closures in any other language, afaik. I rarely code in Js, fyi
Regarding 1, I (and seemingly everyone else writing libraries) usually return (value, error) and treat a nil value as error. We have quite a bit of old code at work that might segfault because it wasn't done like this.
Most of our Kubernetes code looks something like this
_, err := clientInterface.Update(object)
if err != nil {
if machinery_errors.IsNotFound(err) {
//handle nil case
} else
// handle unexpected case
}
}
Yes, gofmt is a wonderful since so many other languages can have very different way of writing code leading to harder-than-necessary digestion of new libs or projects.
Our company uses Go. Most people who join have no experience in Go yet usually they can get comfortable with the language in 1-2 weeks to start contributing. I know nothing about Node/JS but I'm sure that takes much longer?
Most people who join have no experience in Go yet usually they can get comfortable with the language in 1-2 weeks to start contributing.
A lot of what makes anything hard to learn are the non-obvious gotchas. The design philosophy of golang is aimed at avoiding the non-obvious gotchas and the creep-up-on-you-at-scale gotchas.
People completely misread my comment about doing concurrency with only synchronous code. Basically, channels let you do concurrency with only synchronous code as in Node JS, but without using callbacks.
1-2 weeks is standard for all languages though. A week or less is how long it usually takes people at work to pick up Objective C and contribute to iOS development, or pick up Scala and contribute to service development.
Your people can pick up go reflect package or go routines/channels and co in 1-2 weeks and start contributing? impressive... most people can pick up for loops, if,switch statements, variable and function declaration in a week in any language as well...
In fact, a lot of non programmers use JS regularly to write scripts in the browser. I imagine your people know how to program before hand... So what do you think takes longer to learn? mastering go threading model or being able to add interactivity to the DOM?
> So what do you think takes longer to learn? mastering go threading model or being able to add interactivity to the DOM?
For me, the go threading model is easier. I just can't stand DOM and web programming in general. Currently working on a problem doing some computation across dozens of cores on multiple servers and go makes it a lot easier to spawn the 1000 goroutines and coordinate between them. I guess everyone has different interests and perhaps it is similar for those we hire.
> For me, the go threading model is easier. I just can't stand DOM and web programming in general.
Because you didn't take the time to learn the DOM, that's all. I doubt learning a simple composite API made of dumb nodes is more difficult than writing thread safe programs in Go. I'm not talking about tastes or interests, I'm talking about ease of learning.
Go isn't 'easy to pick up'. For loops, if statements and co are basic, and they are in all languages, that's my point. There is nothing particularly easy with Go.
go reflect is not commonly used, goroutines are pretty simple to pick up. I have used several large codebases with zero use of reflection. The point of go is it restricts you to 25 keywords (the basics that you sneer at above) so senior people can't get fancy and produce an impenetrable mess/marvel that beginners will take months to orient themselves in.
It is definitely easier to write well than js IMO, and harder to make a big ball of mud in.
It is so consistent and unsurprising to the point of being boring. Which is a quality I have never thought I would appreciate, but if I am at work I do. It's very utilitarian.
I'll take Scheme for a spin if I am doing something for fun.
You're right, it is dependent on your perspective. Go is intended to be unsurprising to developers who want to just be able to look at code and have a good idea of what it does. Punctuation is very useful to get rid of ambiguity.
Contrast this with a language like Ruby which tries very hard to make punctuation optional, and often ends up with ambiguous cases where you aren't 100% certain how the interpreter will parse some code and so you end up throwing in all the punctuation anyway.
I like how Google tries to masquerade their corporate goals behind the 'portability' banner. Google Cloud is in a distant third place behind AWS and Azure, so ofcourse they don't want developers to have tight coupling to their competitors.
This portable cloud strategy is a mirage. I haven't seen anyone migrate providers easily (we tried in my last company). In reality, you have to break the cloud agnostic abstraction to leverage provider specific functionality. And then you are stuck with unmaintainable mess of 'cloud agnostic' APIs mixed cloud specific APIs.
> I like how Google tries to masquerade their corporate goals behind the 'portability' banner.
This may further Google's corporate goals, but that doesn't mean it wont actually increase portability, or that it's actually a bad thing for people who want to avoid lock-in.
> In reality, you have to break the cloud agnostic abstraction to leverage provider specific functionality.
I agree this is a difficulty, but I think that it's also avoidable if you make a disciplined commitment to interoperability. Just as you have to put in the effort to use this new "portable" API, you also have to choose not to defeat the former investment. Saying that this API is useless because you end up using provider specific functionality anyway is like complaining loudly that something is broken when you're using it wrong in the first place.
Having some common types available seems useful for people writing portable libraries. Maybe it's not going to change the world for most developers, but it could help defragment the library ecosystem a bit.
Yes this is a strategic move on googles part, but that doesn’t make it any less valuable to consumers. Getting less lock in and more choice is good for everyone (except AWS)
I mean, AWS is #1 so that makes sense, and then their own platform as their second choice to implement, that makes sense too. I don't think it's like a conspiracy that they haven't released the Azure support yet.
Am I the only person not excited by this? It seems to be based on the idea that you can just paper over the chasms between vastly different implementations. Putting aside the fact that the differences are important and you can't ignore them, doing this well is simply not scalable. How could it ever be all of: comprehensive, correct, up to date, elegant?
My understanding about a cloud provider lock in that it is a well thought-out, conscious decision (at least it should be) and after you made that step, you can use the specific services of the given cloud provider and you never move out.
Are those hybrid/mixed cloud solutions or move between cloud provider viable or that common?
Is there anybody went through a huge cloud migration which went well?
If you need to move, there are huge amount of data you have to move for example between S3 and Google Cloud Storage, is that even possible to do beyond an early stage? I can't imagine.
I'm working on a new kind of container that allows for lambdas with very-very low latency. (Suitable for running a game loop.) This is absolutely awesome! Programming against the auto-generated AWS golang APIs is pretty painful. It works, but it's not idiomatic and overly verbose. I'm going to give it a try and see if switching to GCP is that easy.
I like to see how Go seems to getting some traction within Google. For years, I had the impression that Google was more Java/JS/Python driven and I was wondering why they weren't using their own language for more projects :-D
Google is actually C++ centric - every other language is a bit of a "second class citizen" when interacting with infrastructure.
Go is still new and Google is "old", so there is surprisingly little Go code at Google - it doesn't make sense to rewrite critical infrastructure just because you have a shiny new language.
The derivative is positive though, and slowly Go is being used for more and more new development.
Yep. Google has built multiple ecosystems and they each have their own preferences. The grandparent was talking about Google's internal, server-side ecosystem. People working in that area sometimes assume that's it's the default.
Client-side development is a whole different thing. It's fragmented between browser, Android, and iOS. (And each of those is further fragmented, with multiple languages available.)
As a Java engineer, I really missed DI in golang. Then I found fx.
Once I wrapped my brain around fx, I really liked it and used it heavily in a project I wrote. It was especially good for initializing 'things' of the same 'type'. Like a bunch of controllers for a web app.
One nice thing about fx is that it does not rely on code generation (uses reflection instead). I'm very curious what other features wire has over fx.
That seems to be a common thing in Kubernetes too, if you don't want to spend the time writing actual code and Go is too "boilerplate heavy" for you, just write a code generator and bring the complexity of other languages right back!
I wonder what kind of monitoring and alert system must exist if any cloud provider changes their SDK or APIs (signature, logic, capability etc.) upstream while your go binaries are in production. Did Google manage to get cloud providers on board to register and update a 1 to 1 match between the upstream api and the go cloud api?
Secondly on the article's mention on multi cloud usage, you would need to be a pretty large place to need (or even bothering to assess and convince yourself that you need) to use multiple cloud providers at once. Just learning and tweaking settings in the cloud providers GUI console is half the battle won sometimes.
Saying all this, this, together with the data portability announcement, it is definitely great for competition and going to bring great resilience in your code base to be SDK agnostic.
I don't think cloud providers would just change their SDK or APIs without notice. It will break anyone using their SDK, not only go-cloud.
For your second point, it's normal for small startups to shop around and pick the cloud provider that's more suitable for their needs. For example on my last job at a small startup, we switched from AWS to GCP for cost reasons, and I implemented an abstraction (similar to go-cloud's) between S3 and GCS because of that.
They never change over the course of say 3 years? 5 years? If it changes frequently that's great - someone could be put to task to monitor. If it changes very rarely, you will have a surprise outage although I am guessing debugging and fixing it would not be an issue unless you have SLAs you need to keep up with.
> Secondly on the article's mention on multi cloud usage, you would need to be a pretty large place to need (or even bothering to assess and convince yourself that you need) to use multiple cloud providers at once.
Oh, this is not the case at all. Do business with any retailer? Try telling them your stuff is hosted on AWS and see how that works for you.
If this occurred frequently for my business, you would architect your system for on-prem flexibility and capabilities. However, I think for just "data" like S3 blob storage, this is definitely a no brainer. I would imagine for VMs its going to be a lot more difficult with load balancing, permissions and what not.
Isn’t this just centralizing the cloud provider SDKs? Instead of AWS, GCP, Azure designing their own SDKs, they have a common contract to adhere to. I’d imagine as the cloud becomes more mainstream, you’ll see similar support in Java and .Net (a System.Cloud package for example)
It looks like it, but think of the power. If you write your app for S3 and Amazon raises their prices 100x you can be running on GCP in no time. Of course, you just create interfaces for your data model and implement them with your favorite service, but nobody has time for that.
Terrafirm isn't provider neutral. It is just a declarative wrapper for cloud provider APIs. Code using one provider is not easily translated to another.
I do think it's an excellent configuration management and provisioning tool, but Go Cloud solves a different problem.
A frustrating choice. I spend a few minutes per week running Terraform on a recent laptop. If it were written in Python, it might use more memory and require a few dependencies (which I don't care about at all) but then I could write plugins without having to deal with IPC and weird language-specific serialization (which Java should have convinced us is a bad idea).
The majority of the team is in SF for Google Cloud Next to present this. I'm about to hop into the car to head up myself to watch, so I can do a little AMA throughout the day. Reply to this to make sure I see your question!
Here's a question I've already seen:
> Is this a Google project or a Go project?
It is a project led by the Go team, and we are very aware of our responsibility to be cloud provider neutral. At the same time, while the project is so new and evolving so rapidly, we need to have working examples that we can change atomically in a single repository (atomic changes necessary for rollbacks and such). We don't want to have cloud provider-specific code in the Go project, so the code is currently sitting under the Google GitHub org. When the interfaces to different products stabilize, the intention is to graduate them up to the Go GitHub org. Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured. The cloud provider-specific code will always live outside the Go project.
But things are very new, so change is possible.