It's a bit weird to see "several weeks" of effort being described as a problematic learning curve. At least the blogpost makes it clear that the effort pays out hugely but still, "several weeks" is not rocket surgery. It's not learning Haskell or category theory! ISTM that they're just running with an assumption that most devs wouldn't be professional and committed to this, which strikes me as an unwitting gatekeeping attitude.
Go appears to have a shorter time, because you don't realize how much higher-level stuff you're just expected to do The Right Way, with no support from the language or libraries. So you're free to think you've finished learning Go, but then the actual learning begins.
Agreed. Worked with a senior who "learned go in a week" then faffed around for months (to prod! because he had the implicit trust of management because he was "a genius") deploying broken software with tons of concurrency bugs because he didn't know how to manage shared state.
Even accounting for what you say, it's a short learning curve compared to most languages. It's not like there's any programming language in the world where you just read the manual through once and, boom, instantly you know exactly how to architect a multi-person-century project right out of the gate or something.
Hmm, that’s a bit of a straw man you’re making here. No one is arguing better languages will magic complex architecture efforts, we’re talking about basic state management here... garden variety implementation details
State in the context of concurrent programming is not a "garden variety implementation detail." It's the Great White Whale of our industry. No language does it especially well. I have my favorites in this arena, but it's still hunting a large sea mammal with a harpoon. What you're suggesting is essentially avoiding hunting it altogether. We're not there yet.
This is very true. I recently ported large portions of Go code to Rust, and while it was somewhat of a steep learning curve and a fight with the compiler, in the end I felt very comfortable that the result is correct, fast, and had good abstractions.
Go was so "easy" that I was immediately productive, but this resulted in often suboptimal and messy code that I had to refactor over and over again due to concurrency, abstraction, or performance issues.
Right, it's a false economy. Though, to be fair, the Java/C# ecosystem has the same problem to an even greater degree. Not to mention popular "dynamic" languages such as Python, Ruby etc.
Go is the golden standard for extracting the most value out of unexperienced computer science grads but it is not the measuring stick, not by a long shot.
Interestingly enough, Java caught hell from people for "pandering" to "average" programmers, where Go seems to be getting kudos for the same thing. Strange times.
I've only ever read through Go tutorials but never used it. I've used Java in a small capacity, and while I found some things convenient, some other things were also pointlessly restrictive. For example that syntactic overhead, having to wrap everything in a class. You can't even have a global variable without typing your fingers sore.
Pretty sure that Go isn't equally restrictive (while still being garbage collected).
Java was derided as being for people who thought C++ was too hard. But those people were right. C++ is too hard, in that "undefined behavior" demands an inhuman degree of perfection.
I have learned both and agree with this statement. I think that Rust is harder to learn if you've only worked with high-level, GC languages, and don't have a background doing lower-level programming in C/C++/Obj-C, as well as some experience with functional languages.
Going from something like Java or Python to Rust, one would have a lot to learn.
Hmm I don't think so. Haskell is so fundamentally different to imperative languages that it requires basically throwing away everything you know about programming.
To learn Rust you simply need to understand how values are kept track of by the compiler. Once you develop an intuition for this it's the same as any other modern imperative programming language.
I can convince myself that data races need not be a race condition. Consider this simple program:
var i int
doneCh := make(chan struct{})
go func() { i = 1; doneCh <- struct{}{} }() // a
go func() { i = 1; doneCh <- struct{}{} }() // b
<-doneCh
<-doneCh
At the end of the program, i is always equal to 1 no matter which order a or b wrote to i. But it's a race because you are assigning to a shared variable without synchronization. A small modification to the program creates a race condition:
var i int
doneCh := make(chan struct{})
go func() { i = 1; doneCh <- struct{}{} }() // a
go func() { i = 2; doneCh <- struct{}{} }() // b
<-doneCh
<-doneCh
Is i 1 or 2? It depends.
It is correct for the race checker to complain about the first program, because after a bit of hacking the first program can very easily change into the second program.
I wasn't sure. After a bit of research, this seems to be a debate [1]. Using the common definitions, it's possible to have a data race that doesn't cause a race condition. [2] It's also possible to have a race condition without a data race.
It is right, race detector is not enabled by default and you should explicitly run tests with it or tell compiler to enable it - it is not compile-time, but run-time.
It is a data race. I'm guessing the race detector (go test -race) didn't detect it because they are layering multiple synchronization primitives (mutexes, channel i/o, and a WaitGroup) and their tests hit the "good" code path but production workloads didn't.
Here's what happens. Delete takes a ResourceList. It delegates to "perform" and then "batchPerform". perform calls batchPerform in a separate goroutine, which calls a helper function in another goroutine for every resource in the ResourceList. The helper function is defined in Delete and updates a data structure defined in Delete. This is a classic case where some synchronization is necessary. The function runs multiple times in multiple goroutines, and updates a single shared structure. (Perhaps not obvious because it delegates to two helper functions, and the list that the function is executed on is a "ResourceList" not a []Resource, so it isn't clear that there is a "for { go func() }" loop anywhere; the programmers did their best to make it non-obvious that a loop is occurring.)
The confounding factor here is that batchPerform tries to synchronize with a WaitGroup, but it's faulty and not enough to protect the data integrity. batchPerform creates a WaitGroup, but only calls Wait() on the WaitGroup when the "kind" of an individual resource is not equal to the "kind" passed to batchPerform. I am guessing that it's very natural to craft some test data where this condition is met, and the for loop in batchPerform only runs the function once at a time (perhaps a ResourceList of length 1). In that case, there is no race condition for the race detector to detect.
All in all, if I were reviewing this code, it would not be checked in its current form. Splitting perform and batchPerform doesn't make sense to me, and they both implement faulty synchronization logic in a slightly different way. (batchPerform uses "for { wg.Add(); go f() }; wg.Wait", perform does "for range x { go func() { ch <- f() }() }; for range x { <- ch }". I consider these pretty much exactly equivalent, but neither prevents f() from running concurrently with itself. The only reason this passed the race checker is because batchPerform doesn't actually use the WaitGroup in the normal way, instead degrading to "for range x { wg.Add(); go f(); wg.Wait() }", which DOES prevent f() from running concurrently with itself, with certain inputs.
The root cause is that the caller of Delete isn't really sure about the semantics of "perform". Does it protect the body of the callback function? There is no documentation, and the author thought "yes". But the answer was "no". In general, the convention in go is to consider something thread-unsafe unless it's marked as thread safe. When you see something like "var foo Foo; f(list, func(bar){ foo = bar })" your spidey sense should be concerned about synchronization. But in this case, the code went out of its way to hide the existence of a loop and the existence of parallel processing, and so the programmer made a mistake. A bug or at least VERY confusing use of WaitGroup in batchPerform allowed the tests to pass. Should the compiler detect this? It would be nice. But a code reviewer should have been super concerned about this implementation.
The bug they caught [1] is one of the reasons some languages require you to explicitly name your captured variables. You still could have typed that code in, especially if you started with a for loop and then made it parallel (fwiw, perform should have been named something clearly suggesting it was parallel), but you'd at least be confronted with "oh, you went from serial, local state to a capture. Still think it's okay to explicitly borrow that state from this scope?". Then again, that's the point of Rust here :).
Fwiw, it's too bad the commit message didn't say something like "Since we're doing delete on many resources in parallel, we need to hold a lock while updating errs/res.Deleted". The reviewer was also obviously confused at first.
“For comparison, last week we caught a significant race condition in another Kubernetes-related project we maintain called Helm (written in Go) that has been there for a year or more, and which passed the race checker for Go. That error would never have escaped the Rust compiler, preventing the bug from ever existing in the first place.”
I’ve heard people brag that Haskell is a great language because it’s supposedly easier to write correct code.
Yes, though I believe Rust has already proven this more in practice than Haskell has.
Rust has many advocates now at places like Mozilla, Amazon and Microsoft that have delivered critical software in Rust that they believe has made it safer.
It’s harder to formally prove Rust code compared to Haskell, the company I work at prototyped in Rust and then used domain information gained to improve the parallel implementation in Haskell.
Yes, for much the same reasons. Pretty much any ML-family language has the same effect; just having proper sum types, polymorphism and first class functions (and not having null) goes a long way to preventing huge classes of bugs.
In practice you have fearless concurrency in every other ML language I know, because they're all immutable-first. It's true that if you wrote some code that mutated data then it wouldn't be concurrency-safe, but why would you do that?
> One of the biggest ones to point out is that async runtimes are still a bit unclear. There are currently two different options to choose from, each of them with their own tradeoffs and problems. Also, many of the implementation details are tied to specific runtimes, meaning that if you have a dependency that uses one runtime over another, you’ll often be locked into that runtime choice.
My understanding of how async/await works in Rust is that you can have multiple async runtimes in one Rust program. Is that not the case?
That is the case, but it's super awkward to use. Basically, you cannot await a tokio future on an async-std runtime, or an async-std future on a tokio runtime. You can, however, have both runtimes running at the same time, and use some form of message-passing to bridge them.
It's definitely easier to only deal with one runtime. Ideally, we should have some kind of abstraction to allow crates to support both runtimes (e.g. a trait that'd allow creating an async TcpSocket of the right "kind" for your runtime), but AFAIK this is not currently done.
Futures are part of the answer, and more specifically the way that the Wakers passed to Future::poll use dynamic dispatch to re-schedule the task.
Other major abstractions that are missing so far include async versions of the Read and Write traits, a Stream trait for the async equivalent of the Iterator trait, and perhaps a way to spawn new tasks.
> That is the case, but it's super awkward to use.
That's really the case for any language where an eventloop is not part of a builtin runtime (like it e.g. is with Javascript or Dart). E.g. in C++ we also have boost asio, libuv, libevent, wagle, seastar,GUI framework eventloops in GTK, QT, etc.
The thing is once you are in async land, nothing is interoperable anymore in most environments. Whether that's ideal or not is a separate discussion.
What I experience however somehow is that Rust users raise a lot more concerns about interoperability than I've seen so far in other ecosystems. I might stem from the fact that those users often never used another native async environment.
IMO it’s more that the Rust community has fostered a culture of doing things carefully and doing them well whenever possible (I mean, it’s the language that will argue with you for hours over reference lifetimes, after all).
There are certainly high expectations in the Rust community about doing things perfectly. But I don't think those "async ecosystem" discussions are a good example of productive discussions. I think e.g. in C++ there had been far more expert talk on standarization, within expert groups - like for the standarization of executors or the networking TS. And yet after 5 years or so nothing had been standardized yet.
In Rust the amount of people that actually work on the low level details and try to make things better is likely < 5. But there are a lot of expectations from everyone else about having perfect interoperability.
Does anyone here have any experience using Kotlin and can compare concurrency (with coroutines) to either Go or Rust? When I was doing more Java I really liked the approach Kotlin took with concurrency, but reading the comments here I'm sure I didn't understand the issues at the depth that is needed.
In theory you can, but in practice it would make your code very messy. If your dependency is using runtime A and you are using runtime B - how would they interact? Runtimes like tokio also provide convenience macros for you main mehtod, kind of locking you into them (I think), at least for that codepath.
If your application has two parts or binaries that are completely separate you could potentially use two different runtimes, but otherwise I don't think it would make sense. And even then, it would just be a mess.
Right now, your runtime is essentially picked for you by your dependencies.
Going off on a tangent, but this exact problem would be a worst-case scenario for Go getting user-defined generic types instead of only the current blessed ones.
t. C++ developer with a mixed std::string/QString/BSTR codebase.
Well, that's the current state of Go if the half dozen blessed versions don't fit your use case. Everyone just writes their own slightly incompatible versions.
> t. C++ developer with a mixed std::string/QString/BSTR codebase.
This has absolutely nothing to do with generics.
None of std::string or QString are generics. They are just an example of historical alternative implementations for 'reasons' ( portability/speed) that create a lesson the long term
It's also unavoidable. It's not like you can pass UTF-8 Go strings to UTF-16 COM interfaces. Somebody wrote code to convert a Go string to a BSTR and vice versa. You can do the exact same thing for std::string if you want.
So much this. Have seen apps where there were 4 diff string types brought by external dependencies and then like 3 or 4 more to deal with from diff Windows APIs.
I wouldn't discount what gp is saying though - having an (or a few) experts on hand from the start, can help training the first new convert "the right way" and they can then mentor the next one and so on.
Even just by being availabletto answer questions or help with code review. Doing some pair programming sessions would probably be useful too.
Depends on your definition of average: I found that to be the case with significant programming experience with traditional languages (notably not something like Haskell) so I think it’s plausible since the compiler, editor, and documentation are rather above average for newcomers. In particular, Cargo providing a lot of easy tools and the compiler providing really helpful error messages seemed to be useful for the time to write a real first program which does something useful.
Edit: one other big factor - presumably in their environment you have coworkers to get advice from. That’s huge when you’re first starting.
Yes, I’m using it in the sense of “can successfully develop a program which does the job” with the assumption that it’ll still take more time to do it quickly, use more advanced techniques, etc.
We became pretty comfy with it in less than a month in Aug 2017. (Let's say average guy had a few years of Python and this-and-that before that, and a ~5 year CS degree before that.) Sure, there was no async/await anywhere yet, but no crossbeam-channels either. And there were a lot less friendly tutorials and there were a bit more rough edges. (Especially that we did "IoT" so cross-compiling was ... an experience.)
”Several weeks of hard effort”, they said. I can buy it, if they actually work hard and are basically competent. Rust is a difficult language in total, that’s for sure, but you can get a lot done without knowing it all.
After reading this article, I'm excited about finding a reason to write a component in Rust and WASM. Can anyone recommended the best getting started guide for dipping your toes in the water? This article didn't have a link to anything that seemed appropriate for that goal.
I looked into WASM / WASI last week but couldn't find an answer to this anywhere: can I write a network service in Rust and compile it to WASM / WASI?
I know that wasmtime can execute a WASM module and give it access to a file system. Can that filesystem contain a socket that the WASM module can interact with?
There’s a version of NGINX that’s compiled to WASM.
Conceivably you could compile all of the CPython runtime into WASM, just that you’d be left with a big binary that gets passed around all the time over the wire.
Good article but it somehow suggests that because there is no garbage collection you would need to fight the borrow checker. This is not fully true because you could put your data in a Box (so that it is stored on heap instead of the function's stack) and you can wrap it in a mutex with reference counting (Arc+Mutex or Rc+RefCell), which roughly gives you what garbage collection does. Also cloning can avoid solving the borrow-check puzzle if you don't need a shared state. Of course you would not want to pack your code with Arc+Mutex or data copying if performance matters, but it's fine for a beginner to start with when writing Rust and then learn to do the optimized borrow version a bit later when needed.
Qt is actually a good example because C++ has a similar memory model to Rust (at least with respect to GC). The Qt solution was basically to give everything a “Cow” (copy on write) wrapper, and to use an event loop-based, somewhat manually-annotated GC for objects that want it.
Rust could totally do the same thing, and you could probably make it way easier to use than the mess that is Qt.
Having dealt with Gtk-rs, and their current solution being the clone! macro, I am not so sure.
Remember that not only is GUI development with proper tooling very interacting, instead of the FOSS alternatives of code-compile-check visually, there is also the whole eco-system of third parties selling component libraries, with no control how they get integrated into the component toolbox.
So whatever solution one comes up with,it needs to be more productive than forcing users to scatter Rc<RefCells<>>, or fix their code that broke compilation, just because moving a widget on the GUI tree invalidated the borrow checker assumptions.
> For comparison, last week we caught a significant race condition in another Kubernetes-related project we maintain called Helm (written in Go) that has been there for a year or more, and which passed the race checker for Go. That error would never have escaped the Rust compiler, preventing the bug from ever existing in the first place
While the possible security benefits of Rust is interesting in software like Kubernetes, it seems like this whole blog-post is an implicit RIIR proposal for the Kubernetes ecosystem from a Microsoft software engineer which isn’t going to happen anytime soon.
> Rust has made great progress in the past year with its async story, but there are still some issues that are being worked out.
On top of that, there are still many crates that aren’t using async-await yet and most are not even 1.0, thus are not stable. I would not touch such crates if they are still immature or even unsafe.
Realistically, a Rust Kubernetes is possible but practically the effort of a production ready version is measured in years.
Kubernetes is an ecosystem. It doesn’t need to be written in Rust for Rust components to play a part. Helm is not Kubernetes, for example, though your comment seems to blur the two. There are folks writing stuff to interact with the broader ecosystem in Rust. That’s one of the interesting bits of networked systems! You can be heterogeneous with languages more easily when the network/api is the boundary.
