Hmm, the first few WATs are pretty bad examples. Any developer, gopher or not, should know that, when you reassign the value of a function parameter, the variable when the function returns still has its original value. It's pass-by-reference 101. Now, sure, slices and the way append work are a bit complicated to grasp at first, especially if you never used C, but this is not a WAT.
A lot of WATs are more bad practice than problems in the language, IMO (like WAT 8, modifying the return value twice in deferred functions, with the order being significant: who the hell writes such a code?)
A few ones, though, really are problematic. For instance I've been bitten more than once by variable shadowing, especially with error values. IMO, this is the weakest point of Go.
I'm not sure views into a fixed-sized array is really confusing/strange at all. Whether you create multiple views in the array, mutate the views, etc. All interactions that occur on slices and arrays make plenty of sense. How you'd write something like append yourself if "Slice" was your type w/ a backing array, a start index, and a length is quite straightforward.
This is absolutely one of the worst aspects of Go. When you call append() on a slice, and then modify the contents of the result, you may or may not modify the original slice.
In a language that encourages concurrency, this indeterminism should be jaw-dropping. How many Go programs are safe only accidentally via an array's allocation policy (especially given how much code it requires to duplicate a slice)? Can you think of any other language that has such a design?
It's not indeterminate, just conditional at runtime, no different than any other may-mutate-parameter-reference function. It fits Go's philosophy of foot guns for the sake of performance/simplicity (be it compiler or runtime performance/simplicity). Use of append by devs should invalidate any future uses of the parameter, but this is better than forcing a copy or abstracting implementation details of linked arrays.
> Can you think of any other language that has such a design?
Sure, happens all the time because it fits the machine naturally though sometimes it may be hidden with unnecessary runtime costs by many languages that do force copies or reallocations due to immutability. C++ is a language that doesn't hide it. Your argument is like saying you're surprised that std::vector::emplace_back may or may not alter the array I had kept from std::vector::data previously.
> I'm not sure views into a fixed-sized array is really confusing/strange at all.
Again, as I already stated in the previous comment the issue is not the slices themselves.
It's that they're doubling up as vectors and they're shitty at it: you can share a backing buffer between mutable slices and append to both and then all hell breaks loose.
That is the issue. And that is why Rust doesn't have that issue despite having the same concept of slices: you can't grow a slice, and you can't have multiple mutable slices to the same backing buffer.
I mean, by that measure C/C++ and many others have the same problems. Your issue about lack of compiler-enforced reuse checks after invalidation is more with the language than this feature. While I agree generally, I don't see how slices are any more of a problem or surprising here than any other mutable reference that may or may not be mutated conditionally by a function.
> I mean, by that measure C/C++ and many others have the same problems.
No. C only has raw pointers so it's not trying to pretend slices are vectors, and C++ has actual functioning vectors with "slicing" being either a copy of the subvector or an iterator. Either way there is no confusion as to the capabilities of what you're given and its relation to the original data.
> Whether you create multiple views in the array, mutate the views
Whether there are multiple views or not is actually an implementation detail, and whether "append" chooses to allocate memory or not will change whether there are multiple copies or multiple views of the array.
All of that behavior is something you absolutely cannot rely on.
Can I not rely on append only allocating when the values to append won't fit in the remaining capacity of the underlying array starting from the end of the slice? Some articles indicate this and definitely don't seem surprising [0]. Granted I don't know the implementation details, but how do copies of the array or numbers of slices of the array affect that? Regardless, with Go as opposed to lower level languages, building reliance on whether functions do memory allocation (even built in ones) is folly.
The behaviour of append and the conditions where it will allocate a new backing array are specified and part of the language. Implementation details don't even enter the picture here. You can absolutely rely on that not changing.
>A lot of WATs are more bad practice than problems in the language, IMO (like WAT 8, modifying the return value twice in deferred functions, with the order being significant: who the hell writes such a code?)
Isn't that the tired old argument from the C camp? "If you are careful you can write perfectly fine C without bugs".
DUH!
The whole idea is for the compiler and types to catch as much of this crap as possible, not to add accidental mental burden to the programmer.
While I agree that compiler should eventually be updated to prevent this sort of behavior or something similar, I agree with the grandparent here. I had same sentiment as the grandparent, in that for several of these errors, if you end up causing them it just belies a lack of understanding of software development in general. It’s great if we can improve compilers to teach people about these sort of behaviors, but they don’t really fall under the definition of WAT that I think is being referenced in the article. That is, a WAT behavior like this is something you would normally expect to behave in a certain way, but because of some language idiosyncrasy or design decision in the language it actually does something different.
Is it though? Slices seem to be passed by value, where the value is a pointer and some bookkeeping information (like a C struct of (data, len, capacity)).
Some operations manipulate just data (which is visible in the caller, since that 1/3 of the "struct" was a pointer passed by value) and some operations manipulate len and/or capacity (which is not visible in the caller, since that 2/3 of the "struct" was also passed by value).
Seems more complex than just hand-waving the details away with "pass-by-reference 101".
It's pass by reference 101 where by reference is actually passing a struct by value, and the struct has a reference to more data, so sometimes the backing data is updated and the struct isn't...
So, probably more like pass-by-reference-and-by-value 201.
Go is only pass by value. Much in the same way JavaScript is (the best description for JS, IMO, is reference by value).
Even if you pass a pointer, the pointer is explicitly copied to a new location in memory, which is why assignment to the pointer can overwrite the pointer, but not affect the original pointer (the memory location value inside the pointer-typed variable) outside the function scope.
Pass by reference is when assignment to a parameter name is transitive to the caller's scope, and can be seen in C#. [1]
Now that C# has ref locals, it's not really pass-by-reference so much so as C++-style (or Algol-68 style, if you go far enough) references that are implicitly dereferenced on access and cannot be rebound.
Go is not really 'pass by value' when passing slices. It just passes the pointer value instead of doing a deep copy(copying the backing array). When trying to grow a slice in a function, the result is unpredicted because of 'apend' implicitly choosing to allocate memory or not. In addition, it is not concurrent-safe to pass mutable data. To avoid the safety problem, you have to copy stuff and consequently suffer a performance penalty. A better way to solve the problem is to introduce immutibilty. If one day immutibilty is introduced in Go 2(or 3), i wish slices are really passed by value. Then everything is passed by value as default with the immutable data passed by reference. You can still use pointers to pass mutable data just like using '_' to explicitly ignore error handling.
Now in WAT 1,
func grow(s []int) { // s is deep copied.
s = append(s, 4, 5, 6) // changing 's' does not effect original slice.
}
Explicitly pass mutable slice,
func grow(s *[]int) { // s is referenced.
*s = append(*s, 4, 5, 6) // changing 's' will always effect origin slice.
}
In the given examples, it is. But I agree slices are complex in Go, probably the most complex thing for newcommers, and the main reason why I wouldn't recommend go as a first language.
My first thought was that a lot of these WAT should elicit an eye roll from anyone who took Comp Sci. My eventual conclusion was that this was just a form of stealth basic Comp Sci education for language hipsters.
The slices are actually confusing and I've written quite a few test programs to teach myself the WATs because the behavior is not obvious from the syntax. Remember, slices are very mutable, regardless of the lack of a * in the function parameter. This works as you'd expect:
type thing struct {
foo int
}
func f(x thing) {
x.foo = 42
}
func main() {
var x thing
x.foo = 1234
fmt.Printf("x before: %v\n", x)
f(x)
fmt.Printf("x after: %v\n", x)
}
This prints 1234 and 1234 as you'd expect. That's pass by value in action. But what if we mutate a slice?
Of course, this prints [1 2 3] and [1 42 3]. While you're passing the slice by value, the backing array is not copied.
As you write more go, you will begin to fully see the existence of a backing array. Try this program:
func main() {
x := []int{1,2,3,4}
y := x[1:2]
y[0] = 42
fmt.Printf("x: %v, y: %v\n", x, y)
}
You know that [:] does no copying (there is a copy() function that copies stuff) and just shares the same backing array between each slice, so you aren't surprised when this prints "x: [1 42 3 4], y: [42]".
Now, add some knowledge from the documentation about how append() works. You use the idiom slice = append(slice, element) because the documentation says, sometimes the slice is updated in place when there is enough capacity for that to occur, and sometimes the slice is copied to a new slice and returned.
This then leads to your "WAT" in WAT 2. Because you _know_ that the data you appended is actually in the backing array, but for some reason Go isn't showing it to you.
You can see this in action with a very poor example I just wrote:
func main() {
x := []int{1, 2, 3, 4, 5, 6}
y := x[0:0]
fmt.Printf("x: %v, y: %v\n", x, y)
fmt.Printf("intermediate result that's not saved: %v\n", append(y, 9, 8, 7, 6, 5))
fmt.Printf("x: %v, y: %v\n", x, y)
}
After all of this, x is [9 8 7 6 5 6]. So you _know_ that append is more than happy to mess with your backing array. You just need a slice with the right length to be able to see all the data in there.
For that reason, I think WAT 2 is worthwhile. Most people know _just enough_ about slices to be dangerous, and so are surprised when there is an additional complication that they haven't thought about.
"The order of iteration for a Go map (a hashmap) is unspecified, by design."
I that surprising behavior? It is not only well-documented, but is in line with most other languages that offer a hash table / map / dictionary / whatever in the base language or standard library.
The Go runtime explicitly randomizes the iteration of map, because the language designers noticed 'Programmers had begun to rely on the stable iteration order'[0]. Most languages, as far as I can tell, tell you that the iteration order is undefined but frequently have some stable iteration order that's consistent for the compiler or the system architecture, and don't take the extra step of randomization.
Having done the work of observing langugage users apparently want an ordered map, were there any steps taken to add such a facility, or did efforts stop at letting developers know they were they were wrong by taking away a de facto feature?
Ordered maps require other tradeoffs. For example, c++ has both: std::map and std::unordered_map. std::map is a treemap, and so has logn access/insertion times. It also does a ton of small allocations, and scatters your data across memory, leading to poor cache performance at scale.
But it has ordered output, and doesn't invalidate iterators on insertion (hashmaps might, because they sometimes need to rehash).
Generally, the suggestion that I've heard is to avoid using std::map unless you really really what it specifically provides, because it's expensive and it's hard to know if you can safely relax those constraints.
C++ ordered maps are in sorted order whereas php-style hashes are in insertion order (recent versions of Ruby and Python have adopted the php semantics; in JS maps are php style but JS objects are unordered)
The biggest advantage of trees is deterministic complexity. There's a bunch of DoS attacks that exploit hashtables, because they can be O(n) in the worst case - and if you control the input, you can deliberately make it worst case. Most "secure C++" guidelines recommend defaulting to std::set and std::map for this reason alone.
If you really need an ordered map, implement one in Go isn't too difficult (but the usual no-generic caveat applies). You can just define a struct that wraps a map and a slice (or some other data structure) to record the order of the keys.
If you want it to be performant, a linked list is better than a slice. Then in your map you store structs that have both the stores value and a pointer to the corresponding node in the linked list. That way random deletions are constant time since they don’t require copying the slice.
Some language implementors deliberately introduce gotchas to bite you in the ass should you rely on implementation-specific, rather than standardized, behavior.
Take this example from Scheme: R5RS specified two procedures, integer->char and char->integer, to bijectively map characters to integer character codes, but it said nothing about a specific encoding. Most Scheme implementations happily mapped characters to their ASCII codes or Unicode codepoints.
Not so Scheme48, which mapped each character to its ASCII code number plus a magic constant (believe it was 1000). Scheme48 provided library functions ascii->char and char->ascii, that did what most people actually wanted and its maintainers insisted that people use those.
Had significant code appeared in the wild that, say, simply subtracted the magic constant from the value returned by char->integer and re-added it again before calling integer->char, the maintainers probably would have either changed the magic constant, or changed the mapping entirely, in a future version.
Many (probably most) popular languages nowadays change the hash function and, with it, iteration order between runs to make it much harder to do a denial of service attack using hash collisions (https://www.securityweek.com/hash-table-collision-attacks-co...)
Reading your [0], I couldn’t say whether go changes iteration order between iterations in the same run. I would guess it doesn’t.
It's not that uncommon, actually:
For the same reason (too many people depending on non-stable orders) - LLVM now deliberately shuffles things when you use non-stable sorts to provoke different results.
One motivation for this is the herculean effort it took to move from an older to newer version of java.
The stable order that exists turns into another footgun when people start to hardcode that into tests, and it breaks out from under them when the compiler changes slightly.
When I first learned about hash tables (in Perl), the book I used said that the iteration order over a hash should never ever be relied upon, and that it could change arbitrarily from one release to the next. If the order mattered, one should use a different approach (get the keys and sort them by whatever criteria), plain and simple. I guess that has sunk in pretty deeply with me. ;-)
The .Net framework provides an OrderedDictionary and a SortedDictionary, both of which make some kind of promise regarding iteration order, but I have never used them myself.
It's easy to print a map in arbitrary order. All you have to do is create an array, iterate the map append to the array, create a new type, define the Len() and Swap() and Less() functions for that type, cast your array to that type, and finally call Sort on it. It's literally less than 20 lines of code.
Wat 4 is just setting up for wat 5 though. On its own it isn't suprising, many languages use insertion order for iteration, but many others use the hash order. But wat 5 is truly surprising, no other language that I can think of iterates through the same map in different orders (different maps, even clones - sure different order makes sense).
I'm actually impressed by the go devs here, it's a great way to avoid people making assumptions if your order is legitimately randomized.
It's not a Go-specific thing, although I'm not aware of anybody else doing it like this, producing different results at runtime for the same map. The more usual approach is to make the hashes themselves random, e.g. by utilizing a seed that's different for every new process. This is the case in .NET Core by default, for example, and in the legacy .NET Framework it's been an off-by-default switch for several years now.
Well map in C++ is ordered, while unordered_map is not.
If the name was hashmap then it wouldn't be surprising, but for "map" and especially "dictionary" I can see how people might expect them to be ordered without further information.
I was really expecting the slice WATs to carry on to talk about what I consider a real WAT: you have to perform robust checks on your input slice's capacity and length if you use append in a function, because append makes its own choice whether the array of a slice is reused or copied.
In particular, note how this behavior can easily be forgotten when the semantics are hidden through variadic arguments that are passed an existing slice (instead of the automatically created one when you pass in actual variadic arguments).
> I was really expecting the slice WATs to carry on to talk about what I consider a real WAT: you have to perform robust checks on your input slice's capacity and length if you use append in a function, because append makes its own choice whether the array of a slice is reused or copied.
At a fundamental level, the issue is that Go's design decisions make any modification of a shared slice dangerous, and the language provides no way to mitigate it.
A really fun one is divergent appends to slices with a shared backing array and enough capacity for an in-place append: https://play.golang.org/p/ZHWo3bFOR0X
That's the one I show in my example above :) The final example shows how paired with variadic arguments very "weird" things can happen since your slice may come from an entirely different codebase and you may get very difficult to debug results.
Appending to the end of the slice creates a new slice and therefore the caller doesn't see the mutation.
However appending to a slice-of-the-slice leaves capacity in mutate's copy-of-the-slice to append without allocating a new backing array. Therefore mutate happy bumps the len of its slice copy and mutates the callers slice!
This bit me once in real production code when reusing a []byte array to avoid allocations. The bug was obviously my fault, but this behavior can be easy to inadvertently trigger if you're trying to avoid allocations!
Edit: fixed code formatting. It's 2018 YC, please please please implement at least a subset of markdown.
Nils are among the biggest wats in Go in my experience. "useful nils" just compounds "the billion dollar mistake" into something even more nefarious and even harder to identify during code review.
Along similar lines: WAT 15. Under what circumstances do you expect a nil var return to become non-nil? Did you even know that was possible?
> Along similar lines: WAT 15. Under what circumstances do you expect a nil var return to become non-nil? Did you even know that was possible?
Yeah. Typed nils are a horrible, horrible feature: when you cast a value to an interface, it creates a fat pointer of (Type, Value). From a nil, that's (nil, nil) but from a nil Foo that's (Foo, nil). And since `==` just does a straight value comparison, (nil, nil) != (Foo, nil).
I really enjoyed Go until I started trying to work with interfaces. The fact that you have to understand the implementation details at this level in order to use them reliably, IMO, shows that when the Go team talks about language simplicity, what they really mean is _compiler_ simplicity. At some point, the complexity starts getting offloaded onto the end developer.
That said, the fact that the crowd only offered an incorrect guess four times out of 16 is telling. This really didn't have the same feel to me as the original WAT talk, which is really full of truly strange things.
IMO, Go is an attempt to design a modern language using C philosophy, and it shows. Like, this whole thing about slices is because they insist on trying to make arrays + slices into a data structure that does everything "good enough", while mostly retaining their performance profile. And so you end up with all of those abstractions that look kinda sorta like what you'd expect in a more typical modern language - but the moment you start poking hard at them, they leak like a sieve.
Sure there is. Do what the vast majority of other languages do - don't bundle behavior and nil like this. Or even better, make nullability a proper first-class citizen so this nonsense can be prevented at compile time.
I would add garbage collection to that. Being able to ignore memory management and trust it to be handled at a non-deterministic time by a separate entity outside of your control is certainly useful, but it's also about as "magic" as programming gets at the language level.
"The default type (used for inference) for an int constant is int, which is a 32-bit type" -- this is not true, the size of int is platform-defined. See https://golang.org/ref/spec#Numeric_types. The error you're seeing is because on 32-bit platforms where int is 32 bits, 2^64-1 does not fit an int, and on 64-bit platforms where int is 64 bits, 2^64-1 does not fit an int either (int's are signed). This will work on 64-bit platforms though:
At first I was wondering, what is this? Anyone with a Comp Sci degree who read the docs should already know most of this! Then I thought, "This is one of the best stealth Go basic Comp Sci education presentations I've seen in awhile!"
Looking up a key in a map you didn't "make" gives you the zero value. I think that's consistent with much of Go. But if appending to a nil slice works, assigning to a key should never panic, even if you didn't "make" it first.
I feel like every language should have at least one "wat" style talk or article. Even for languages you adore, it's important to understand the quirks, shortcomings, and issues in a context that might seem absurd to an outside developer.
This was a design chose to minimize the number of keywords in the language.
> The rationale is pretty simple: Only identifiers that must be keywords for syntactic reasons are keywords.
> In fact, in the very beginning there was some discussion as to whether things like nil, iota, etc. should be keywords. Eventually we agreed on the rule above which settled it.
WATs are usually never real code, some of the ones in there are but I would encourage you to look at the linked video in the article for a fun look at other languages WATs: https://www.destroyallsoftware.com/talks/wat
Some of those in the original presentation can actually happen accidentally because of dynamic typing, though. For example, you could get a value from a JSON object, expect it to be a number, but have objects instead, and perform a {} + {} without actually knowing it, have the interpreter execute it with no error, and wonder where in your code you caught that NaN value (because it would propagate all along).
OTOH, it's very unlikely you type accidentally `true := x` in your code, and even if you did, that would very probably be caught by the compiler anyway.
What typo can that be though? If you wanted to do `if true == x` and typed `if true := x`, the compiler will tell you. If you mistyped your variable name, and wanted to do `tue := false` for instance, but inadvertantly typed `true := false`, the compiler will tell you "true declared and not used". I can't really find a compelling example.
>If you mistyped your variable name, and wanted to do `tue := false` for instance, but inadvertantly typed `true := false`
You don't need to have a variable like "tue" to make such a typo. Since your writing an if statement, you already have "true" and "false" in mind, so you just need to be a little absentminded and voila, instead of foo := false you've written true := false. Plus, lots of naive editors will offer to autocomplete something starting with t to true (if they find the token true used elsewhere within the file).
>the compiler will tell you "true declared and not used".
Only if you don't use it. But a few lines below you could very well be using true legitimately too, in which case it wont tell you:
> Only if you don't use it. But a few lines below you could very well be using true legitimately too, in which case it wont tell you:
Good point!
Edit: but then, your expected original variable name, `tue` in my example, would be used while not yet defined:
true := false
if foo() || bar {
tue = true // Compilation error
}
...
if condition() == true {
...
}
Only possible case I can think of: you already defined tue, but tried to redefine it via variable shadowing:
tue := true
...
if cond() {
true := false
if foo() || bar() {
tue = true // Will compile, but not the tue you expected
}
...
if condition == true {
...
}
}
tbh I think it's a lot less of an issue. It's creating a new var named `true` (or `false`) - that only exists in the scope it's declared in, not globally.
I think it's crazy that this isn't a compile-time error, but it's not even close to "you can change the global value of builtin values".
nil is weird in Go, and in most languages with an equivalent concept. It's a "hole in the type system"
In Smalltalk, nil just contains the sole instance of the UndefinedObject class. It's not a hole in the type system. Instead, it becomes a paradox in the inheritance system, because it's used as a superclass.
Here's a WAT. Smalltalk is actually strongly typed. It's just that everything has the same type of Object. (The type system has no holes. However, it's the size of a thimble!)
I didn't finish reading all the WATs, but if these are the best WATs about Go, it must be a very consistent language.
The first two, for example, aren't WATs at all. The Go book is very clear that you need to use the result of `append` to get the modified slice. WAT 4, map traversal is unordered, is not at all a WAT. WAT 7, maps are reference types, is also not surprising at all. WAT 8 is also not a WAT, defers are defined to be processed in LIFO order, and the rest falls out of how named returns work. This was a waste of time :(
I think that what you are saying is true, in a manner.
To my eyes, it is however in the manner that for the small syntax and the small number of idioms, it consistently surpises me how weirdly they interact.
> it consistently surpises me how weirdly they interact.
What exactly surprises you?
I just read the Go book and none of what I saw was surprising in the least. I consider a WAT something like Python mutable default arguments that are certain to surprise every Python programmer at some point.
Once you realize slices are just (pointer, length, capacity) structures, and the structure itself is copied by value, the first 2 WAT is pretty trivial.
Reading from a nil slice and map feel consistent to me. If you read from a nil map you get the same result as reading a key that is not in a non-nil map (because that key is clearly not in the map because it's nil). Similarly, if you read from a nil slice you get the same result as reading an index that's not in the slice, an out of bounds access.
Trivial in isolation, but it is much harder to see the problem in a heap of source code. In general such "surprise moments" should be regarded seriously however it seems trivial in isolation.
A lot of WATs are more bad practice than problems in the language, IMO (like WAT 8, modifying the return value twice in deferred functions, with the order being significant: who the hell writes such a code?)
A few ones, though, really are problematic. For instance I've been bitten more than once by variable shadowing, especially with error values. IMO, this is the weakest point of Go.