How is that different from compiling other software with build dependencies? I mean, if you consider libc etc most applications have quite a large tree of build deps if you need to download _all_ libraries from scratch - it's just that in most cases you already have those deps.
Next time you need to build a golang project you most likely won't have to download all of those libs again, unless you remove them for some reason.
Next time you need to build a golang project you most likely won't have to download all of those libs again, unless you remove them for some reason.