I think the only realistic solution to this is to vendor your go dependencies. If Github ever changes its URL schema, a whole bunch of stuff is going to break.
The Go maintainers have conflated what a library is and where a library is. That's an important distinction that should not be glossed over like this.
I've been using Go professionally since 0.8 and in production since 1.0; this is correct.
Since day one we have forked every (external) package dependency in production code to our own github account and for all the constant complaining I hear, it has been very little heartache for us.
This has worked great, we periodically update our forks from upstream, once or twice a year need to fixup our code to match, run our unit tests an integration tests, do a little manual testing and call it good.
Vendoring through tarball dumps of GOPATH I can understand, but how do you deal with the dependencies of your dependencies when vendoring through forking?
Of our 40 or so external dependencies (huge project), only a handful themselves have dependencies we needed to also vendor. Its sounds painful, but in practice it hasn't been. We actually stopped using godep, we were getting no reward for the hassle and added complexity.
Unless there is a bug, or new feature we needed, we often just update this stuff when we update the Go release we are using.
My thoughts; KISS until more complexity is actually needed.
Wouldn't that involve changing the references in the mid-level to the leafs? Now you have to maintain changes to all the mid-level packages, sounds like not fun.
> The Go maintainers have conflated what a library is and where a library is. That's an important distinction that should not be glossed over like this.
I think this depends on whether GitHub treats URLs as just URLs, or more generally as URIs. Based on their documentation [0], they refer to them as URIs, which means they are intended to represent both Location and Identity [1]. The domain represents the "where", the repository path is the "what".
They should be able to change their URL schema relatively easily, by including the appropriate redirects - a mapping from the old schema to the new. Which is exactly the strategy they discuss in the HTTP Redirects section.
It's nothing Gradle-specific - as you say it's the Maven scheme and used by almost all JVM dependency managers. Note also there's a convention of using a domain you control in reverse order as the groupId (i.e. com.mycompany:myproject:1.0) which makes collisions extremely unlikely. Finally it's very easy in the maven ecosystem to run a "proxy repository" for your organization that indefinitely caches all the artifacts you depend on, thus ensuring that your builds will always be reproducible.
On the positive side, some of the complexity comes through enforcing some standards about what packages go into central - in particular you must have a properly declared license, developer contact information, and GPG sign your releases. Which are things other language ecosystems would do well to enforce.
The first time definitely is a drag, but I had to do it exactly once a few years ago. If I ever need to publish another library, it's trivial. Bintray isn't standard, which isn't the worst thing in the world but does become aggravating at times.
Re Bintray isn't standard: There is no standard, both Bintray and Maven Central are just internet repositories. JCenter is bigger, so if you define "standard" as "bigger one", Bintray is the way to do.
If by "standard" you meant "default", like Maven Central is the default in Maven, here I have some news for you as well.
Bintray is the default in Mac OS' Homebrew, Android Studio, Groovy's @Grab, and first class citizen in Gradle, Ivy and SBT.
That's my biggest peeve with Go. Well second biggest after hushed tone - generics. I really really wish there was something like a Gemfile or Rust's Cargo or Clojure's Meiningen, something that's not...whatever it is that we have today, which is technically nothing.
I can kind of understand the Go devs' POV which is that basically at Google all source code vendored under one giant tree, but that's just not how not-Google companies operate.
Rust is looking really attractive to me lately, as soon as I figure out lifetimes and the borrow checker completely....
It will be great fun when Github is shut down. Many npm packages, jspm, and more are using hard coded links to repositories or even paths in repositories.
Or a bad player buys them and over night removes all traces of certain projects. I still remember how e.g. MaxDB (former open source database) vanished.
Exactly, a bad player could do much more damage than simply turn the services off. A lot of the time there is no verification for the downloaded code. It's just being run.
Yea, if someone at a nation state level manage to fake a cert and redirect BGP, I'm sure they could affect a huge number of systems pretty quickly. Do git clients even do a good job of SSL verification?
My company currently pays for GitHub hosting for unlimited private repos and X contributors.
Doesn't the fact that GitHub makes the enterprise pay for usage make it more likely that GitHub will be around longer than Google Code?
Honestly, if GitHub can stick around for another 10-15 years that would be ideal. By that time I should either be out of the industry or off to another company that uses something other than GitHub.
That's not the only way there will be trouble. A more likely scenario is "We've been going over the numbers. To keep our costs down, we are making the following changes to our support of open source projects..." I'd be surprised if that didn't happen in some capacity in the next five years.
Actually perhaps. If the current team continues to own and run it, it'll be around forever. But eventually they'll want to sell it, the team will change, and its goals will become more feature-oriented for the sake of grabbing more premium members. This will give it less value (due to the removal of simplicity) to the average user, so its userbase will gradually shrink until a few years later they pull the plug. It's happened to almost every web service more than a decade old.
Well... maybe? SourceForge was always a really terrible experience for me, using it in say 2004/05. Loads of ads, interstitials and other general bad behaviour. It's not exactly surprising.
SourceForge used to be okay though already looked a bit dated, back then. Slashdot and SF were owned by the same company. Though adding adware to binary downloads is downright incorrect and alarming. SF doesn't even host the binary files themselves, they just provide an interface for various open file mirrors like HEAnet. HEAnet and co should block SF and we would need an open community website that links to the mirrors to replace SF.
Five years is a long time in tech. I think it's more likely that we will see same thing happen with Sourceforge and Google Code - a better thing comes along and people move to it.
This is something which is seen over and over again. Its not that loved active services and sites get turned off, it's that the userbase moves somewhere else first.
There has also been a fair amount of executive turnover lately. Plus it's not a great business which means as cash is spent some type of business change is inevitable which includes selling it and relinquishing control to some new owner which might have a different idea (see Oracle and Java).
I like your crystal ball. Maybe I should use it to gamble in the stock market.
As for the rest of us, you can't possibly see or guess the future that will affect that business. Maybe they'll get sued out of existence by then? Maybe the owners will run it into the ground? We hope it will continue to exist, but we can't possibly know until then.
Please note that the linked URL is just a commit to remove support from code.google.com in go get, and add a warning to users trying to go get from it.
Also remember that while not very widely used, Go does provide support against this problems via custom URLs and canonical import paths [0].
And most importantly, whatever your programming language, remember to vendor your dependencies.
I just got into Go, and I'm a fan a lot of language design decisions. But a lot of configuration leaves me wondering: is it me who doesn't understand something, or did the same folks who designed such a good language actually screwed up configuration stuff?
No way to specify a version of library or language I'm working with is already a little bit annoying, although I haven't actually discovered problems because of it. But the $GOPATH just drives me nuts: why the hell would the language infrastructure work out of assumption that all my projects that are based on a single language are contained in a single folder? On the contrary, I often have projects that include several different languages and frameworks in them, and I'm usually keeping client and server in the same repo. Language is a part of a project, not the other way around. I ended up with this hackery in my server Makefile:
build:
GOPATH=$(shell pwd) go install main
Overall, just as Go itself is transparent and simple, it's `import` statement and build infrastructure is not. "Convention over configuration" may be a good approach for simple web apps, but when this convention just doesn't fit your workflow, it brings more bad than good.
My opinion: What they got right is what Rob e.g. concurrency, better C, etc., had figured out (so it was version n). Other stuff he apparently doesn't understand at the same level (e.g. [type systems, versioning,] hard coding repo locations in package imports <head smack>) ...
Could you elaborate more about type systems? I don't have a definite opinion about Go's type systems so far, and it surely has it quirks, but TBH, I like the concepts. In particular, the complete separation of data types (structs) and behaviour types (interfaces) is something that I've never actually seen before.
That horse has been beaten to death elsewhere. Go is [a] sweet language, don't get me wrong, but it is a mixed bag of extreme brilliance ('select') & head-scratching wtfs (OP).
Interfaces are hardly new. Go's novel take is the 'auto-morphism'[1] of types that (happen) to implement the function set aggregated in an interface. And this approach has its strengths and its weaknesses.
The idea with using $GOPATH is due to go having its own packaging tool; in other systems, you often see something similar (Maven for Java is probably the largest one), i.e. a directory structure somewhere. The $GOPATH makes it a lot easier to contribute on other packages though, you can just jump in there because it's right next to your own project/packages.
Well, in that department I think that NPM got it better than anything else — although I dislike Node.js and don't have a lot of experience with it apart from a couple of hobby projects.
I'd hazard a guess that it reflects Google's approach to dependency management. Internally, Google only uses a single version of any library. If you want to update your version of library X, you have to go around and make sure that every project that depends on X works with the new version of X.
I've flagged this editorialized title (Code.google.com was shut down, Go packaging is broken). Apart from being against the rules, it's intentionally misleading. It gives the false impression that Google Code shutting down somehow caused go get functionality to fail in general, which is false.
Go get works as always; of course you can't go get go packages on Google Code, because Google Code doesn't exist anymore! The link just points to a change that fixes some trivial Go test which didn't impact anything else; something that people working on Go have to deal with, not something users of Go have to deal with.
As for the news about Google Code itself, that was announced almost an year ago. The Go project itself migrated to git well before that, and of course, every Go package that's maintained also migrated off Google Code in the meantime. This is (misleading) non-news.
The "website going out of business destroyed my content" problem is one that needs to be addressed. One promising idea is content-addressable storage, like IPFS-hosted repos -- anyone can self-host at a stable URL. (I don't know if IPFS has a good solution yet to the problem branches solve in git, namely having a named, auto-advancing pointer to the content address of the latest version.)
Just host it yourself. People act like managing a server is akin to brain surgery these days. The 'cloud' is great, but this is what you have to plan for and deal with if you don't want to do it yourself.
They do! IIRC, the originator of the content can have a cryptographically signed hash that is a pointer to the latest version. That way everyone who is interested pins the originator's hash and automatically get directed to the newest content.
Google shuts down yet another service developers rely on. Its getting quite a bad track record. Trust is not so easily rebuilt. I think more developers will avoid depending on google services in the future if they have another choice.
Edit.
What happens to the thousands of webpages that link to jquery or google fonts on google.code. This must break lots of things ...
To be fair they did give ten months notice so people had time to manage the change, and what we are talking about here isn't "correct" use for that service anyway (it was for source and issue management, not library hosting or a more general CDN).
Doesn't help the fact that Google repeatedly shut down services that are in use. It is getting harder and harder to use Google services in business because it has become a liability.
"it" being "The idea they might shut down a free service with only 10 months notice".
If your company relies on Google to this extent, and does not have the staff to fix the issues inside 10 months then you need to reconsider your business plan.
AFAIK, Google uses code to host internal projects as well as open source projects.
Apparently some users were spamming and/or hosting illegal content for download. Instead of spending resources fixing what is essentially a free service to the public, they just shut it down and converted all repos to read-only mode.
At this point, I pretty much assume that any service that Google offers for free sans advertising exists to attract free beta testers. Eventually, it'll be converted to either a paid service or internal-only use.
Source: lots of time wasted migrating projects to GitHub.
I never really understood why would they map package names to the sites that host the sources. It was obvious something like this was going to happen and it's happening on smaller scale all the time. Projects move.
The idea was that people should never make API-breaking changes for URLs and instead offer only new interfaces or make a new package at a different URL. Funny coincidence that google itself broke that idea.
'go get' is a highly useful bag of cats. The fact that code.google.com needed to be hard coded rubs me in such the wrong way for what should be a generalized tool. I wish instead of magic self resolving paths, it just took git repo ssh/http paths.
I hate go packaging. I want to fork a package for my private purposes, maintain it in my hosted repo or somewhere private, but no, I have to change all the required import paths to my repo path. Or I have to provide it in vendor folder, or have custom gopath folders, but then go get does not work, I have to git clone and put it in some weird old repo's path. You could use relative paths, or package names but no, you had to use site urls so you could make it harder for anyone that does not work open source or private. I have not found a better solution for forking a repo and working it on my own besides vendor folder solution.
Vendoring external dependencies adds a lot of noise to your git history. Especially with github's UI it's easy to miss an important change because of skipping trough the vendor changes or getting the "too many changes to display" message.
Honest question: Isn't this approach one of the selling points of Go? The designers of the language decide what is the right way to do things, and then that's how it's done, leading to simplicity? It's not for me, but these restrictions are often pushed as a feature (others here have explained the reasoning in this particular case).
I'm right now TAing a course which uses Go. Tons of people are having trouble with this, and the interaction with $GOPATH.
This course has fourth year undergrads and masters' students. Most are familiar with some languages but aren't too used to production code practices (basically, one of the target audiences for Go -- be simple so that everyone can ramp up). Some have said that they preferred submodule deps or pythonesque easy-to-copy and easy-to-play-with import statements (and pip install) over this.
There is the valid argument to be made that any model for publishing your package is going to be tougher than a simple github URL, but it doesn't have to be much worse. With Rust, for example, you need to run `cargo login` once (which will direct you to a page where you can make an account if you need to) to set up, and then `cargo publish`. Pretty easy to learn.
I like a lot of things about Go, including it's simplicity. I wish I had more time to write Go, because it's a fun language to work with. But $GOPATH and URL-packaging are not amongst the Go features which I like.
An advantage of using C is its longevity. Go was designed to be a possible replacement/redesign of C (with Ken Thompson as an initial author). So this design decision doesn't make any sense, since URLs are some of the most volatile things in the programming world.
https://github.com/tools/godep should solve all these issues. Yes, it vendors packages. But it happens completely automatically and you can switch to your own forks at will.
Some apps/packages still use packages hosted on code.google.com. Basicly you have to update dependency tree for packages that were hosted there. If you vendor packages then you are not immediately affected but it still poses a problem in long term.
There is no central repository system in Go. No Maven, no rubygems, no packagist , ...
code.google.com/foo/bar package is the physical address were go get command will fetch the package. If that address is broken then the code is broken.
Go doesn't have a package manager either. Go Get is certainly not a package manager.
So when you use a third party package you have to fork it somewhere. The problem is that 2 libraries with the same exact code are different packages if their address is different. So sometimes it becomes a bit insane to work with third party libraries if people start forking them for safety.
I consider this whole package url thing as a design mistake. But most of the Go community sees no problem with it.
Isn't any packaging system reliant on a server and ultimately fragile if you don't vendor? I don't want to see an overengineered packaging system. Go is open source, has anyone looked at improving "go get"?
The Go maintainers have conflated what a library is and where a library is. That's an important distinction that should not be glossed over like this.