This seems pretty cool, from looking over the documentation. Two things I wish I could do that I don't see in the documentation:
- Replace an import with another import. This is used when forking a go package.
- Rename a package.
It's also pretty unclear from the documentation which operations are "UNIMPLEMENTED", since the label simply falls directly between two different operations. It would be nice if this were made more explicit.
It would be great if go.mod file’s replace directive also supported a replace that applies to all downstream users (url to url , not just url to local path) This would allow us to maintain simple forks (ones where you are only fixing a few lines and awaiting PR approval to upstream) but still don’t want to wait for the whole cycle to finish for your own internal company dependencies. I feel this is better than changing all imports.
If you are truly forking with no intention to merge back, isn’t the manual find-and-replace by IDE just fine?
It's important to note that it still doesn't apply to downstream users (those adding your package to their go.mod), only to those building your package directly. I think this decision makes sense, so this is not a complaint.
Assume module A depends on module B, and module B has a dependency on module github.com/C, and a replace rule for github.com/C => company.internal/C.
When you build module A, it will pull module B and module github.com/C, instead of B and company.internal/C, unless A has its own replace directive for C.
The problem I had was that I was forking a monorepo package, so I had to update its internal dependencies to point to the new place. It's possible I missed something, but I don't think I was able to do this with go.mod.
And yes, manually doing the rename "is fine", but I don't think that has any reason to be stated in a discussion about a refactoring tool.
> I don't think that has any reason to be stated in a discussion about a refactoring tool.
Find and replace is the baseline, and for many refactorings is insufficient and fragile, however in replacing a stable, unique, and isomorphic token it is basically as good as it gets, so I think it is relevant. Its probably the least useful thing a refactoring tool could do.
I am not sure. You want to make sure that you are only modifying import statements and those can’t necessarily be identified from just a single line (else you might change an errant URL in a string literal), and you also need to move the files themselves, and update the package line in the files. So sure, you can do it yourself, but why are we building a refactoring tool at all?
I used to joke at work that the fastest, simplest way to get promoted to a staff engineering position was to finish a 1 yr refactor.
Getting existing codebases to do new things is hard, and most of enterprise software engineering is basically "rework this API (in the general sense, not web-API sense) for a new product use case".
This tool seems to borrow the philosophy of Go's keeping the language simple, with the tradeoff of a more complex (but fairly well supported) ecosystem to account for missing language features.
IME, Golang (some would say paradoxically due to a lack of generics) is one of the best languages to refactor because of:
- Forced error handling semantics so that unhappy paths are easily enumerated
- Tooling like `rdep` to quickly grok the impact of a package refactor
- String templates/codegen being a first-class "blessed" part of the ecosystem (i.e. you can find tooling by the Go team as a reference when writing your own)
Some would balk at things like codegen being how Go handles this, but for the iteration loops at most companies at scale (where a few tools to do codegen are written & maintained by one team and consumed by others) it works well. For paradigms where you want everyone to contribute to the tooling, it works slightly less well since the barrier of understanding the ecosystem is greater than just understanding the language. Nevertheless, it seems to be the right tradeoff for most enterprise use cases.
I'm a Googler, and while that might be true at another place, unfortunately this is not true at Google. Refactoring/Tech Debt/Eng Excellence work will not get you promoted pass L4 for the majority of teams. In some team, it will get you to L5, but that's rare. Getting to L6 based on "1 yr refactor" is... simply extraordinary. I haven't seen or heard of one case in 7 years here.
I'm pretty sure that the comment you replied to was talking about "the general rule" and not "the exception to the rule." you don't need to call out that your specific employer is different than other employers. everyone here already knew that.
I’m guessing this was really just a way to brag about where they work. Seems oddly common among people that work at that company.
For what it’s worth, I’ve been at more than a few companies and this lines up. Managing a refactor or migration that’s long term and creates significant impact across an org chart or product area is definitely in the purview of promotion to senior or staff, whatever it’s called.
Given that the tool mentioned in this thread, rf, is written by Russ Cox, who works at Google, to help refactor go, a language developed and maintained by Google, how Google addresses these things feels not totally irrelevant.
I also think that "migration" and refactor are completely different concepts. And only the first is going to get you promoted (in the context of Google), because there's so much tooling for making even large refactoring relatively trivial. Pretty much every language has something like this, the build system also supports it too, so the process of, for example, taking a file, renaming it, querying and updating all dependencies (both source files and the build system), testing and sending out pull requests is essentially entirely automated. Unless there's something unique about a refactoring, it's not challenging enough to be promo-worthy.
Refactors are rarely as simple as moving a few file names or symbols around, at least where I work. While we do have some automation to help (and not break existing consumers), it’s significantly more involved in an SOA architecture for tier1 services. These sorts of big refactors also generally come with data model changes, performance profiling etc. This tool seems fantastic for parts of that, but not all of it.
Just as a friendly aside, your tone seems strangely defensive (though this may just be my fault reading this tired at 3AM) - Levels tells me you make on the order of half a million a year, while I’ll be lucky to make half of that in the next decade at my employer, so you really can’t feel persecuted by this when you most likely work half as much as me while being rewarded for it while folks like me struggle to tread water and are considered mentally deficient.
> Refactors are rarely as simple as moving a few file names or symbols around, at least where I work. While we do have some automation to help (and not break existing consumers), it’s significantly more involved in an SOA architecture for tier1 services. These sorts of big refactors also generally come with data model changes, performance profiling etc. This tool seems fantastic for parts of that, but not all of it.
Yes, I'd refer to that as a migration, which absolutely is a lot of effort, but a tool like this isn't going to be particularly helpful with a migration precisely because the majority of the effort with such a migration will be like...doing all the data stuff and traffic stuff. Changing the code is a minimal part. A refactor is often not a thing that even registers at a smaller company, because its on the scale of a few files. But "Large Scale Changes" are a thing when you end up having millions of lines of code (https://www.youtube.com/watch?v=TrC6ROeV4GI), and tools like this one are useful for those kinds of changes. But a single LSC won't get you promoted.
> Just as a friendly aside, your tone seems strangely defensive (though this may just be my fault reading this tired at 3AM)
Not at all. I was however, pointing out that "this was really just a way to brag about where they work. Seems oddly common among people that work at that company." was certainly unnecessary, and perhaps unkind.
Unnecessary? I sure don’t see people from my company point it out constantly. The superiority complex seems to be a Google only phenomenon in my experience. I guess that’s caused by constant external validation - perhaps it’s people like you that are “unkind” by flaunting your wealth and status while the rest of us have literally nothing.
The market has determined people like me are worth half of you or less. Please be cognizant of this.
Food for thought: there are people on this board that make an order of magnitude less, simply due to location lottery. Half of FAANG level is still far from nothing.
To be clear, I also work at a FAANG, doing very similar work but am consistently condescended to by these folks, literally everywhere from real life to online. The fact that I'm basically only worth 50% of them just adds insult to injury.
Sorry I should have clarified - by "refactoring APIs" I meant the sort of work that would e.g. be unifying all photo abstractions at Facebook behind a new service, not just refactoring package-level abstractions. That sort of work does involve a lot of symbol-level renaming and dependency hell management, but that's just a part of the work.
I feel like I've been involved in quite a few projects like that; Some Guy sold technology X to a company, so now it's all hands on deck.
Over the years, one company I've worked for off and on (as a consultant/contractor) went from a jQuery UI library, to a CSS component library (which was fine), to an AngularJS component library, then some ex-googler (younger than me) swept in and became CTO or similar and Decided that the whole company should use, drumroll please, Polymer. Then Polymer 2 came around and they had to scramble to make everything Polymer 2 compatible. They were just about done when Polymer 3 came around, which did things Differently once again.
But I'm sure the guy made decent scratch and left the company right before things came crashing down to an even more cushy job. That's usually what happens.
The whole idea of web components was somehow sold inside Google, hard. They even re-wrote Youtube with web components v0 which was almost immediately deprecated. And Chrome had to support it for years before Youtube rewrote stuff in v1.
Since then Google has abandoned Polymer and now has a new toy called lit-html. And is now busy writing stuff in lit-html.
Last year they shipped Constructible Stylesheets by default in Chrome despite the fact that both Safari and Mozilla were against the state of the spec and the proposed implementation.
The only reason? lit-html was going to use it. And the justification for not hiding it back behind the flag? "Oh, our internal metrics show 0.8% of sites using them", which, at the time, was exclusively Google-developed lit-html exclusively on Google-developed properties.
The sibling comment touches on this, but the class of tooling that lets you analyze (R)everse (dep)endencies. There's a bunch that do similar things, but examples:
Doesn't that just mean that, while Go the language is simple, to become productive one must not only learn the language but an ecosystem that is bigger than the ecosystems of other languages?
Relevant: https://haskellwingman.dev/ . it's actually possible to use it if you have the haskell language server installed, as a code action (at least i think it is, haven't personally tried it yet tho)
I hope more refactoring tools come into existence in the future. Right now, writing transforms similar to this complexity using Clang would require a fairly large amount of boilerplate code, implementing at least some AST matching code and wrapping it into a frontend action. While having this for Go is good, I hope it grows and inspires others to do similar things in other languages, the same way gofmt once did.
Since you're talking about Clang, I assume you're coding in C? In that case, Coccinelle [https://coccinelle.gitlabpages.inria.fr/website/sp.html] may serve your purposes, and has a long history of use in the Linux kernel to modify shared APIs.
In case of Clang (and more specifically libClang,) I am usually using it to do source-to-source translations on C++ code. In those cases, sometimes I need some fairly advanced knowledge, such as the type of an expression, to match on or perform logic on. However, this still looks quite interesting for some use cases (and I am interested in such tools.)
I think this is cool but I wonder what the expected API surface for editor integrations (with mouse-click menu selection to apply a refactor) would work. Would editors code-generate this mini scripting language? Or will there be a more abstract way of providing the same intention to the tool? If editor integrations are expected to send the script-snippets, will there be facilities for programmatically generating valid script snippets?
I've always wanted a tool that finds duplicate code (with minor variable name differences) so I can factor it out. Often when prototyping I'll copy a function thinking it will change a lot or maybe just not knowing what will change. As the application matures I'll go back and clean these up but it shouldn't be too hard for a tool to identify.
I wonder if this is some sort of precursor work for the generics release? Feels like gofmt -r on steroids, and could be eventually used to do large scale transformations to the standard library?
- Replace an import with another import. This is used when forking a go package.
- Rename a package.
It's also pretty unclear from the documentation which operations are "UNIMPLEMENTED", since the label simply falls directly between two different operations. It would be nice if this were made more explicit.