Yandex (and ClickHouse) does some really, really cool technical stuff.
I like that a lot of the stuff they write about and open-source has a fairly strong focus on efficiency + performance. I also find the supporting docs (those that detail the problem space, how it solves the problem, etc) quite well written and often a lot more comprehensible than docs from other mainstream tech companies (MS docs are the worst, followed by Google).
I’m not sure why this is the case though, is it a different development culture? Different project management culture (provisions time for documentation?) something else?
Unfortunately, larger tech companies in the US have become increasingly (in the last 5-10 years) political inside and a desire to ship things fast has ultimately meant building extremely high performance scalable infra that's foundational enough to be OSS-able is just not a priority since compute is cheap enough and it's not worth the cost of generalizing it to make it OSS. It's just very hard to pitch these kinds of investments. Documentation follows this pattern-- engineers just don't get promoted or rewarded for writing documentation and with IC's constantly being pushed to deliver faster and faster it just won't take priority. And that's a big problem.
In short, it's my impression it's an engineering culture problem in large part caused by bad leadership at the types of companies where these projects could be possible.
I recently had the displeasure of consulting a start-up whose pitch was a heavy technical infrastructure solution for other developers. I can't really give details, but their pitch felt to me like hearing about AWS in 2010... then I go into the nitty gritty of how they work. 0 engineering. Just rushed out non-fleshed ideas turned into one-line tickets and sent off to devs around the world without any instruction or guidance. Their employees were producing massive amounts of code, infrastructure, deployments etc.
Their concern was the low velocity which felt insane to me because velocity was huge but in an unfocused direction. I am scared of this new world where paper CEO think truly grand visions come to life by half-assing it and playing the part only to the surface level; like I get it if that's your thing, mooning a company and exiting, it's just so hypocritical to act like you believe in the mission statement when you can't even verbalize it, let alone push it forward.
I am hopeful for a new wave of Venture Capitalism with a focus on fundamentals and truly hard problems that are not going to bring in a 3000% ROI in 2 - 3 week-long sprints.
Playing devil's advocate, but why is that a big problem? large tech companies aren't altruistic and leadership isn't stupid. If shipping fast gives them a better competitive advantage than writing OSS, why should they invest in OSS?
It really depends on the perspective you're looking at the situation from. If you're concerned with better competitive advantages and the final bottom line for company X or Y, in this current market, in the short term, sure, that sounds amazing, sign me up on that moon rocket!
If your concern is a bit more hard to quantify, for example maximizing human potential, allowing individuals access to high quality technological infrastructure so that we may develop as a species and reach new heights, then you might feel like this approach is a bit short sighted.
Or even long term investment in the company. Look at where IBM is now vs where they were 50 years ago; Microsoft would be in the same place except for Satya Nadella completely reinvigorating the company. Who knows where some of the big tech companies now will end up in a decade or two given the shortsightedness that seems endemic.
"maximizing human potential" is not terribly specific and can mean different things to different people.
I'm not looking to be a shill in this thread but we might be what you're looking for, and even if it isn't, I'd love to keep in touch as my own life goals involve leveling humanity up in some capacity.
Happy to share details and answer questions if you want to shoot me an email (info is on my user page).
> so that we may develop as a species and reach new heights
unless the company doing this can reap exclusively those new heights, no one will be altruistic enough to make such an investment to the betterment of mankind!
This role is left up to the publicly funded institutions such as gov'ts.
This is /extremely/ uncharitable. It depends more on the project.
For example, both Go and .NET are very well written engineering achievements with thorough documentation. Many, many other deeply influential projects have also been absorbed by the Apache Foundation or other FOSS initiatives.
Furthermore, every software megacorp has boat loads of teams working on hundreds even thousands (!) of different open source projects. The quality naturally varies.
Let me pick two more esoteric projects as a point of comparison.
Yandex Odyssey [0] an advanced multi-threaded PostgreSQL connection pooler and request router. Figuring out how exactly and when to use this is not quite clear. There is no "getting started" guide for this package. There is barely any explanation for how it works or what it does.
pg_auto_failover [1] run by Citus (owned by Microsoft) monitors and manages automated failover for a Postgres cluster. This repo even has diagrams explaining the workflow and complete instructions.
Frankly, it's unreal just how much code is open nowadays especially from the giants.
Visual Studio, for example, had been under lockdown for 20 years. Now Visual Studio Code is being developed in the open and has extremely thorough documentation that walks developers through the entire build process in painstaking detail.[2] This is a blazing fast editor written in JavaScript of all things! Or consider TypeScript, an incredible engineering feat. There is an entire repo dedicated to engineering notes on how the compiler works with links to interesting feature contributions and a video! [3]
I'm only picking Microsoft here as the primary example since you dumped on them the hardest.
*No other industry does this. It is absolutely wild how much copyleft influenced software culture.*
I concur that Odyssey docs are insufficient to build a big picture. But I'm trying to answer every question in the Internets about Odyssey :)
Documenting is hard - everything seems obvious to me, but understand that may things are not that obvious.
// Odyssey maintainer
Yes, we have previous experience with products from the former USSR. Good product with basically no docs other than a 10 page PDF, just ask Andrey if you have any questions. We're integrators. One could reasonably expect to sell low volumes in Africa, because any European customer would basically laugh in your face.
Okay, you’re just being obtuse. The original comment was about the quality of Russian software documentation. That is unrelated to cyberwarfare, a subject I wrote about extensively, and earned recognition for. [0]
1) everyone knows you and what you have written and that makes you the authority (not true, obviously, and kind of makes you look like a top hat)
2) what you have written applies to the situation (not true; Russia is an authoritarian dictatorship where every business entity is potentially a state actor)
My point wasn’t to claim that I’m some sort of famous writer, it was simply to prove I have background knowledge in the subject and still disagree with you. The fundamental flaw of this conversation is you assuming that my disagreement with you is out of a lack of information, rather than simply reaching a different conclusion.
Certain MS documentation is actually incredible. For instance, the docs for the Win32 API are some of the best out there, rich with notes, examples, warnings, links to relevant topics. For instance, here's the page for CreateWindowEx: https://docs.microsoft.com/en-us/windows/win32/api/winuser/n.... Having never written Win32 code before, I was able to get a window open complete with simple rendering from a bitmap backbuffer in about two hours using nothing but those docs.
Most of the more recent library documentation they have is not great though -- seems like autogenerated junk, mostly.
In my experience, documentation quality for closed Yandex projects ranged from excellent to absent. It is definitely not project management culture. Perhaps, Yandex tries to use its open source projects as kind of advertisement targeted to developers, and puts extra efforts into its quality.
Google's product naming. Ever tried to search for Google Assistant, Home, Auto, Sheets, etc...? You would think Google of all places would be very aware of useful it is to be searchable.
I like that a lot of the stuff they write about and open-source has a fairly strong focus on efficiency + performance. I also find the supporting docs (those that detail the problem space, how it solves the problem, etc) quite well written and often a lot more comprehensible than docs from other mainstream tech companies (MS docs are the worst, followed by Google).
I’m not sure why this is the case though, is it a different development culture? Different project management culture (provisions time for documentation?) something else?