> Another important reason for software complexity lies in monolithic design, wherein all conceivable features are part of the system's design.
This is really a call for modularity. I'm a believer in modularity, but it is not a cure-all.
Modularity brings a lot of baggage with it, like duplication of code and lifecycle management, along with increased resource usage (ironic, as the article is mainly about reducing resource usage).
But I am still a believer in it. In my experience, it makes for a massive increase in system stability, quality, agility, and speed of development.
Maybe I'm misunderstanding, but it sounds like you talk about modularity at the unit of deployments, i.e. separate services.
But you can have modularity within a single codebase and some languages / build-systems allow you to define very clear boundaries between your different modules. In that case, I don't think you have necessary a lot of code duplication (you might have slightly increased consumption due to the use of more indirection). But you might have to code to interfaces and explicitly eliminate coupling etc.
Oh, no. I use it all the time, for code components.
If you look at my portfolio[0], you’ll see it packed with things like widgets and adornments. Most of these, I use in my own projects. I eat my own dog food.
But today’s languages, with their reliance on extensibility and interfaces, makes writing standalone code assets a lot less cumbersome. I write in Swift, and there are some pretty big Swift "extension suites" out there[1]. The Apple Swift Package Manager is nice, too[2].
It's just that when I write a code module, I do it as a standalone product, with a full test suite (usually a harness[3], which often takes the form of a "full-fat" application), and a unique release management cycle.
My modules are pretty independent, so they aren't usually custom-tailored for all of their implementations; acting in a fairly general capacity.
If the "modules" are source code you don't know or can't control, then it's a dependency that can create dependency headaches. For example, a new OS or browser version comes out and the existing components are not compatible with the upgrade. Happens all the time. External components are thus not a free lunch.
What's really missing is the willingness to say "no". There's pressure to put fancy or trendy toys into applications to look up to date or to get experience on the latest fads. KISS and YAGNI are often ignored for eye candy or Fear of Falling Behind.
For example, fairly early in the 2000's I had a web form that only did server-side date validation. It worked just fine. But client-side date validation was becoming "expected" because everyone started doing it and the manager asked me to put it in. So I found a JavaScript component on the web, and it worked fine at first. But a few months later a bug was found in it that appeared on a certain date, and it generated a lot of phone calls from users. Those phone calls would have never existed had I gone with my gut and left it out. Few users are going to spend all day making date errors such that client-side date validation is NOT necessary. It's a me-too toy.
Warren Buffett often cites his ability to say "no" as to why he's so wealthy. He looks at the actual tradeoff numbers, not peer pressure.
Modularity is, by definition, a feature of the code. No feature is free. Some do have major benefits. More often, some are required by tradeoffs or circumstances of deployment.
> Modularity brings a lot of baggage with it, like duplication of code and lifecycle management, along with increased resource usage (ironic, as the article is mainly about reducing resource usage).
I just can't conceive of how modularity, in general, causes duplicate code rather than reduces it. And this is before considering something like generic code which could potentially further reduce duplication. Every time we've been deliberate about using modularity in projects it ends up creating libraries that reduce the total amount of code across projects. Even if it's just in one project, good modularity has been used to reduce a lot of duplication.
As I mentioned above, if the module is expressed as a code extension, then it is, sort of, "free." It is compiled into the main code body.
But, for example, say we are talking about a communication module. One module may use a well-known library as a bundled dependency, and another module may use the same one. They may be smart enough to use dylibs, in which case, you have the potential for "DLL hell," or they may use static libs, in which case you have duplication of code, and maybe resource contention.
I would not consider static libraries to be duplication of code. Duplication of assemblies or compilation unit, sure, but not code. The code itself has not been duplicated.
Modularity doesn't cause that. Modularity doesn't inherently reduce duplication either. But modular code offers opportunities for reuse (or clearer opportunities for reuse) that non-modular code does not (or where the reuse potential is so obfuscated that it may as well be non-existent).
Others have already pointed to the distinction between code duplication and runtime duplication.
Regarding the development experience:
I think it's a tug and back-and-forth kind of dynamic.
Modularity has exactly the kind of effect you talk about, until it doesn't.
At some point, the cost of communication (between people!) responsible for different modules can increase until it's a net negative to try to adapt a module, or even find out it exists and is suitable.
Of course, a monolithic project of comparable size has likely collapsed from its weight into a ball of mud.
Development tools have come quite a ways, but the science of getting teams to develop software together, perhaps not so much.
On an idea level, i'm all for DDD (domain driven design) and many of the ways to achieve modularity - be it at code level, like in the new versions of Java, be it on a library level, with shared common code, or maybe even having a hexagonal architecture, or microservices.
That said, in reality, doesn't any sort of modularity imply increased complexity?
- you need to manage your modules and figure out where to put the code
- you need to figure out mechanisms to share and reuse it, as well as test every single integration (breaking changes will be harder)
- you need to invest significant resources into the Ops side of things - maintaining multiple deployments and making them talk to one another
It seems like it would lead to trading one set of constraints for another, each having their downsides. In my subjective opinion, monoliths are the only way to go for small teams (~1 pizza) and relatively small systems (~10k MAU), maybe migrating over to modular monoliths as time goes on, and only then branching out to other approaches.
Regardless, it feels like the industry perhaps jumped over the concept of modular monoliths entirely and went straight for microservices.
Of course, if we're talking about modularity purely on the code level, then dynamic loading of code is its own can of worms, as can also be many methods of installing extensions.
Because of the slightly tongue in cheek style. It's something that would fit better in a Medium article, perhaps.
That's not to say that everything on HN needs to be cut and dry, but factual discussions seem a bit more productive, in my experience. To that note, having any stats on the prevalence of modular monoliths would have been awesome, or maybe even more information about feature flags - those are often not discussed all that much!
One thing I observed as a data analyst (prev job) was that modularity could make data anlysis pretty difficult.
Example:
In a monolistic application you can easily send out telemetrics of different "components", e.g. player hitpoints, coins in inventory, days in game, etc. But in a microservice structure (at least the one I saw), since hitpoint service doesn't care about coins, and coins service doesn't care about days in game, to get all three in one telemetry, developers have to break the barrier and connect them in one way or another.
So essentially you still get to write them into sort of monolithtic, but uglier and needs more arguments (but still have to write them at the end because CXO needs the report).
Microservices are not the only way to modularize. But yes, cross-cutting concerns tend to wreck modularization; in desktop environments it's usually global settings or events that cause this.
(Long ago, the software industry wanted to copy the electronics industry, where people would assemble software from components listed in catalogs. We never quite got there, but you can see the influence of this metaphor all over the place when people talk about "components". It required the development of libre+gratuit Free online libraries to get there, causing an absolute Cambrian explosion and the left-pad problem)
Most things are cross-cutting concerns. For example, database access and security are issues that entity-specific modules have to deal with, and it's best to share those features rather than have every entity reinvent it.
Early in the GUI days, you had UI components that were pretty independent. But now they are expected to match the look and feel of given OS versions or brands such that they can no longer be truly isolated: they are expected to integrate somewhat with the OS. If people had the guts to live with "ugly", a lot of things could be simplified. I discuss saying "no" to the IT Fashion Show nearby. It may read like a git-off-my-lawn rant, but software could be cheaper if you kick fadsters off your lawn. Beauty ain't free.
That's an interesting take. My observation is quite the opposite, with each application seemingly overriding systems style willy nilly. I'd say that application styling has become MORE differentiated.
This is what api's are for. A telemetry service asks each service for the specific data, combines them and outputs the combined metrics. This doesn't really break a barrier, but it does increase overhead.
That's what data warehouses are good for though. So long as you have something you can join on, you don't need to worry about how the application developers structure their systems.
Theoretically it's true (but again the burden is simple transferred to someone else).
But in reality, as a BI Developer (prev as a data analyst), what I really don't want to do is a join by timestamp (and a userId), pretty much the only way to join two "subsystems" that are not interconnected.
Example:
Analysts want to know what's the coin balance when player starts purchasing energy bundles. Now that they don't know about each other so the telemetry only has one common field: userId. Eventually either BI developer (who develops DWH/View/etc.) or DA has to write a join by userId and timestamp to get the most accurate coin balance when player purchases an energy bundle.
It's not that easy. Join by what? In a lot of my work I have to join by common userId (sure) and timestamp which is pretty heavy. Something like this:
SELECT
*,
ROW_NUMBER() OVER(PARTITION BY userId ORDER BY timestamp DESC) AS rowNum
FROM
A LEFT JOIN B
ON A.timestamp <= B.timestamp and A.date = B.date
And then write another query to take the top row based on timestamp, and for sure this method may lose some data (e.g. what if userId doesn't have data in table B that day?).
(warning: old man rant ahead, apologies in advance)
My original answer was perhaps a bit terse and should have been more along the lines of "that is what a good schema design should fix". Your original position that "microservices make things more difficult for analysis" is absolutely true, but I guess my observation is that while you might get lucky and have everything you need in one place in your raw data, for any sufficiently complex system (especially if there are 3rd party components involved) you're going to need to do some real work to transform that data to build a repository with a flexible schema to support your analyses and make yours and your colleagues work easier and more productive. You are 100% right in that every layer of indirection makes this slightly more difficult, but unfortunately that is something you need to design for in all but the most simple systems - this is true today the same as it was true 20-30 years ago.
Hopefully you're already doing that sort of design, but my experience is that disciplined schema design for effective analysis is incredibly rare, despite it being an old, crufty, well established and documented area ala Kimball and old school Enterprise Data Warehousing and BI. Not all of it translates perfectly to the world of Big Data(TM) but the fundamental concepts of designing such that your data can be turned into easily queryable and useful measures and dimensions still very much applies.
For your example, the correct correlation ID completely depends on the context in which you need to do your analysis, and that should be ideally baked into your telemetry schema where applicable (and where it makes sense from an architectural/performance/data volume perspective), but if not, then baked into your intermediate ETL schema such that it is easily accessible for analysis. Perhaps it's a contextual or global incrementing index that more easily allows you to get the "latest" transaction per context (maybe a session or a match in your game, or an encounter, or outside of games, a shopping cart checkout flow), maybe your client passes more contextual information to the microservice to allow you to more easily hierarchically partition into your context. Maybe if you don't have that level of control downstream (like in a 3rd party service for example) you have an ETL or streaming job generating the global or local context from the raw telemetry that makes it easy to join against.
And lack of context for analysis isn't even just a microservices issue, it can even happen in any kind of modular software - in the game I'm working on right now, for example, the crafting system has no need to know about overall state from the player, hence the crafting module has no (easy) access to player state that might be relevant for downstream analysis - so even in what would be considered a monolith, getting access to player health/armor/xp/etc in the context of crafting analyses (e.g. do players craft medkits at low health more often than at high health?) is difficult and really uglies things up, much like you imply. But making available a playerStateIndex that can be used to join back to that state (managed by the player state system) is doable and introduces no real problematic module coupling, and is a repeatable and extensible pattern to use across other modules and even external services.
I've spent the better part of half of my 20+ years in the industry doing some form of data analytics, and my message to the analysts/data scientists I work with is always "if you have to do crazy gymnastics to do your analyses then our schema design needs work". You don't want to be in the (all too common) world of analysts spending 80-90% of their time doing data transformations and cleaning and 10-20% of their time doing analysis.
>if you have to do crazy gymnastics to do your analyses then our schema design needs work.
Yeah I figured out that if the query is over say 50 lines long and has weird marks everywhere (say DISTINCT with some windows function) then it's pretty much a bad database design.
Then I move to the data side and figured out that it actually also has something to do with the original raw data sent by the app.
So eventually _someone_ has to take the burden and do the gymnatics and frankly it totally depends on who is closer to the CXO. In our company the developers are in favor so they can do whatever they want.
>Hopefully you're already doing that sort of design, but my experience is that disciplined schema design for effective analysis is incredibly rare, despite it being an old, crufty, well established and documented area ala Kimball and old school Enterprise Data Warehousing and BI. Not all of it translates perfectly to the world of Big Data(TM) but the fundamental concepts of designing such that your data can be turned into easily queryable and useful measures and dimensions still very much applies.
EXACTLY what I'm feeling now. We are trying to work out a DWH on top of the raw telemetries (basically some simple transformations and routing done by Spark from HQ). However I really think that Kimball doesn't fit very well with the big data thing. I don't know how to say it clearly because I just moved to the data team for a few months so I'm a total newbie and everything I _feel_ could be 100% BS. I think we should start from business requirement (i.e. from our data analytics lords), and then bend the rules (Kimball will require a LOT of extra joins if what I'm reading it correctly) and keep the DWH tables sort of flattened but in a way that makes more sense than the original raw telemetry tables.
I'm not in control so I don't know what it will result into.
>For your example, the correct correlation ID completely depends on the context in which you need to do your analysis, and that should be ideally baked into your telemetry schema where applicable (and where it makes sense from an architectural/performance/data volume perspective), but if not, then baked into your intermediate ETL schema such that it is easily accessible for analysis. Perhaps it's a contextual or global incrementing index that more easily allows you to get the "latest" transaction per context (maybe a session or a match in your game, or an encounter, or outside of games, a shopping cart checkout flow), maybe your client passes more contextual information to the microservice to allow you to more easily hierarchically partition into your context. Maybe if you don't have that level of control downstream (like in a 3rd party service for example) you have an ETL or streaming job generating the global or local context from the raw telemetry that makes it easy to join against.
It's...not really easy to come up with an intermediate "common context" id sometimes. Session Id doesn't work because each session takes a lot of time and nothing else really are real-time. Since I'm the only guy in the company for requirements taking (I really think developers should have their own business analyst), it's pretty painful that each time I have to fight both sides. I have to ask the DA team to relax their requirements (realtime? No it's not possible, how about same session?) and ask the developers to bestow more connection fields (if you have this field DA will be really happy), thus the vent above :D
Keep in mind that modularity is a complex notion: the space of components can be rather non-trivial, in that the dependency graph can have a pretty interesting topology - if you include, in this notion, for instance, inheritance, generics (templates), and all the related usage patterns that have been invented over the past 40 years. (I believe Modula and Oberon both failed to address this complexity, having restricted the programmer's understanding of the concept to a very basic use case.)
Exactly. But these days, everyone likes to bundle in dependencies.
Also, for cross-platform SDKs and libraries, the provider may be forced to "reinvent the wheel." That may well be worth it, as developer bandwidth, and time-to-market are important variables.
Yup, but these have become so common, in the industry, that they are now synonymous.
For example, I no longer trust people that describe themselves as "iOS developers" to know the first thing about actually developing for the Apple ecosystem. In some cases, they don't even know how to use Xcode.
I'm convinced that the interesting stuff happens at the interface. It's just like for neural networks: the information bottleneck makes each tool distill information inside its output.
1+1 > 2 when multiple people cooperate, thanks to this. I think the same can be applied to software. It really has plenty of advantages: code more audited, swappable parts, common interfaces, standardization on data exchange formats, testability of the components.
I've found the part where the author talks about "arbitrarily overlapping windows" and "fancy icons" as "cute but not essential" pretty telling. Find me someone on the street and ask them if these things are "cute but not essential". Sometimes I feel like software engineers forget what world we live in. In the virtual world all these things are obviously not needed and wasting precious computing power on them is a disgrace to the "craftsmanship". In the real world most applications are made for flawed humans with needs and preferences which stem not from logic and reasoning but from emotions and experience. This doesn't excuse shoddy engineering in any way (everybody has some horror story of somebody not understanding runtime complexity or something, so there certainly is a lot of it). But especially with the examples the author brings up like text editors and operating systems I find these critic/plea severly lacking. Are the core features of ed and visual studio code the same? Absolutely, you edit text. Are the extra ressources used for visual studio code compared to ed a waste? For some people here probably, but I would bet quite a lot that for most the answer would be no. In the end I often find that discussions about optimatization are often more for people to show off and prove that they are a "real engineer" than any real concern about performance.
Is it really telling? A couple of things to consider:
- The original Macintosh did not use icons as heavily as software in 1995. From my vague recollections of 1995, the computers of today may even use fewer icons. (It was a time when there were icons on buttons, like a check-mark on an Okay button, and icons on individual menu items.)
- Even arbitrarily overlapping Windows seem less common these days. People have been maximizing applications for ages. These days, tiling is increasingly popular on desktops (e.g. Windows Snap) and people typically run apps full screen on mobile devices (with split screen, rather than floating window managers, being the only alternative). It's also worth noting that 1995 was part of the era of multiple document interfaces, which were arbitrarily overlapping windows inside of arbitrarily overlapping windows.
Wirth seems to be taking a far more extreme stance than the examples I have provided, yet the early years of the 1990's were also an era of excess. While 1995 Wirth would probably strongly disagree with the software of today, modern software looks austere compared to some of the software of the 1990's while the performance of modern hardware means the user is going to feel less of an impact due to the user interface.
>It was a time when there were icons on buttons, like a check-mark on an Okay button, and icons on individual menu items.
tbh I wish we would go back to this. Even with the variation in styles, those icons provided language-agnostic "names" for things. If you're using a computer stuck in a different language / with different layout patterns, a checkmark in a primary position is much less ambiguous than having to guess if this is a yes/no/ok/cancel/abort/retry/fail/ignore of some kind.
Unfortunately, checkmarks aren't language agnostic - abstract iconography as a whole is a linguistic phenomenon. I'll just copy paste Wikipedia's notes(edit: HN breaks on the actual symbols so I'll link too):
It is common in Swedish schools for a to indicate that an answer is incorrect,[2][3][4] while "R", from the Swedish rätt, i.e., "correct", is used to indicate that an answer is correct.
In Finnish, stands for väärin, i.e., "wrong", due to its similarity to a slanted v. The opposite, "correct", is marked with {\displaystyle \cdot \!/\!\cdot }\cdot \!/\!\cdot , a slanted vertical line emphasized with two dots.[5]
In Japan[6] and Korea, the O mark is used instead of the check mark, and the X or mark are commonly used for wrong (alongside a triangle △ for entries that are problematic but not wholly incorrect).[citation needed]
In the Netherlands a 'V' is used to show that things are missing while the flourish of approval (or krul) is used for approving a section or sum.
In Brazil, the check character is used to indicate the "yes" concept. For example "yes, it was checked" or "yes, this is the right answer".
Fair, there's next to nothing that's actually universal across all languages / cultures. I do always find the variations interesting though, thanks for the link!
Do you disagree that they aren't essential? Can you not do the computing without them?
Obviously you can, although you may not want to. That's exactly the point of the article. You WANT arbitrarily overlapping windows, but that comes at a cost. You HAVE to pay that cost if you WANT the feature.
In the very next chapter, the author even says as much: "Increase in complexity results in large part from our recent penchant for friendly user interaction."
What makes such a realization interesting is that if you don't need the "friendly user interaction", say you're making a POS system or some embedded device, you don't need all that extra complexity.
A lot of POS systems run full Windows systems. Do they NEED arbitrarily overlapping windows? If not, then why do they carry around that extra complexity?
"Essential" is such a loaded term, and talking about it can easily get us to a slippery slope. Like much else, this should be taken in context, and contexts abound.
For many use cases the (excellent) interface provided by, say, AS/400 (a.k.a. IBM i) is totally sufficient, but this fact alone does not mean that something like that should be used universally.
>What makes such a realization interesting is that if you don't need the "friendly user interaction", say you're making a POS system or some embedded device, you don't need all that extra complexity.
Why would a POS system not require a friendly user interface.
This attitude towards users is why corporate software stinks so much, it's based on the idea that the developers have a captive audience so have no choice in the matter, they should just accept what they are given and stop complaining.
There's a difference between a "friendly user interface" (which is pretty clearly referring to the earliest points about design-heavy skeuomorphic icons and complex window management functionality) and a friendly, e.g. accessible and functional, user interface.
A POS system doesn't need window management, it doesn't need skeuomorphic design, and I'm sure there three or four dozen other features that come along with a full-blown Windows installation that are irrelevant to a functional, usable POS system.
The running example in my comment was overlapping windows. a POS system might not need windows, much less arbitrary overlapping ones. That was the specific "friendly user interaction" i was talking about.
I think you're taking me a bit out of context in order to talk about something you care about. I agree with you, so please don't hold me up as something I'm not.
For context: keep in mind that the personal computer that Wirth and team built for ETH Zurich, project Oberon, was graphical, but eschewed overlapping windows for a paned interface and didn't buy into the skeuomorphic desktop WIMP metaphor.
So "arbitrarily overlapping windows" and "fancy icons" as "cute but not essential" may be telling, but is probably not telling what you thing it does.
I agree, taking these quotes out of its time context is a bit silly but then again this is the implied intention of posting this here, right? Most people reading this and posting a comment will not be talking about the state of software in 1995 but today, using the article to show that the things described in it are still true/got worse. My comment itself was more of a quick reaction to the general sentiment in the comment section than a serious critic of the article itself.
Since I'm a huge fan of tiling windows managers and spend a lot of my time in terminal emulators as a power user I believe to understand his vision about computer usage and have a lot of admiration for what he and his colleagues were working on. But for the general population I believe he did not have the "right" idea (easy to say with almost 30 years of hindsight).
It also should be noted that Oberon's UI (which you can try out by downloading the emulator from here[0]) might be an initial culture shock if you are used to a Windows-like interface, but when you learn how it works it actually is both very powerful and very consistent - which helps into usability since everything works the same way.
AFAIK Oberon was actually used by "regular people" at ETH Zurich for a long time.
(and a lot of people ran and still run their programs maximized anyway)
> Why did Amiga, the series of computers beloved by artists and musicians, have overlapping windows and colorful icons?
To be fair, these things were computationally intensive at the time. The Amiga had a hardware blitter and very good color capabilities, so it could afford to show them off.
The creators of Amiga introduced them, and added hardware support to make it performant. The only thing which felt sluggish in an Amiga was the floppy disk.
Likely these creators understood the importance of overlapping, draggable, resizeable windows for the audience, and the audience indeed enjoyed them. Same thing with the classic Mac.
If you tried a tiling WM, you realize that it requires learning, tweaking, and discipline. Those are expensive qualities to require from users; one should ask for limited amounts of them.
Overlapping windows also require learning and discipline, perhaps more so than tiled screen splits. We are just more used to the former than the latter.
Amiga may have had some overlapping windows within a screen but it also had multiple screens with significantly restricted arrangement options, if memory serves me right.
This is pretty much the standard issue "Ignorant/arrogant code jockeys have no idea how alone and scared us users are, and our need to be mollycoddled" rant.
It's Niklaus Wirth. He's the creator of Oberon, a language/OS/development environment which, while a bit idiosyncratic by our standards, is far more powerful than ed.
Oh, and while we're talking about the "real world", why do we never discuss the data entry clerks, sales associates, and customer service reps who learned to operate terminal-based F-key-driven UIs quite efficiently -- and were then forced on to graphical MFC, Java, or Web-based UIs with little thought given to efficient operation with the keyboard alone? The standard Macintosh-based WIMP paradigm is great for onboarding new users, but it's less good for efficient use by a trained operator when speed or volume is important.
Personally I find overlapping windows and icons to be more essential than... ugh... composited desktops. Translucent windows, drop shadows, and the "genie effect" are fun for half an hour tops, but these desktops consume additional processing power and memory and introduce latency! I'd rather go with a 90s style stacking WM.
> Translucent windows, drop shadows, and the "genie effect" are fun for half an hour tops, but these desktops consume additional processing power and memory and introduce latency
There's certainly some room for optimization in how desktops work today. I can see the obvious case for tear-free (v-synced) compositing when dealing expressly with full-screen (or close to full-screen) animation or full-screen video, since screen tearing is highly noticeable in these use cases. Wrt. everything else though, there's nothing wrong with rendering without vsync and avoiding that added latency. These optimizations can however be hard to make precise beyond very simple cases, like a single foregrounded full-screen app being enabled to take over the whole display. Tiled windows might be the next most obvious step.
Icons and square-ish buttons are convenient in mobile UI's because they make for good touch targets. At the same time, arbitrarily overlapping windows are replaced by tiled screen splits for much the same reason. So what's 'essential' will often depend on what the software is being used for.
It is also common trope that people say "but consider the typical user." I think both arguments are valid. At differing points in user skill level.
For example I would not expect a "typical user" to know or want to use a terminal to navigate directories, however as a more experienced user I know one can navigate much more quickly using a terminal than using the hunt and peck style of most file system GUIs.
Some things are useful before they can be made easy. Some things are easy that have more uses with more effort.
A good TUI-based file manager is way quicker than typical GUI's while still preserving much of the same intuition. Whether it's also quicker than the CLI depends on your use case; in some cases it can be.
Very many ideas of Pascal, Modula, and Oberon live on in the lean (to me, excessively lean) language named Go, which got pretty popular during last decade.
This is such a classic paper that I felt it deserved a more readable copy. So I copied the text into Word and cleaned it up a bit. May be easier for those with visual impairments etc.
I don’t know how you would do this, other than setting core affinity, but I’m surprised I can’t limit processing power or RAM per process on any modern OS.
Both for setting restrictions per another comment here, but also for testing purposes.
Like, I’d love to be able to run an executable with equivalent processing power to that of a specific CPU generation.
On macOS, there is a legacy CPU throttler called AppPolice [0] which as of right now still works on Catalina, despite the last release being made in 2016 (!). The user experience is very flaky, but it can help out when it's really needed. I haven't tested it on Big Sur.
Definitely agree though, this is something that should be baked into the OS as a first class citizen, especially in the modern era of Electron-all-the-things.
I mean, some software can really get rid of bloatware without losing much functionality. For example: The AMD Adrenalin driver takes about 1GB of disk space and comes bundled with a browser. The control panel takes forever to start, even on a laptop with NVME SSD. This is why I have switched to installing drivers through Windows Settings.
> Time pressure is probably the foremost reason behind the emergence of bulky software.
Indeed, but for startups now I'd add easy scaling. With the public cloud why worry about efficiency now (vs time to market) if you can easily scale up later.
One of the key quotes:
> Another important reason for software complexity lies in monolithic design, wherein all conceivable features are part of the system's design. Each customer pays for all features but actually uses very few. Ideally, only a basic system with essential facilities would be offered, a system that would lend itself to various extensions. Every customer could then select the extensions genuinely required fora given task.
Such a system sounds nice in theory. In practice it leads to a byzantine nightmare of configurable interfaces, support systems, and so on, with a result that is ultimately fatter and harder to maintain.
As an user, all the software that I take actual pleasure in using and make me productive, are large plug-in based software.
As a developer of a mainly plug-in based application, the iterating ability it gives is pretty incredible - nowadays for a large majority of my feature requests, I don't have to touch the original codebase at all, just create a small plug-in for the feature.
That was my first reaction, but I think it can work in some cases. Take for example plugins in a paint program, or dissectors in wireshark. Both of those can have any number of user-created extensions for different packet types or editing that the general populace doesn't need.
However, to your point, when wireshark changes their API every few versions, all the plugins have to change as well or you either break your dissectors or are stuck on an old version. The base program should be designed with extension maintainability in mind.
I don’t know. It sounds similar to a lot of IDEs. Produce an interface that plugins can make use of, then people can extend the system to support new languages and capabilities. Seems to work well in a lot of cases.
If the interfaces are defined poorly, it can result in what you are describing.
But I've seen systems do this well and the result is something far less complex.
I agree with the author's view that good interfaces require iteration. So expect that several irrational will have incorrectly drawn boundaries and poorly defined interfaces.
This kind of design takes time, but if you expect the system but if you expect that it will need to evolve over the years, it pays for itself.
plugins works for things that are plugins. Most people seem to equate separation of concerns with everything should be a plugin. Otherwise you get massive issues actually working across each plugin (or package).
So unless your component is actually used in multiple core products it's not a component, it's just a part of your monolith. Making it a separate component usually, just spreads your monolith across several git repo's
Emacs is a system that adheres to this theory, being a small C core that implements an elisp runtime, and then the rest of the built-in functionality (as well as the entire plugin ecosystem) implemented as easily-tweakable elisp on top of that - but has not turned into "a byzantine nightmare of configurable interfaces, support systems, and so on". It's quite the counterexample to this claim.
I'd argue that emacs is more monolithic in is main focus of text editing and elisp execution. It doesn't just build a plugin architecture. It literally built a platform and does most of it's work in the same way that user code can run.
Well it depends on the level of abstraction you use. You could think of the operating system as the platform, and each piece of software as a plugin. That can work well enough. I can save a PNG in photoshop, and then open it in other programs and it just works. But the file paradigm has lost popularity and there are probably some good reasons for that.
I know that when I initially started setting my system up, it was a bit of a mess trying to figure out what USE flags I needed/etc., but once it stabilized, it was great to know that I had a system that worked and there was no trace of software or libraries I didn't want or need.
It definitely is a bit more legwork to do up-front, but I feel like the idea of "this is what I want, please build only to that spec" has saved me a bit of heartache as I've been using my system and while the leanness doesn't matter per se, it feels like it's helping my system be a bit more organized.
This is why when iOS restricts apps from eating up the entire physical RAM on the device, and developers complain that this is not like on the mac/PC, they should realize Apple is extremely familiar that it's not. And it'll never be. When you let go of restrictions, applications bloat up to take the available resources while adding almost nothing in terms of functionality.
It's one reason why iOS had five years head start on Android when it comes to performance, perceived or otherwise, even with much slower hardware - they made much better use of it. Background apps? Yeah no, if you need any background activity, use a push notification service. Whereas on Android, for years it was the norm to just have your application run in the background and poll every once in a while.
You can put almost infinitely much effort into optimization. When do you stop? Do you test every user-interaction takes less than 100 ms below which it can be considered good?
It’s not as easy as that. What’s acceptable in test might not be acceptable in your customers conditions. It’s difficult to correctly measure the time for every interaction and humans are extremely bad at telling you how fast something is. Then you also have the problems of regressions that stem from other unrelated changes.
I believe it can be made easy, if you have an easy way to profile your GUI. Test the feature, get a timer. If it took more than 20ms on your killer test machine, it might be too slow for some users. (Adjust your heuristic with the context of your application of course.)
In my "software development" course at university (so, not programming, but every "management" thing around it: unit testing, git, waterfall/scrum, etc), I remember the instructor telling us that the first approach to making your program faster was upgrading the hardware.
The more time passes, the more shocked I am at an instructor telling this to his students, to the point that I'm now thinking I might recall a cruder version of what was originally said.
In any case, that may be the cause!
To some extend I don’t disagree with: “Just buy a bigger server”, but for software for laptops, home PCs or mobile it’s not a great option. If you’re trying to tweak every single query to get more performance from your SQL server, then maybe the money is better spend on hardware than developer time.
Weird counter point to “buy more hardware”: I worked with a client who have some code they developed, it takes a dataset, run some calculation and returns the result. To speed up part of the calculation they use CUDA.
We help migrate they system from older physical hardware to AWS, pretty standard. They got more memory, more CPU cores and jumped three generations of Nvidia GPUs. The result was 10% speed increase, no where near what hardware alone should have yielded, so clearly in this case there’s something in code that prevent the hardware from being fully utilised.
We do more things, and have more complex graphics.
Eg, a 320x200x8 display has 64000 bytes of memory. A 4Kx32 bit display has 33177600 bytes of memory. That's 518 more times memory used just for a single screen. Take into account that these days we have compositors and each window is present in memory on its own, before it's composed into the result, and it's even worse.
We also work on far bigger projects. The project I work on as a hobby has 400K lines of C++. Imagine the amount of processing an IDE needs to do to parse all of that and provide its various convenient functions. And of course a good chunk of that is in RAM, because that's far too much stuff to parse everything every time.
Old tools also don't really cut it anymore. If I try running ctags on it, it comes up with a 14GB tags file. I don't think it handles it quite right.
Tell a Product Manager "I can do a preliminary version of that feature for the next release, but it'll be really inefficient" and they hear only "I can do...that feature for the next release."
It's so much faster to develop wasteful software. It's also a lot easier.
There's no need to use high-performance, but difficult to develop languages (c, c++). There's no need to be conscious of data structure choices. There's no need to profile code, etc.
Optimizing code can be a deep rabbit hole. If you remember a few months ago, some random person dug into why GTA Online was taking longer each day to load. The reason turned out to be a string parsing inefficiency in the strlen method, a core library. But it only manifested itself in certain situations.
“Premature optimization is the root of all evil”
- Knuth
So the only exposure most students get is not to do it. I doubt any CS programs make performance analysis or optimization a requirement. Just buy a faster system.
Human effort and attention span are still limited quantities. There are so many more libraries, frameworks, utilities, and doodads, that it's easier to plug something together from lego blocks rather than craft (and debug!) a custom solution that is both efficient and small.
Absolutely. Your example, the App Store, and lots of other aspects of the iOS ecosystem are how Apple prevents most (not all) of the "tragedy of the commons" problems on that platform.
What does the App Store have to do with any of this? I can't imagine any technical restrictions that the App Store enforces that could not be enforced by the OS alone.
For instance, "no background services" and "you can't use up all the physical RAM" are both OS-level restrictions, not app-store-level restrictions.
How about many of those OS-level restrictions can be bypassed by hacking into private APIs, which is one of the things that'll get you banned from the App Store?
The OS does what it can, but the OS is entirely out in the open to be hacked, bypassed, subverted, worked around and so on. A pair of human eyes and a private app scanning process does what the OS can't.
No, it's not perfect. It's not even great lately. But it'd be worse without it, as you can witness on desktop operating systems.
What works is not putting up walls between developers and power users. Linux has no such restrictions and bloated software isn't something I have to deal with on it (I have the option to if I wanted to, but I don't.)
The big problem we have is that people don't care just about software features, but about software bugs. The biggest example is the web. We could have something that implements the features of a web browser relatively clearly, but what about the bugs that are treated as features? There are billions of pages that depend on bugs of current browsers to display correctly. The same for other areas such as word processors, operating systems, etc.
I don't know that I agree with this assessment. I think the bigger issue isn't "bugs" per se but rather that we keep accruing capabilities.
Take the web as an example. HTML3 was so simple that we had hobbyist implementing it. It could be ran by 100MHz computers with 16mb of ram. Now, the HTML5 spec is impossibly large. So large that we are starting to see big major companies consolidate parts of the browser because it's too complex and too big for 1 multibillion dollar company to keep up to date.
It's so large that a team of engineers which worked primarily on browsers never really finished a from scratch rendering engine (Servo) even after several YEARS of effort.
The same thing happened to the PDF spec.
This happens with a bunch of desktop/consumer products that get old.
Excel can still open xls docs from the 90s, even though there's been about 30 different changes to the spec since it was first created. Consider what that does to a product.
Even microsoft windows runs into exactly this problem. Microsoft can still run games from the 90s. Considering they completely changed out the kernel, imagine how crazy that is. By and large, they accomplish this feat by keeping around dlls from the 90s and reusing them. This is a primary reason why windows installs are now pushing 10+gb. They can't throw out support or there would be a corporate mutiny.
And before we give MS too much shade for doing this, consider what we've done in the linux world. In order to get applications running consistently across OS versions we've decided the best way to do that is shipping almost all of an operating system along with our runnable (Hello docker images)!
It's not a question of bugs. It's a question of behavior. And as time goes on, new behavior and capabilities are added to software. The longer it lives as a product, the more true that is. This is where bloat comes from.
And consider this, as a software developer, arguing to remove capabilities is almost always a losing proposition. Imagine, for example, if photoshop decided "you know what, nobody really uses JNG anymore, so we should remove it from our products." Any PO worth there salt would say "Hold it, why should we remove this? Is it costing us anything to have it? Just a few MB? Then we keep it!"
> By and large, they accomplish this feat by keeping around dlls from the 90s and reusing them. This is a primary reason why windows installs are now pushing 10+gb.
Not really. I mean, sure, from my Win10 installation (which is around 11GB), 6.8GB are DLL files HOWEVER pretty much all of them are inside the WinSxS which aren't the "DLLs from 90s", but duplicates of various DLLs that applications bring along.
Outside of WinSxS the only other place with lots of DLLs is System32, the bulk of which is taken by AMD's drivers and the rest of the DLLs take around 800MB. Then there is SysWOW64 where a similar layout can be found with most of the space being taken by AMD's drivers and around 500MB or so for the rest of the DLLs. Inside other Windows directories there are also around 900MB of various .NET installation DLL.
So in total, the Windows DLLs, including .NET are probably around 2GB. However keep in mind that i have a ton of stuff installed, with pretty much every VC and .NET runtime that existed and my installation is a bit older than 3 years old, on a fresh Win10 installation will probably have less stuff in there.
> I don't know that I agree... per-say... hobbiest... accomplish this feet
I hole-hardedly agree, but allow me to play doubles advocate here for a moment. For all intensive purposes I think you are wrong. In an age where false morals are a diamond dozen, true virtues are a blessing in the skies...
The 'inflated' software Wirth complains about is the same as is often fondly remembered in places such as here by people talking about the Good Old Days.
Meh - it will always be complicated and get more complicated going forward.
Also has anyone looked at what it takes to build a home, let alone a hospital. I don't think software is any more complicated than our society allows other processes to become complicated.
I don't see how this vulgar reduction of the problem in question to just size of binaries has anything to do with what Wirth is talking about in that article, which are among other things the factors of maintainability, cognitive load, reusability and such.
This is really a call for modularity. I'm a believer in modularity, but it is not a cure-all.
Modularity brings a lot of baggage with it, like duplication of code and lifecycle management, along with increased resource usage (ironic, as the article is mainly about reducing resource usage).
But I am still a believer in it. In my experience, it makes for a massive increase in system stability, quality, agility, and speed of development.