I'm really impressed by how backwards compatible it is. I just changed the Clojure version from "1.6.0" to "1.7.0" for one of my side projects, without updating any library versions, ran it, all the tests passed, and it seems to work perfectly. It also didn't break my Emacs setup, which is a breath of fresh air compared to how much work it was to get the Haskell tooling working with the new 7.10 GHC release (GHC-mod for instance still doesn't have a compatible release, although the trunk mostly works with 7.10). Similarly, even though it's months after the 7.10 release there are still libraries that don't support it, like reactive-banana-wx, whilst a couple of the Clojure libraries I'm using haven't been updated in over a year yet still work fine on 1.7, and none of the libraries I'm using break on 1.7.
To be fair, GHC and the Haskell ecosystem is far more complex than Clojure and its ecosystem/standard library. Nevertheless it's pleasant how easy Clojure was to upgrade (although of course this stability is nothing special: more conservative languages like Go and Java generally break almost nothing on upgrade).
Yes this is something clojure users take for granted. I dont know the state of Scala right now but a year back even minor version bump was horrible in scala in terms of backward compatibility. Really impressive job by clojure code devs in terms of maintaining such stable releases.
Scala's minor versions don't break compatibility. A year ago Scala 2.11 (major) was being released, which means you're talking about either 2.10.x or 2.11.x, neither of which broke binary compatibility between minor versions. And major versions usually have source level compatibility, our upgrade from 2.10 to 2.11 being pretty smooth.
The fundamental difference is that Clojure gets distributed as source and not as compiled .class files. Being an interpreted language that gets compiled on the fly does have some advantages. But it also has drawbacks. Startup performance suffers, the Java interoperability story is worse, many tools such as Android's toolchain expect bytecode, etc ...
The problem that Scala has (and also Clojure, as soon as you do AOT) is that Scala's idioms do not translate well into Java bytecode, as Java's bytecode is designed for, well, Java. Therefore even small and source-compatible changes, like adding another parameter with a default value to a method or like adding a method to a trait, can trigger binary incompatibilities.
The plan for Scala is to embed in the .class, besides the bytecode, the abstract syntax tree built by the compiler and then the compiler can take a JAR and repurpose it for whatever platform you want. This is part of the TASTY and the Scala-meta projects. If you think about it it's not that far from what Clojure is doing, except that this is done in the context of a static language that doesn't rely on the presence of an interpreter at runtime. Of course, LISPs are pretty cool. And of course, you'll still need to recompile your projects, but at least then the dependencies won't have to be changed.
Clojure itself actually is distributed as compiled .class files. We take some effort to ensure that there are not changes that break binary compatibility from AOT-compiled Clojure code (which is not uncommon) and break its calls into the Clojure compiler or runtime.
Not so much. You can do separate compilation with Clojure to a far greater extent than Scala (perhaps even completely). The Java interfaces of Clojure functions haven't changed in incompatible ways in a very, very long time.
The Java interfaces that Clojure relies on don't matter that much. On the other hand what is the bytecode representation of a protocol or of a multi-method?
Scala has an equivalent representation for everything it has. For example traits are just Java interfaces but with corresponding static methods. Another example is default parameters (a concept Java doesn't have) is done with method overloading. The Scala compiler has to infer Scala-specific stuff (like signatures using Scala features) from compiled bytecode. Functions that aren't methods (e.g. anonymous functions) get compiled either to static methods, or to classes that have to be instantiated (in case it's about closures closing over their context). Scala does not have the concept of static methods, but it has singleton objects, which can also implement interfaces. Of course, these get compiled to static methods, but there's also a singleton instance instantiated for those cases in which you want to use that singleton object as a real polymorphic instance. Etc, etc... So there's a protocol in place for how to encode this in the bytecode, there's a protocol for everything. And so even small changes in Scala's standard libraries can trigger big bytecode changes that end up being backwards incompatible.
Clojure doesn't have to do this, because Clojure dependencies get distributed as source-code, as I've said.
> Clojure doesn't have to do this, because Clojure dependencies get distributed as source-code
Clojure doesn't have to do this because it was designed to allow separate compilation as much as possible[1], and because it hardly ever changes binary representation in backwards-incompatible ways. Protocols and multimethods are indeed handled at the call-site, but in such a way that a change to the protocol/multimethods don't require re-compilation of the callsite[2]. Similarly, Kotlin, a statically typed language distributed in binary, is also designed to allow separate compilation as much as possible[3]. If a feature would break that property (e.g. require re-compilation of a callsite when an implementation changes at the call target), that feature simply isn't added to the language.
This is a design that admits that extra-linguistic features (like separate compilation) are as important as linguistic abstractions (sometimes more important).
BTW, separate compilation isn't only a concern with Java class files. Object file linking also places limits on how languages can implement abstractions yet still support separate compilation. Some languages place less emphasis on this than others.
[1]: In fact, as a Lisp, Clojure's unit of (separate) compilation isn't even the file but the top-level expression. No top-level expression should require re-compilation if anything else changes.
In my opinion choosing a different language than the host (in this instance Java) has to bring enough advantages to balance out the disadvantages, like less (idiomatic) libraries, less documentation, less tools, less developers available on the market. Not to pick on Kotlin, I'm sure there are people that love it, however I personally don't see what advantages a language like Kotlin brings over Java 8, being CoffeeScript versus Javascript all over again. With Clojure or Scala we are talking about languages going into territories that Java will never venture into, for the simple reason that Java is designed to be mainstream, which really means appealing to the common denominator. Of course, as we've seen with CoffeeScript, the market can be irrational, but it started to go away already, after a new version of Javascript plus the realization that everything brought by those small syntactic differences was bullshit.
Going back to Clojure and the need or lack thereof to recompile call-sites, given that Clojure is a dynamic language that doesn't have to concern itself with much static information and that does get distributed as source-code, I feel that this is an apples versus oranges discussion. But anyway, lets get back to protocols. So protocols do generate corresponding Java interfaces, just like Scala's traits. Except that Clojure being a dynamic language, there isn't much to do when your functions look like this to the JVM: https://github.com/clojure/clojure/blob/master/src/jvm/cloju...
There's also the issue that Clojure's standard library has been more stable. Well, that can be a virtue and in the eye of the beholder can be seen as a good design, however it has many things that need to be cleaned out. As a Clojure newbie I couldn't understand for example why I can't implement things that work with map or filter or mapcat, or things that can be counted, or compared, only to find out that its protocols and multi-methods aren't used in its collections and that there isn't a Clojure specific thing I can implement to make my own things that behave like the builtins. It's also disheartening to see that the sorted-set wants things that implement Java's Comparable. Clojure's collections have sometimes surprising behavior - like for example I might choose a Vector because it has certain properties, but if you're not careful with the operations applied you can end up with a sequence or a list and then you have to back-trace your steps (e.g. conj is cool, but the concept should have been applied to other operators as well IMHO). Transducers are freaking cool, however I feel that the protocol of communication isn't generic enough, as I can't see how it can be applied to reactive streams (e.g. Rx) when back-pressure is involved (might be wrong, haven't reasoned about it enough). As any other language or standard library, Clojure is not immune to mistakes and personally I prefer languages that fix their mistakes (which I'm sure Clojure will do).
EDIT: on "separate compilation", not sure why we're having this conversation, but generally speaking Scala provides backwards compatibility for everything that matches Java. Adding a new method to a class? Changing the implementation of a function? No problem. On the other hand certain features, like traits providing method implementations or default parameters are landmines. Along with Java 8 things might improve, as the class format is now more capable. As I said, there's a roadmap to deal with this and with targeting different platforms (e.g. Java 8 versus Javascript versus LLVM) through TASTY, but I don't know when they'll deliver.
Re collections: There are Java interfaces you can implement to make your own collections that work as built-ins. In general, nothing in the Clojure compiler or runtime is written based on concrete collection classes, only on the internal interfaces. It's perfectly feasible to write a deftype that implements Counted and Associative and whatever else and make your own things that work with all the existing standard library. There are plenty of examples of this.
Re sorted-set: Any 2-arg Clojure function that returns -1/0/1 comparable semantics will work.
Re staying "in vectors": The subtle blurring of the sequential collections between the collections and sequences is part of what makes so much of working with data in Clojure easy. However, there are certainly times when you want more control over this; fortunately in 1.7 with transducers you have the ability to control your output context (with functions like into or conj) and this is easier now than ever.
Re rx: several people have already implemented the use of transducers with Rx. I know there's a RxJs and I'm pretty sure I saw something re JavaRx although I can't put my finger on it right now.
Hey, please consider my rants to be of good faith, plus I'm a rookie :-)
On Rx, the step method is synchronous. It works for the original Rx protocol and it works for RxJS because RxJS has implemented backpressure with the suspend/resume model (e.g. in the observer you get a handle with which you can suspend or resume the data source), but otherwise the model is pretty much like the original, in that the source calls `onNext(elem)` continuously until it has nothing left, so it is pretty much compatible with transducers. RxJava started as a clone of Rx.NET, but is now migrating to the Reactive Streams protocol, which means that the observer gets a `request(n)` function and a `cancel()`.
And this changes everything because `request(n)` and `cancel()` have asynchronous behavior, very much unlike the step function in transducers. And furthermore, implementing operators that are asynchronous in nature requires juggling with `request(n)` and with buffering. For example the classic flatMap (or concatMap as called in Rx) requires to `request(1)`, then subscribe to the resulting stream and only after we're done we need to request the next element in the original stream.
Then there's the issue that the protocol can be dirty for practical reasons. For example `onComplete` can be sent without the observer doing a `request(n)`, so no back-pressure can happen for that final notification and if the logic in the last onNext is happening asynchronously, well this can mean concurrency issues even for simple operators. Although truth be told for such cases there isn't much that the transducers stuff can do.
I might be wrong, I'd love to see how people can implement this stuff on top of transducers.
> for the simple reason that Java is designed to be mainstream,
Not really. Java was designed to be practical, and it ended up being mainstream.
Most languages want to become mainstream (including Scala), most of them just fail to achieve that goal for a variety of reasons, often due to their design decisions (but not always).
> Most languages want to become mainstream (including Scala), most of them just fail to achieve that goal for a variety of reasons, often due to their design decisions
A language will often become used in one particular domain, often not the domain it was designed for. Take another JVM language Groovy -- it's used a lot for scripting throwaways used in testing, including build files for Gradle, and was extended with a meta-object protocol for Grails which has risen and fallen in sync with Rails. But then its backers retrofitted it with a static typing system and have tried to promote it for building systems, both on the JVM and Android but virtually no-one's biting.
I disagree, Java was designed to be marketable to the layman and for a very good reason ...
Software companies can scale either horizontally (e.g. hiring more people) or vertically (hiring the best). Because of the classical bell curve distribution, really good people are hard to find, therefore horizontal scalability is very much preferred (and obviously not everybody can have "the best"). Unfortunately you can't really grow a company both horizontally and vertically. The reason for it is because getting good results from mediocre people requires lots of processes, coordination/synchronization and politics of the sort that doesn't appeal to really good people and that's because really good people have a choice, because of the supply-demand equation which favors them.
In the wild you'll also notice another reason for why horizontal scalability is preferred. Corporate departments and consultancy companies are usually scaled horizontally because they get paid per developer/hour and unfortunately there are only 24 hours in a day. Hence the bigger the team, the bigger the budget. On the other hand small teams happen in startups where you often see 2-3 people doing everything. Small teams also happen in big software companies, when speaking of either privileged seniors or folks that handle real research.
Big teams are getting things done, as Alan Kay was saying, on the same model as the Egyptian pyramids, with thousands of people pushing large bricks around. Small teams on the other hand tend to handle complexity by means of better abstractions. Unfortunately better abstractions are harder to understand, as they are first of all, unfamiliar.
So back to Java ... it's actually a pattern that you tend to notice. Such languages are afraid of doing anything that wasn't proven already in the market. The marketing for such languages also relies on fear and doubt of the alternatives, which Sun took advantage of, appealing to the insecurity of the many. This is one language that refused to implement generics because they were too hard and then when they happened, the implementation chose use-site variance by means of wildcards, with optional usage, possibly the poorest choice they could have made. Generic classes are invariant of course, except for arrays. This is the language that doesn't do operator overloading, as if "BigInteger.add" is more intuitive than a "+", but on the other hand it does have a + for Strings. This is the language in which == always does referential equality, so you've got tools converting every == to .equals because that's what you need most of the time. The language which until recently didn't have anonymous functions because anonymous classes should be enough. This is the language in which everything needs to be in a class, even though the concept of "static method" makes no sense whatsoever. This is the language in which at least 30% of all libraries are doing bytecode manipulation to get around the language's deficiencies. Such languages also avoid metalinguistic abstractions like a plague, such as macros, because OMG, OMG, people aren't smart enough for that (though truth be told, you're expanding your language every time you write a function, but don't tell that to the weak). Also, AOP is a thing that has a name in Java, which is pretty funny, given that in other languages it tends to be effortless function composition.
As an environment it turned out to be great, eventually. If you want another shinning example, take a look at Go. Same story.
In other words Clojure or Scala will never reach the popularity of something like Java. Sorry, people can wish for it, but it will never happen. And that's OK.
I disagree. Java's design wasn't timid -- it was radical in its intentional reduction of expressivity compared to the most popular language around, because the much more expressive language was proven to be far too costly to maintain for the duration software has to be maintained.
But you are certainly right that Java was also meant to be familiar and unthreatening. It allowed itself to do that because it realized -- and was proven right -- that extra-linguistic features such as a GC and dynamic linking are essentially the features that contribute to productivity more than abstractions.
> Software companies can scale either horizontally (e.g. hiring more people) or vertically (hiring the best).
While this may be true, it has little to do with Java directly. For years I worked with the best algorithmeticians, physicists and mathematicians (physics simulation, real-time control). We were very much involved with the hard problems we were trying to solve with novel algorithms, and frankly couldn't be bothered with spending precious resources on elegant abstractions. Our problems was one of essential complexity -- not accidental complexity -- so we needed a simple language that can be easily shared and understood by everyone, one that was fast, and one that got the job done.
Believing that better developers opt for more powerful languages is what I call the "Blub developer paradox": the blub developer has only worked on simple, blub, problems (CRUD applications, web etc.), and hence believes his cool abstractions are the mark of the "smart developer". If he'd only look at developers working on seriously hard algorithms, he'd realize that abstractions are secondary, and it is developers above him on the essential complexity spectrum that opt for simpler languages with weaker abstractions.
That better developers choose the more powerfully-abstracting languages is completely and utterly false. The best developers I've ever known -- those who really came up with radical, game-changing solutions that truly advanced the state-of-the-art -- were all C and Java people, and I don't think it is a coincidence. Their mind is focused on the algorithm rather than on alluring, but ultimately distracting abstractions.
However, saying that only people who work on boring problems have the free mental resources to spend on nifty abstractions is as valid a generalization as the one you've made. There are many reasons to choose a particular programming language.
Personally, though, I believe that while the choice of the programming language matters, it matter so much less than design, choice of algorithms and availability of other tools. I no longer equate software with the code it's written in. A running program has many factors affecting its quality and cost, and the code used to write it is just one of them. Still, if I need to write a web app, I'd choose Clojure over Java every time. If I need to write a military command-and-control app, or a power-station control app? I'd probably go with Java.
You're attributing to me and my message things I haven't said, which is a sign of an unproductive conversation.
> For years I worked with the best algorithmeticians, physicists and mathematicians (physics simulation, real-time control)
If we are talking about experience, I've built an RTB system with soft real-time requirements handling tens of thousands of transactions per second and in which Scala was awesome because we used better abstractions for handling multi-threading, concurrency and processing of huge amounts of data in real-time.
Right now I'm working on a project for E.On that monitors and controls power plants. We are talking about real-time processing of signals, modeling of state machines that evolve according to those signals and machine learning for finding the optimal dispatch algorithm and for better fault detection. Scala has been awesome because functional programming along with its very static type system allowed us to better handle the event-based nature of the system and the business logic which is like a freaking fractal of complexity.
I'm now moving on to another project from the health-care industry that handles, you probably guessed it, monitoring of patients by analyzing signals coming from medical devices. Also in Scala, though there's lots of flexibility required in configuring the system and I might propose Clojure for some parts because read-eval.
Thinking that you're the only one that interacted with "the best algorithmeticians, physicists and mathematicians" or that your problems are more interesting than what other people are doing is snobbery. I never said that the best people choose different languages, all I said is that the best people choose better abstractions and that some languages are meant for building better abstractions. Speaking of which, for "blub CRUD applications" I would never choose something like Clojure or Scala, simply because for those types of apps PHP or Ruby have got that covered.
I don't think I'm the only one etc.. It's just so happens that all the best people I've known to work on algorithms preferred languages with minimal abstractions. I very well acknowledge that others, who are no less talented, might prefer more abstractions.
However, I am saying that the assumption that better developers invariably prefer more abstractions and more powerful languages is absolutely wrong.
Personally, I've had a terrible experience with Scala, found its abstractions distracting, its complexity hindering readability, its DSLism obstructing code sharing and staff migration among various teams, and the grapple with the tool stack a horrible waste of time -- all with little benefit and lots of harm. I know that my experience isn't universal, but it's far from unique, either.
Clojure is a completely different story, partly because its designers are not purists and not interested (at least not as a primary concern) in PL research, but even with Clojure I don't think we have enough data to conclude that its cleverness (which, thankfully, is much lower than Scala's[1]) is a net gain for large projects that require maintenance for a decade or more by large teams.
All in all, I've seen that a solid, fast runtime with a good compilation and linking story, excellent profilers and deep monitoring (all provided by the JVM) yield much bigger gains than most new languages. Given that those extra-linguistic features are free and switching a language is extremely costly, I'm always hesitant about whether it's worth it. Staying on the JVM reduces the switching costs, and good interoperation with Java -- like Clojure has -- reduces it further, so I find the risk to be worth it for some less-risky projects.
[1]: Then again, so is every other language in history.
In a static language like Scala or Haskell you usually work with a type-class, which is pretty cool because you can provide implementations for types you don't control (not implying that Scala's Ordering is perfect). Instead of Java's Comparable interface I was expecting a protocol, which are almost equivalent to type-classes, or a multi-method.
Of course, in many implementations you usually also get a version of a constructor that uses a provided comparison function. However certain things, like plain numbers or strings, have a natural ordering to them, hence the need to have the sorted-set in Clojure also work with Java's Comparable. But again, I was expecting a Clojure specific protocol.
The reason for why this happens, I believe, is because protocols weren't there from the beginning, being a feature introduced in Clojure 1.2. From what I understood, in ClojureScript protocols are "at the bottom" as they say, so there's hope :-)
Thanks! So if I read your comment correctly there is nothing inherently wrong with the Comparable interface it's just that Clojure's sorted sets and maps could have used protocols instead. I can see why that would be useful for extending existing types (as you mentioned).
OTOH it’s not often I’ve seen third-party types that I wanted to be comparable but were not. In general I think that if a type does not implement a core interface you have to consider the possibility that the designer chose not to implement it for a reason.
Protocols were added in Clojure 1.2, well after the Clojure collections or stdlib were created. In a perfect world, Clojure itself could leverage protocols more widely across the stdlib. For practical reasons, this is difficult now.
Maybe a mitigating factor is that any Clojure 2-arg function extends AFunction which implements Comparator. So any Clojure function that returns -1/0/1 will work transparently. Example, use - as a comparator:
Yes, that helps in this case, but that doesn't mean being statically compiled allows you to disregard extra-linguistic downsides.
> however it has many things that need to be cleaned out.
No! There are things that could have been better; sure. But they don't need to be cleaned out because that would break backwards compatibility. Backwards compatibility is a very, very, very important thing. So much more important than a "perfect implementation" (something that can never be achieved anyway). This is something that is very hard for PL purists to understand, but extra-linguistic features and guarantees trump 100% consistency almost every time.
> It's also disheartening to see that the sorted-set wants things that implement Java's Comparable.
That's only disheartening if you're a purist. If you care about extra-linguistic concerns, such as Java interoperation, this is a mark of great design. You see how the language designers took into account concerns outside the language itself. Every Clojure map is a Java map and vice versa. Every Clojure sequence is a Java collection and vice versa. Every Clojure sorted set is a Java sorted set and vice versa. This is great design around constraints and what Rich Hickey always talks about. The language itself is no more important than how it fits within the larger ecosystem and its constraints.
> as I can't see how it can be applied to reactive streams (e.g. Rx) when back-pressure is involved
Clojure is an imperative language first and functional second. It is even more imperative than Scala (in spite of Scala's mutability) -- at least with Scala's (recent-ish) emphasis on pure-functional concepts (a terrible idea, BTW, but that's a separate discussion). Back-pressure is therefore automatic (and implicit) with pull-based channels (that transducers can be applied to).
> I prefer languages that fix their mistakes
Because you're a purist. I prefer languages that see the big picture consider what's important in the grand scheme of things, and realize that code is important but secondary to working programs. As Rich Hickey is not a PL researcher, I am sure that those mistakes will only be fixed if they don't break compatibility or Java interoperability, which are more important for the industry than a perfectly clean language.
> not sure why we're having this conversation
Because supporting separate compilation -- if not perfectly then at least treating it as a top-priority concern -- is one of the key ways to ensure binary compatibility between runtime-library versions.
> On the other hand certain features, like traits providing method implementations or default parameters are landmines.
That's exactly my point. Separate compilation (as many other crucial extra-linguistic concerns) is just not a priority for Scala. Kotlin, OTOH, implements default arguments on the receiver end rather than at the call-site, so changing default values in the target doesn't necessitate recompiling the caller.
Kotlin announced that until the upcoming 1.0 release there will be breaking changes, but not after. Binary compatibility will not be 100% guaranteed, but it's a high priority.
If you read what it says, they've changed an implementation of a function which may change ordering of unordered collections, and that may break some people's code if they relied on some specific ordering. They are not talking about breaking interfaces. In fact, they specifically say they are not aware of any binary incompatibilities. Their "strive to maintain" works out very well in practice, so far.
I wrote that sentence and you should take it as written. In general, we strive to maintain binary compatibility (old AOT-compiled Clojure code should continue to run on newer version), we do not exhaustively test for or guarantee that. At times we have broken this in alphas etc and we take it as a high priority to fix such things.
I'm not a Scala user, but Clojure the language tends to be extremely stable. Existing stdlib functions almost never get updated, unless they add a new (backwards-compatible) arities.
Releases tend to consist mostly of new features, and a small number of bugfixes.
Part of it is that the standard library API has been stable for years, where new things are added but no breaking changes are introduced.
The second reason is that Clojure libraries are typically shipped as source. This avoids the whole binary compatibility problem you see in Scala. Typically, you only compile the end applications to byte code.
I don't think so. It's because of a strong commitment to users not to introduce breaking changes. I've been mostly using Java and Clojure in recent years, and before that C/C++ and I'm pretty surprised people don't take backwards compatibility for granted, and that there are languages that don't consider it a top priority. (Well, new C++ compiler versions sometimes broke existing code, but that was quite rare)
Probably, more because its a lisp, whereas Scala was more of a fundamentally new language that isn't a member of such a well-studied family, but instead was led by inspiration from lots of different places, and has a lot more work in shaking out how to reconcile those influences into a coherent whole.
Though Scala being not merely statically typed by aimed at both interoperating with Java's type system and supporting a more robust and powerful type system than Java does did pose particular challenges.
No, it's because Scala's dependencies / libraries get distributed as compiled .class files, whereas Clojure's get distributed as source-code. Tried explaining this above: https://news.ycombinator.com/item?id=9807323
If you want to build a website in Clojure, I highly recommend checking out http://luminusweb.net - the documentation is amazing, and it incorporates nearly all of the best practices I have seen.
Making an API in Clojure using Swagger gives you a full, interactive UI and documentation for your API, while also having a schema which makes sure you know what is submitted and that it validates (i.e. is that a string or a number?)
>Transducers are composable algorithmic transformations. They are independent from the context of their input and output sources and specify only the essence of the transformation in terms of an individual element. Because transducers are decoupled from input or output sources,
The biggest thing holding me back from learning Clojure is that I fear it will take me a decade to become remotely competent in it.
If you have an understanding of functions like map, filter, and reduce, transducers are actually pretty easy.
Say you have `(map inc [1 2])`. You can run that, and get `'(2 3)`.
A transducer is the `(map inc)` part of that call (slightly confusingly, this isn't partial application or currying). You can apply it to something like `[1 2]`, but you can also compose with it, by combining it with say, `(filter even?)` to get something that represents the process of incrementing everything, then removing odd numbers. Or you can put in things that aren't collections, like asynchronous channels, and get back a new channel with the values modified accordingly.
That's pretty much it.
What I think I love most about Clojure is that there are fantastic, esoteric, academic ideas that, when I read about them in a context like this for the first time, I a) do not understand them, and b) have no idea how they would be useful. Then I read an example or two, and suddenly it's apparent that the tool is really as simple as it can be--there's very little accidental complexity--and is extremely useful.
The way you explain it, it's no different from functions and function composition; in which case, why invent new vocabulary?
I do remember looking into them before and translating them into Haskell and they ended up not being identical to functions in the trivial sense that you suggest, but I forget how.
Transducers are functions. The thing is that they are functions that are designed to serve as the functional argument to reduce. And they pair with ordinary functions which are not transducers.
For instance if we have (map inc [1 2]), there exists a transducer function T such that:
(reduce T [1 2]) == (map inc [1 2])
I.e. we can somehow do "map inc" using reduce.
Okay?
Now, the clever thing is this: why don't we allow map to be called without the list argument? Just let it be called like this:
(map inc)
This looks like partial application, right? Now what would partial application do? It would curry the "inc", returning a function of one argument that takes a list, i.e.:
;; if it were partial application, then:
((map inc) [1 2]) ;; same as (map inc [1 2])
But Hickey did something clever; he overloaded functions like map so that (map inc) returns T!
(reduce (map inc) [1 2]) ;; same as (map inc [1 2])
The cool thing is that this (map inc) composes with other functions of its kind. So you can compose together the transducers of list processing operations, which are then put into effect inside a single reduce, and the behavior is like the composition of the original list processors.
It's like a linear operator; like LaPlace. Composition of entire list operations in the regular domain corresponds to composition of operations on individual elements in the "t-domain".
> (reduce (map inc) [1 2]) ;; same as (map inc [1 2])
This is wrong. You've missed the point.
(map inc [1 2])
is actually roughly equivalent to
(reduce ((map inc) conj) [] [1 2])
which, due to the use of `reduce`, is eager. To get laziness back:
(sequence (map inc) [1 2])
Transducers are not reducing functions, they return reducing functions when applied to reducing functions. `((map inc) conj)` is a version of `conj` that calls `inc` on all the rhs args before `conj`ing them into the lhs arg.
I suspected I had to be not quite understanding something, because why would we want, say, a map based on reduce that chokes on lazy lists, in a language where laziness is important.
Transducers are mostly function and function composition, but with a specific signature. There are a handful of contracts on a transducer, so it is useful to have a name, so you can say "this function takes a transducer" and "this function returns a transducer".
Older collection functions were semi-lazy by default. They auto-realized in chunks of 32. Transducers wrap the functions then pass the data through, allowing full-laziness (or not if you want) by default.
They also work well in cases where you don't know if/when a new value is coming, like in channels or observables. This is because they aren't necessarily wed to the seq abstraction from the get-go.
I think people are seriously underestimating the protocol involved and understanding that protocol is required if you want to build your own transducers compatible streaming, which is always useful. There's also the issue that the protocol in question may not be generic enough. From the presentations I've seen it claims that it might work with Rx streams as well, however I don't think it can deal with back-pressure. This is just a personal opinion and I'd like to be proven wrong.
That said the concept is pretty cool and we shouldn't fear learning new things. After all, the whole point of learning a new programming language is to be exposed to new ways of solving problems, otherwise why bother?
Adding to weavejester comment, back-pressure is to be handled with core.async channels [1]. I guess in Rich's terms, RX/FRP "complects" the communication of messages with flow of control [2]. Although this last statement may not be true anymore given that RX now has many functions to control the scheduler (eg. observeOn/buffer/delay, etc).
Because Clojure isn't a pure functional language, transducers may be stateful. `take` uses a volatile (i.e. a fast, mutable variable) to retain state. I don't believe a `flatMap` transducer exists in Clojure yet.
Don't worry too much if you don't get the esoteric stuff in the beginning. The fact that it's there just means that there's room to grow. The core language is actually very simple and quite straightforward. It just allows for some quite mind-boggling stuff.
Try Carin Meier's Living Clojure if you feel up for an intro.
Clojure Applied is an excellent book so far. I've been using Clojure for a long time, and it's still been very helpful in updating idioms for modeling domains, composing applications, etc.
I'd say it's required reading for working Clojure programmers, of any experience level. We really needed a "how to build applications in Clojure" to complement all the "how to do stuff in Clojure" books that are already out.
Transducers create functions that are meant to be passed to reduce (i.e. take accumulator and element and return a new accumulator).
Its possible to implement most other functions (map, filter, take etc.) using reduce. Transducers take advantage of that.
Unfortunately, if you try to do that, you'll notice that the implementation is sometimes tied with the way you build the entity: e.g. for vectors, map implemented in terms of reduce would start with an empty vector then append to it.
That sucks. We want operation chains that are independent of the data structure they operate on. We want them to work on vectors, lists, channels, whatever - anything that can have a reduce-like operation (anything reducible)
However, it turns out you don't necessarily have to recreate the entity (e.g. list) when chaining. All you need to know how to do is invoke the next operation in the chain, i.e. the next reduction function. For example:
* map can apply the function on the element to get a new result, then call the next reduction function with the accumulator and that new result.
* filter can check the element and either return the old accumulator or apply the next reduction function with the element, etc.
Therefore, transducers simply take an additional argument: the next reduction function that should be applied. This lets you build up a chain of operations (e.g. map filter etc) that doesn't care about the kind of entity they're operating on. Only the last "step" must be a reduction function that needs to know how to build the result.
Actually the learning curve is very gentle, and you can gradually expand from basic features into the more advanced stuff when ready. Many of the Clojure features that have sophisticated underlying design principles (like transducers) can be used cookbook style from easy-to-understandable examples. Jump in, the water's warm.
Clojure is one of the easiest languages to learn. You need to know very few concepts to become productive in it. I get co-op students very 4 months at work and they generally start doing useful stuff within a week or so.
You don't need to learn every feature to be competent in the language. Some features are advanced features that are not required in day-to-day programming (eg, transients, you can go forever without ever touching them, but if you need performance they're there). A lot of the advanced things are of most interest to library authors.
Transducers are part of cultural drive of the Clojure community to identify common patterns and simplify design. Rich's presentations do a nice job of explaining how a transducer is about separating [how to do the work] from [where the work is done]. By "work" I mean mapping, filtering, reducing, and so on. By "where the work is done", I mean that a transducer doesn't care what kind of data structure it operates on.
As a general comment, to those new to Clojure, when you find something in Clojure that seems strange or different, I encourage you to ask "What does [function X] care about? (e.g. need to know)" and "What does [function X] not need to know?". This relentless drive to simplify design and responsibilities mean that functions are small and "opinionated", but in ways that are driven by constraints, not arbitrary decisions. So these choices make a lot of sense -- I'd argue they flow pretty naturally.
Don't get me wrong; I'm not knocking it. I definitely see the value in these kinds of features.
Like zippers, I'm sure it's another case of "If you understand it, it's really useful."
The point is, you don't necessarily need it to get the job done, and you certainly don't need to be intimidated about it or feel obligated to learn it right off.
Even Haskell isn't actually so difficult to grasp the basics of, if you keep this in mind and just think "Do IO in a 'do' block" and go about your way. See [1].
You don't have to know everything in any language just to get some work done, a lot of times advanced features are just that: advanced, stuff for doing things a bit more efficiently, or to handle certain rough edge cases. Learn them in their own time, and they'll make you better at what you do, but don't get wrapped up too much in expecting perfect efficiency from yourself.
A lot of programmers seem to be, as a people, kinda bad at this kind of self-reflection, like we're all sheep in wolves' clothing trying to avoid showing a hint of weakness. I know I sure am. There's nothing wrong with not knowing something, just try and take a moment learn it when you can.
No need. Getting started, and to a level where you can be productive, is very easy. LISP is a very simple language, and lots of the features of Clojure mean that it's simple in practice too. That's compared to Haskell and Scala, which do require a little theory in order to get started.
IMHO.
There will perhaps come a time, where you want to know about and transducers, but by that time the concepts won't seem intimidating.
This is only slightly complicated if you're writing your own transducer operation.
What is interesting is that these transducing operations can be written quite generically - code reuse is huge.
But, usage of transducers is similar to Java 8 Streams, or Haskell stream fusion, only that the implementation is fully generic, not dependent on the container/source.
That's not a very plain-language description of transducers. Transducers are functions which can be applied to any sort of collection or sequence. You can use then to do something like filter a collection or an input stream with the same code. They are basically just a way to make things like 'map' and 'filter' work on datatypes other than sequences, such as streams or channels.
Again, that's not true. Transducers are about reducing things. That's why a transducer first needs a reducing function. But I won't go into details what trasnducers are. There is many good (and long) blog posts about it.
No, reducers are about reducing things. Transducers are about creating reducers that can be used on arbitrary collection-like things.
Reducers come from the observation that many higher order operations on collections can be built on reduce. You just supply the right reducing function and you get the equivalent to 'map' or 'filter'. This is useful because when you start chaining these higher order functions you can gain performance be replacing the multiple function calls with a single reducer based function.
To take the example from the main docs:
(fold + (filter even? (map inc [1 1 1 2])))
Here, each function returns a reducer which is a combination of the original collection ([1 1 1 2]) and a reducing function which will be applied to the collection. Ultimately this code will result in a single call to reduce with a single function applied to [1 1 1 2]. This differs from the 'standard' way of doing this:
(reduce + (filter even? (map inc [1 1 1 2])))
...in that no intermediary representations of the collection are necessary. Neat.
A transducer takes the same idea, but does it in a way that lets you apply this to arbitrary collection-like things, not just seqs. A transducer works by cutting out the original collection; you build the reducing function by chaining transducers together and pass it later another function which will pick apart the collection-like thing.
Which is not exciting until you realize that that middle term can be passed to 'chan' al la:
(chan 1 (comp (filter even?) (map inc)))
Which means that everything going through that channel will increased by filtered with 'even?'. Now you have a suite of functions which will take a transducer and use it in a lot of different contexts allowing you use the same logic on streams, sequences, channels etc. This same logic, that you would originally have expressed with a series calls to 'map' 'filter' 'take' and applied only to a sequence.
Clojure is sophisticated [0] but it is hard boiled down to necessary complexity. This means that ordinary language features like interop [1] are well sugared and more abstract concepts like transducers are simple to implement and map well to their description [2]. Stuart Halloway's Programming Clojure captures this idea.
[0] and big for practical purposes like Common Lisp.
[1] you can get at all of Java perhaps better than you can from Java using the REPL.
[2] like lexical scope and continuations in Scheme.
I had a similar feeling when I first starting using Clojure, but I found it was more of a discomfort from discussing programming language with a direct and accurate vocabulary. After a while I've become much more comfortable reading about and watching videos on programming languages in this style and I really enjoy being able to discuss and think about design decisions in a straightforward way.
Basic competence comes pretty quick, it's a small core language, compared to most. Most things like transducers you can ignore for a long time. The longest part of learning it for me was learning to think functionally, but you can do that in pieces, and nearly all of what you learn is applicable to any functional language, or even imperative languages with functional aspects.
It is one of the simplest and easiest languages around.
The addition of transducers is an unfortunate case of Clojure "pulling a Haskell", valuing an elegant abstraction over ease of understanding and learning. Indeed, your comment alone shows that doing them (and especially giving them a high profile) was a mistake. Just because you can abstract something elegantly doesn't mean you should. No beautiful abstraction is worth scaring people away. Fortunately, Clojure doesn't make many such mistakes, and it usually tries to err on the side of pragmatism. I hope transducers aren't the beginning of a trend.
But just don't use transducers until you feel comfortable enough with the language. They're not an essential feature.
I don't understand this argument. Clojure shouldn't have transducers because the word sounds scary? Programming language designers should avoid adding powerful, higher-order abstractions because they are hard to understand? This sentiment is incredibly anti-intellectual. And like you said, if you don't understand it you don't have to use it.
Language design requires tradeoffs: the power of a feature, vs. how much more complex it makes the language to read/learn/etc. Making something easy to use doesn't commit one to anti-intellectualism; it just shifts the field of intellectual challenge elsewhere.
See Yegge's Perl essays[1] for examples of core-language features that are 'powerful' but problematic; hopefully that convinces you that at least the form of argument is legitimate. It could be that those Perl features are bad whereas transducers are great. I happen to share some skepticism of transducers because they overly-resemble existing features (partial application) and can be implemented pretty easy with existing Clojure tools. Also the marquee use case (sharing transformations between sequences and channels) so far seems exceedingly rare to me, though maybe things are moving in a direction where that's more important.
> And like you said, if you don't understand it you don't have to use it.
True to an extent, but (a) the high prominence given to the feature may well lead to a situation where you can't read other peoples' code without learning the concept, and (b) it's easy to create transducers by accident.
Most languages with map/reduce don't have transducers (and they're certainly not central features). Also, don't forget that while Clojure is functional, it is also very much imperative (it's a functional-imperative language rather than a pure-functional language).
> Most languages with map/reduce don't have transducers
That's true but mostly irrelevant to the point that transducer usage isn't complex once one understands map/reduce, which are common, whether or not transducers are.
> Programming language designers should avoid adding powerful, higher-order abstractions because they are hard to understand?
In general? Absolutely![1] Making algorithms easier for humans to understand is the whole purpose of abstractions.
Programming is based two things: algorithms and abstractions, with algorithms being fundamental to computation and abstractions are usability features designed to help people write code -- that is their one and only purpose (computers and even theoretical computational models don't care about abstractions). Unlike algorithms, abstractions are not useful in isolation, and their utility is not a function of mathematical "power" but of psychological benefit. Their utility is measured by how much they help the human programmer write and read an algorithm[2]. Another way to look at it is that algorithms tackle essential complexity and abstractions tackle accidental complexity[3].
More specifically, abstractions help human programmers in two ways: they increase code reuse and improve code readability.
But here's the thing. There are things other than powerful abstractions that help humans program -- for example, a clear execution model (what happens when), debuggability etc.. This means that the more powerful (i.e. abstract) abstractions become they do not necessarily perform their function -- namely, assisting developers -- better. So every abstraction must be carefully weighed: how much wasted code does it save, how much more readable it makes code, vs. how much does it hurt understanding or debuggability.
I'd say transducers are just about the point where the abstraction starts hurting more than it helps. It's a borderline case. Now, I am not saying Clojure shouldn't have transducers (again, borderline), but that they most certainly shouldn't be emphasized.
> if you don't understand it you don't have to use it.
That doesn't quite work. I said, don't use them right away. Once a programming language and its libraries use an abstraction, you must learn it sooner or later. After all, most code you'll read isn't your own. This is why every language feature has a cost, and why good language developers are hesitant about introducing new features (transducers aren't a language feature, but they do have a prominent place in the standard library).
---
[1]: For example, Java and Go are both languages whose designers intentionally and radically reduced the use of many of the abstractions available in other popular languages of their time. Java in particular drastically removed abstractions possible by the most popular language at the time it was introduced, and attained tremendous success, partly because of that (well, at the time it was already apparent that C++'s power -- in addition to its lack of safety -- was greatly detrimental to code maintenance in most parts of the industry).
[2]: Abstractions are secondary to algorithms. Also note how almost all sub-disciplines of computer science deal with algorithms, and just one, rather small, discipline -- PL research -- is concerned with abstractions.
[3]: Even algorithms are often not judged in isolation; there are lots and lots of useful algorithms that aren't used because they are too hard to implement and maintain correctly -- regardless of abstractions used.
For example, Java and Go are both languages whose designers intentionally and radically reduced the use of many of the abstractions available in other popular languages of their time.
Java is a perfect example of proliferation of accidental complexity caused by the unwillingness to provide facilities for composing abstractions in the core language. The Java community has resorted to massive external XML-based configuration files to provide code reuse where it is impossible to achieve inside the language.
The whole point of Lisp (and Clojure is a Lisp) is the power to compose functions and lists together, building higher and higher-level abstractions to allow concise expression of logic to solve a problem. The point of Lisp is to be expressive and powerful, not popular and readable. BASIC and COBOL were popular and readable.
> Java is a perfect example of proliferation of accidental complexity caused by the unwillingness to provide facilities for composing abstractions in the core language.
Well, those are all tradeoffs, and the fact is that since the addition of annotations and later lambdas, all those "external XML-based configuration files" are receding, to the point they no longer exist in almost any of the newer libraries (or new versions of old libraries).
Java didn't start out with insufficient abstractions that were later added by other languages. Java started out as a reaction to languages with overly-powerful abstractions that hindered maintenance. You may not like the result and think it aesthetically unpleasing (although it's been getting better and better for quite a while now), but it is a fact that Java codebases are extremely maintainable. This is not a guess or a gut feeling. Those legacy Java codebases exist as a living proof of that. Other languages legacy code was either thrown away or frozen, unmaintained. And if you think "good, codebases shouldn't live for too long", well, that's a nice sentiment, but the fact is that long lived codebases save the industry a lot of time and money (even if young developers think they could have done it better if they'd just started from scratch). Again, we know that because we've seen the alternative.
BASIC and COBOL were never nearly as maintainable as Java (I know becas. I love Lisp (Scheme was among my first languages, and Clojure is my favorite application-programming language), but the point of a language designed for the industry shouldn't be expressiveness or power, nor readability, but usefulness and maintainability. Any other property should serve that. Professional software is written to serve a purpose -- they're not pieces of art (or not just pieces of art). You must remember that the average lifespan of a codebase is about a decade, and the cost of the project is spread -- unevenly -- over that decade. A language for the industry -- as well as other related tools -- is meant to reduce that cost.
I think Java does an excellent job of that, and I believe Clojure can do an excellent job -- we just don't have the data yet. But if you design a language for the industry, you must always look at the big picture -- at those ten years of the codebase as a goal -- that's the challenge. Coming up with a language that's powerful and expressive is easy. Doing that in a way that really serves the industry's need is much harder. Rich Hickey is a pragmatist, and he gets that. I think that the emphasis on transducers was a stumble, because he may have lost sight of the real goal.
Dive in! It's like beginning ballet over twenty: you'd really have to trust your slow, consistent, building momentum to eventually get you there. Learning Clojure up to getting remotely competent with it did take me a year and a half, during which I used it in all my projects. Also including getting into Emacs. I breathed it all throughout. Now I'm good in it, I have ease designing big FP systems, and I finally decided to learn transducers a few weeks ago and it took me me just an hour to grasp/implement it in my app. Neither a decade :)
Its more like 18 months. Unless you already know a lisp or Haskell. The only thing I can say is that it is totally worth it, I know very few people who regret putting it in their head. The same is less true for Ruby, php, Visual Basic.
They help relieve pressure on the garbage collector by eliminating intermediate sequence allocations (eg. (comp (map a) (map b) (filter c) (map d)) would create 4 sequences, transducers create none). Consequently you can work with results incrementally unless there is an intermediate flush required such as for a windowed aggregation.
I imagine most applications aren't very sensitive to the performance gain, but it's good to know for when you need it. In addition it's a nice and testable, compos-able pattern. I've only done backend work, no cljs yet, but I'd call it reasonable practice to think in terms of compositions/transducers for most data transformation pipelines in the future though, throughout the stack, whether performance matters or not.
It would be great to see a refactor of some code using transducers to get a better sense of what they're useful for. From an abstraction standpoint, I can see the attraction; but I am having a tough time imagining how it would improve code in practice.
Much Clojure code relies on applying nested transformations to sequences:
(reduce + 0 (filter odd? (map inc (range 1000))))
Conceptually, this takes an input sequence of 1000 elements (range 100), then creates an intermediate sequence of (map inc), then creates an intermediate sequence of (filter odd?), then reduces that with +. (I am glossing over a number of internal optimizations - chunking and transients, but the idea holds.)
Transducers let you represent the transformation parts as an independent (reusable) thing:
(def xf (comp (map inc) (filter odd?)))
You then apply that composite transformation in a single pass over the input:
(transduce xf + 0 (range 1000))
transduce combines (a) input iteration (b) transformation application and (c) what to do with the results - in this case apply + as a reduce. Other functions also exist that make different choices about how to combine these parts: into collects results into a collection, sequence can incrementally produce a result, eduction can delay execution, core.async chans can apply them in a queue-like way.
There are a number of benefits here:
1. Composability - transformations are isolated from their input and output contexts and thus reusable in many contexts.
2. Performance - the sequence example above allocates two large intermediate sequences. The transduce version does no intermediate allocation. Additionally, some collection inputs are "internally reducible" (including range in 1.7) which allows them to be traversed more quickly via reduce/transduce contexts. The performance benefits of these changes can be dramatic as you increase the size of input data or number of transformations. If you happen to have your data in an input collection (like a vector), into with transducers can be used to move data directly from one vector to another without ever producing a sequence.
3. Eagerness - if you know you will process the entire input into the entire output, you can do so eagerly and avoid overhead. Or you can get laziness via sequence (although there are some differences in what that means with transducers). The upshot is you now have more choices.
4. Resource management - because you have the ability to eagerly process an external input source, you have more knowledge about when it's "done" so you can release the resource.
If you don't know Clojure, you should know that Clojure has a sort of generic collection type called a 'sequence'. Arrays, Lists and Lazy Sequences are data structures that implement the sequence interface. There is a standard set of functions, 'map', 'reduce', 'filter' that operates on that sequence interface and provide a common set of functions for different kinds of datatypes. This lets the programmer reuse code and idioms and makes programs easier to understand overall.
The problem is that there are datatypes, such as streams and channels, which are fundamentally sequence abstractions but for one reason or other don't implement the 'sequence' interface. You could wrap them in a lazy sequence but that can be complex and will introduce a performance penalty. When they wrote core.async, they the authors found themselves reimplementing 'map', 'reduce' etc, for channels.
Transducers are a way to provide sequence operations to any kind of collection split the part that interfaces with the collection from the operation. This way we can dispense with the old 'map', 'filter', etc and just use transducers for everything. Because they're implemented with reducers, they're faster as well.
It improves the performance by fusing some of the operations doing them in one pass instead of multiple passes, and it does so generically - operations are composed regardless of the source, and the implementation isn't looking at the type of the source at all.
I believe one of the arguments was also that these couldn't be written in statically typed languages. Although, I do not know if this turned out to be true.
>I believe one of the arguments was also that these couldn't be written in statically typed languages. Although, I do not know if this turned out to be true.
It's not true, they were just abstrusely defined and there's some weirdness around the implicit effects that you can choose to either ignore or incorporate into the equivalent.
Fusion is one of the main things Haskell is known for and an API that gets close enough to what one would use transducers for is `Control.Lens.Fold`.
Rich didn't claim you couldn't implement it. He just claimed that some aspects of transducers are difficult to represent as types. Or at least, that's how I took it.
My use cases are 1) to emulate yield / a pipeline of generators 2) to avoid intermediate object creation. My code is an indirect port of some Haskell code, so it's a tad awkward, but overall Transducers were a small perf win over reducers (which was a big perf win over lazy seqs) and also provide a good effectful-ish model for my pipeline.
My only negativity with Clojure isn't really with the language itself, but with the Debian / Ubuntu packages, they're way outdated. Not sure who ever maintained them, or why they stopped doing so 1.4 being the last version. Outside of that for anyone wanting to check it out, you could try downloading LightTable and using ClojureScript, seems to be close enough I am able to use Clojure books with ClojureScript, not sure entirely of it's differences or how backwards compatible one is meant to be with the other, though I suspect they're meant to be near identical aside from their available libraries maybe.
Clojure is often billed as "just another JAR" and in my experience I've never felt the need to install via any package manager. My workflow often consists of a new leiningen project where I define the version of Clojure that I want to use for the current project, along with whatever I else I need for the task at hand. Likewise, if I just need a quick repl to test something out, I'll simply use `lein repl` and fire away.
If you're familiar with Python Leiningen takes the place of both pip and virtualenv. Every project has a project.clj file where you declare your project dependencies and running "lein deps" from your project root handles it from there. This includes libraries and the version of Clojure you're targeting. When you add/delete/change a dependency in project.clj you simply run "lein deps" again. You never have to run "pip freeze" or make a requirements.txt file because project.clj serves that purpose as well.
It really doesn't make sense to 'install' clojure the language. Generally one would use apt to install java. Then, you'd download 'lein' the clojure build tool (which does not require a clojure 'install'), and then specify the clojure version in your project.clj file. Lein builds your project and downloads the correct clojure version.
For distributing your app, you just compile it to a JAR, and the user only needs plain old java to run it.
The reason is that nobody (that I know, at least) uses them. The Clojure community uses leiningen, and leiningen takes care of fetching the version of Clojure a specific project needs. There is no point in installing one version of Clojure when the core is small enough that every project can have its own.
Just install "lein" and then in any project run "lein deps" to use maven to pull in all dependencies, including the version of Clojure that you have in your project.clj file.
Clojure is really a simple language to use, which is one reason why I like it a lot (and use it for most of my projects). Compared to Scala and Haskell, Clojure is fast to learn. Java interop with Clojure is also really simple. I often use existing Java code by including it in my Clojure project and adding the magic Java build stuff to my project.clj file. Some purists might like to have just Clojure in their projects, but being able to pull in my old Java code (and other open source libraries) as needed is a large time saver. If Java libraries are small, I like to have them in my project. I use IntelliJ for both Clojure and Java development and having all the code in one place is often nicer than pulling in JAR dependencies.
I am reading several negative responses to this comment and I feel the need to throw him a hand. I like the idea of having a single place where every application is installed, to the extent possible. I like to try to install everything with apt-get, and I feel there is value in working that way.
It's way more complicated to have apt-get for some things, lein for other ones, pip install, quicklisp, the ruby gems, emacs packages, and every other environment reimplementing something that should be possible to reuse.
From the outside, and without researching a lot, it seems that the problem is that authority/responsibility is not easily delegated in the right way.
Perhaps someone can create something like a federated-containerized(contained?)-buzzword-github-like-blockchain-social-aptget-buzzword-thingie. Only half-joking here, but perhaps there's something a la git/dropbox, a product that you don't see coming until you turn the corner and They Just Got It Right. Something created on top of http://ipfs.io , for example, which looks very promising!
EDIT: Perhaps a simpler solution is to enable apt to delegate specific domains to packages marked as package managers, and have them talk through a specified protocol...
Yup, when they started appearing I didn't like them either but then I realised they have some nice wins as well, who better to package the packages the community produces than the community who wrote them.
It sounds like you are referring to various ideas around delayed var loading to improve startup performance - most of that was delayed, although there was one change that improved compilation speed and improves the performance of some projects.
To be fair, GHC and the Haskell ecosystem is far more complex than Clojure and its ecosystem/standard library. Nevertheless it's pleasant how easy Clojure was to upgrade (although of course this stability is nothing special: more conservative languages like Go and Java generally break almost nothing on upgrade).