If this were generally true (of any technology X) the dollars would be flowing that way in an unstoppable torrent, and anyone using anything else would be the subject of ridicule and/or pity. It would be like writing custom websites in C.
Since that isn't happening, I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people. So yeah, citation(s) please.
The reason is simply that more powerful tools require more skill and experience to use effectively. It's not just a question of switching from imperative language X to clojure and seeing a huge productivity boost. You must also understand why clojure is designed the way it is and how to use it effectively. For awhile, you'll likely be less productive as you learn how to approach problems functionally and wrap your head around the capabilities of lisp, but you'll also be building stronger, more reliable systems, and at the end of the learning process, you will indeed be significantly more productive.
Managing a transition like this is far from trivial for a company facing a variety of immediate demands and constraints, even if it promises to pay dividends in the long run.
> It's not just a question of switching from imperative language X to clojure and seeing a huge productivity boost.
It depends on who you are. I recently used Clojure on a skunkworks feature for our product for which I had three days to complete. I am absolutely convinced that it was at least 3x smaller (LOC) and took me at least 1/3 of the time it would have taken in Java. I am an expert Java programmer and not a newbie to Clojure, but I still have to look up a lot of things.
The killer productivity feature of Clojure (LISPS and other languages) is the REPL. Trying things out, analyzing data structures, etc. is where I would have spent a ton of time in Java. Even if you're not an expert Clojure programmer, just using the REPL can save you tons of time.
So not necessarily disagreeing with you, but I also think that a non-expert in Clojure can still run circles around an expert Java programmer when considering productivity.
I'm not disagreeing with you, but exactly what is it that you're doing in the REPL that you're not able to do in, say, Eclipse's Display view while having changes pushed into your program under debug (when it's able to)? Java disallows pushing some changes into the VM, but many changes go right in during debug.
It's not about what's possible in principle, but whether the tools facilitate a certain workflow. When you work in Clojure everything is interactive. You can write a function evaluate it immediately, see the result, use it in the next function and so on.
The workflow is very natural and you practically never have to restart anything. You can build up state in the application, see how things interact, reload functions add new functions, and so on. Using a debugger in Eclipse is a very pale imitation of this.
I hadn't ever used a REPL to facilitate my programming before utilizing Gradle & Groovy to code some build files. The REPL made it really easy to experiment with syntax and facilitated my learning as much as Intellisense used to when I worked in the Microsoft stack.
Every café in town might claim to have the best coffee. While most of them are lying, that doesn't mean that some don't in fact have better coffee than others. It's up to you to judge each on its own merits, read reviews, or get a recommendation from someone you think has good taste in coffee.
Support for clojure's worldview is easy to find on the web. It comes both from the clojure community itself (see Rich Hickey's talks) and from the wider communities that clojure draws its ideas from (other functional languages, lisp, etc.) Decide for yourself if you find the arguments compelling. In my experience, the clojure community is not interested in promoting clojure for reasons of tribalism or vanity, but because they can indeed use it to produce better work faster.
> In my experience, the clojure community is not interested in promoting clojure for reasons of tribalism or vanity, but because they can indeed use it to produce better work faster.
While I agree the quoted assertion is bogus, I disagree with your reasoning. There are many arbitrage opportunities that are being missed, whenever technology, science, or anything that requires thinking is involved. This is due to human stupidity and/or unwillingness to learn new things.
And there's many cultural forces at play. Many managers would pay a premium for a large disposable supply of programmers, rather than a smaller more skilled supply. Bargaining power becomes a factor; the smaller labor supply may demand more decisionmaking power than you culturally want to allow.
Just to point out, MapR isn't a slam dunk for a couple reasons. Their HBase implementation is very fast, but it involves using their whole Hadoop distribution. In my experience:
- the management UI isn't as good as Cloudera Manager, or even Ambari in HDP2.0
- 3rd party support isn't there. If you want to use anything you find on GitHub, get ready to add support for MapRFS, and revert the API version back to 0.21
- there are bugs. Weird, specific bugs in the filesystem implementation which will bite you once every few weeks. This is one area where they lag the ASF project: many eyes do make these kind of things shallow. Their smaller install base and fewer developers means these things take a long time to notice and fix.
Their HBase implementation is very fast, but it involves using their whole Hadoop distribution...3rd party support isn't there. If you want to use anything you find on GitHub, get ready to add support for MapRFS, and revert the API version back to 0.21
M7 Tables are designed to be a drop in for HBase, with better performance and without the HBase complexity (it's much easier to run M7 Tables as the primary datastore for online apps).
M7 Tables support and certification for several key projects is coming along...
Confusing correlation and causation. Clojure coders might be 3x more productive, but they would be in any language. The chance is high that only the top 1% (or something like that) of all coders would dare to make the switch to this language, so you get a biased sample to begin with.
The problem with this of course is that when a company tries to switch their more mundane engineers to Clojure, the project will likely fail.
> Confusing correlation and causation. Clojure coders might be 3x more productive, but they would be in any language. The chance is high that only the top 1% (or something like that) of all coders would dare to make the switch to this language
It's too bad that these top coders are so bad at mundane tasks like high-school-level statistics.
I disagree. I think that top-1% programmers actually fare worse when thrown into a typical Java codebase. Sure, they're very productive on their own stuff (whether in Clojure or Java, although I still think those programmers will fare better with Clojure) but if you dropped a 1%er (or even a 10%er) into a pile of someone else's FactoryFactory garbage, he'd either rewrite it or fail completely to make any headway. Bad code is the Waterloo of good engineers.
I've seen companies end up in inversion, which means that the most apparently productive engineers are the worst ones. It's a common effect in "design pattern" Java shops with all their Visitor and Factory and Vibrator and AbstractSingletonFactory patterns. It's not pretty. Good people are either taken badly (because they keep agitating for rewrites) or become disengaged and eventually leave or are fired.
I, as a single Perl (and regex, and VBA -- laugh all you want) programmer, have done things for which an EDS contract was ruled out by a major institution because it (and however much in terms of resources and expense it would entail) would be "too expensive". (Instead, they chose to continue "throwing people" -- full-time staff -- at the problem. Use that as a measure of just how expensive that prospective EDS work was considered to be.)
The language, or its aficionado -- either way, specific technologies have made me, in my own estimation, multiple factors more productive. Yes, there's the learning curve, and understanding things that are not explicitly stated line by line throughout the code. But then, there's getting shit done.
Unfortunately, a lot of people, processes, and institutions get hung up at the sentence before last of that previous paragraph.
I also turned other people onto some useful technologies, such as pointing a consultant at some reporting software that my department didn't want to spring for but for which -- of course -- the outside consultant had budget. I heard from her a month or two later that it had greatly eased things and turned her into a sort of reporting superhero.
Throughout my career, I've seen people who, both on their own as well as through dint of better technology awareness and selection, have been multiple factors more productive than their coworkers.
And yes, often they are fighting the trend and significant inertia, rather than being embraced.
IMO the problem with the unstoppable torrent is that the people who decide where company dollars flow to can hardly tell C from Clojure so they rely on the opinion of those whose job would then be to say "ah yeah right, me and my team have been using the wrong tools all the way" which doesn't happen for obvious reasons.
I started up with clj and I am trying kinda heavy arbitrage here :)
The line on slide 111, "Engineers 3-10x more productive, smaller teams", is probably in reference to...
Slide 16: "Code is typically 3-10x smaller when written in Clojure."
Slide 110: "3-10x less code. Limiting factor in programming speed is essential complexity of the problem, not typing."
I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people.
That there is a "a gigantic arbitrage opportunity being missed" is essentially the thesis of the talk.
He states the reason why (Clojure is cloaked in 50 years of Lisp misconceptions) on slide 21: "Lisp is an old weird AI language no one uses anymore. It's so slow it requires a supercomputer. It's impossible to hire people who know how to use it. Functional programming has been proven impractical. Also, parentheses."
The body of the talk is designed to dispel these beliefs.
Yeah, I think he's just confusing the economy of LoC with productivity. I might actually be 2-3X faster with Clojure over Java, but I'm sure I could crank out a CRUD web app in Rails 2-3X faster than in Clojure at the moment..
> I enjoyed the presentation very much, until
> "Engineers 3-10x more productive"
> Citation needed.
Oh, it's a perfectly plausible number. It increases the productivity of each developer by a factor of 3 to 10 and also increases the maintenance costs by 3^n_engineers or 10^n_engineers. This seems to be the tradeoff that startups make.
I remember when Ruby on Rails developers were claiming this. Yet, I couldn't find one to deliver a website at 1/10th the price...or even 1/5th the price.
EDIT: (yes, being a little cheeky because as we know the physical time programming isn't everything in building a website)
Here I wanted to cite the (moderately) well-known result that defect rate is proportional to lines of code, independent of language, but I can't actually find where it comes from.
3-10x is, of course, intended as a 95%-of-cases interval with some basic assumptions. Some contexts (e.g. type of problem, engineer, timeframe) will be below 3x and others will be above 10x. (The 99.99% interval is probably 0.2-100x.) Among those basic assumptions is that the engineers have the curiosity, drive, and ability to learn it. Ability is the least issue of the three; if they're not smart enough to learn Clojure, they're not smart enough to Java or C++ well, and they're mostly making messes.
Lisp has a bad reputation compared to what it is. People think it's impractical, that Lispers are arrogant (due to a few vocal but influential people in the Lisp community) and that it's archaic or weird. I've addressed those perceptions, but they're out there. Clojure is, in fact, a pretty damn practical language.
Now, the arbitrage argument: in larger companies, the concerns aren't engineer productivity so much as appearances. For a middle-manager to launch a meaningful Clojure/Lisp initiative would be to put himself out there in a major way, and since programmer productivity is only one piece of a much larger system, it might not have that big an impact on the company as a whole. It might not be worth it. (Would Google's market cap go up 3-10x if it replaced Java with Clojure? Almost certainly not. Maybe the fair value would improve by 1.05x on fundamentals, which would be drowned out by month-to-month volatility.) On the other hand, startups generally can't afford (or don't think they can afford) to experiment with newer, quirkier languages. VC-funded startups tend (surprisingly?) to have stodgy closed-allocation work cultures and boring tech stacks because they're expected to allocate 100% of the "risk allowance" to one department: business-model risk. That's not to say that there aren't other kinds of companies that can (and, in some cases, do) use more innovative tools; but the two dominant cultural forces (VC startups and large corporations) in technology don't favor it.
For the large companies, I'll admit that there's an Amdahl's Law sort of effect at play. Let's say that you improve productivity of engineers by 3-10x. Does that "3x" the whole company? As I alluded to for Google, probably not. It makes something else the limiting factor. For most firms, using the wrong programming languages is nowhere near their top problem. To use Google as the example, it might deserve a 5% bump on market cap if it included Clojure (a small gain compared to monthly volatility, so small that no one could prove causality and take credit) on the white-list but the fair value would go up 50-200% (again, on fundamentals, as I can't predict investor opinion) if it implemented open allocation. Programming language innovation just isn't the top priority for most companies. But for an individual developer, the advantage conferred by a powerful language can be pretty serious.
People choose what they work on. No "headcount" numbers that have to be agreed upon by people far away from the actual work. It's the one thing Google could still do at this point that would 2-3x the company.
Thanks for this. It's a great overview that manages to be both detailed and easily accessible.
Your visual approach to explaining macros with arrows to demonstrate the substitutions also really helped me to grok them in a way I haven't been able to previously. They are actually quite simple aren't they? It's basically the same idea as html templating, but outputs code instead of markup.
I'd love to see you get more in depth on some of the headier topics--core.logic, monads, storm/hadoop interop, etc. You seem to have a gift for making difficult concepts approachable.
To be clear, that ` character is called "syntax-quote" (in Clojure; "quasi-quote" elsewhere) and it's the "templating" part. It's both possible and common to use macros without syntax-quote and vice versa.
In fact, common practice in Clojure is to define a your macros in terms of a backing function:
(defn foo-fn [body]
(fancy stuff here that probably uses syntax-quote somewhere))
(defmacro foo [& body]
(foo-fn body))
Clojure's style of macro system is known as a "procedural" macro system because the macros can be any arbitrary procedures which returns code as data. There are other types of macro systems, like Scheme's syntax-rules, which are more directly embrace the fact that most macros are tree rewrites.
Syntax-quote is a little bit different from quasiquote. Clojure's macros look a lot like Common Lisp's unhygienic macros, but Clojure's syntax-quote offers a sort of "hygiene by default" by forcing namespaces on variables and making it hard to introduce non-gensym symbols accidentally.
Question - there was a recent post about how the dailymail group moved 140k lines of java code to 4k lines in clojure.
Since I have never programmed much in Java (I went from c++ to ruby!), could someone tell me where the gains are coming from ?
I know that eliminating the setters and getters (and the private/protected/etc system) would eliminate a lot of code, but I dont understand if it would be that much.
Possibly, I'm also trying to understand if they would have gotten similar improvements in kloc if they had switched to.. ruby/python for instance.
Hi, I'm the CTO at the MailOnline so can comment. The biggest reduction in loc was due to the move from a relational DB to elastic search containing flatter representations of our content. This basically meant that we could delete thousands of lines of boilerplate and create a very simple processing pipeline to render content, and for this clojure really shines of course.
The REPL made the first week of exploring the problem and solution space really effective, and even as a node guy I was impressed with what could be done.
I can say that there is definitely a big productivity boost, but as others have said it isn't purely the language, but rather thinking about the problem in a completely new way that really did it. I do credit clojure though with giving our engineers the ability to think differently, as that is often the hardest part.
There was a comment I made in a speech recently about functional programming being frictionless for content management system design, this is what I was alluding to.
I don't want to start a flame war, but we built an earlier version in ruby (with one of the best OO guys I know leading it), and it was much more complex through attempting to model our home page as a set of inter related objects, using visitor patterns to traverse etc., than the clojure version.
very interesting. I had a similar experience with elasticsearch in modelling a ecommerce catalog - however that was with jruby. In fact, it took me quite some convincing to not go with something like mongodb - elasticsearch served us quite well.
Could you comment a bit more on the interaction of functional programming to content management ? Intuitively, it feels to me that a lot of it has to do with the NOSQL-like data model of elasticsearch. Being a lisp beginner, I feel that a visitor pattern is very analogous to "map" in functional programming - again, without all the OO getter/setter functions (e.g. "accept", "visit").
A lot of the simplification was definitely down to flattening the structure that was represented in 20-30 tables down into a smaller set of documents in ES. But the functional approach to then rendering these documents simplified it further.
A functional map is definitely analogous to a visitor pattern, but don't underestimate how much of a productivity boost arrives by being able to see all of the code in a small set of terse statements in one place, vs having a set of fairly abstract classes with accept/visit scattered all over the place.
We just found that the OO solution resulted in more code and it was more difficult to hold the resulting model in your head as the number of things that could be visited grew - we run a very big site.
The other big enabler vs a traditional CMS approach is that we have a very clear separation between the part of our stack that allows editors to create content, and the part that renders it (connected via messaging) - so we aren't mixing r/w domains and our rendering pipeline isn't burdened with the additional complexity of having to understand how content changes, it just renders what it is given as fast as it can.
Java (at least prior to Java 7), has some severe limitations in how code can be abstracted. As an example, it doesn't allow you to pass functions as arguments to other functions directly. As a result, the canonical way to handle situations where you would do this (such as passing a comparison operator to a sort function), is to declare an interface, define the function in a class that implements that interface (command pattern, etc.), and then instantiate an instance of the class and pass that. Closures are imitated by explicitly passing data to class constructors and storing it in member variables, and currying imitated through patterns like the onerous abstract factory pattern.
Additionally, the abstraction features Java does offer are implemented with some heavy language syntax, and often can't be generated at runtime or pre-compile time, at least not easily, and so there's a limit to the abstraction, even taking into account Java Generics. That's something where homoiconic languages like Clojure have a particular advantage.
As a result, abstracting things in these ways (which in other languages is incredibly common) gets to be painfully verbose. Often, things that would be beautifully abstracted in these ways are just left unabstracted (as in-line boilerplate) - the verbosity introduced makes the code less readable, rather than more, and so there's a stigma against "over-abstracted" code. The result is that code that does, or could, make use of these things takes exceedingly more code in Java than in a language like Clojure. Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.
> Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.
My comment is basically a free-association from this.
In C and Java both (and, um, CSS), the people I've encountered seem to strongly prefer giving each matched pair of curly braces one line (that is, the opening { doesn't get its own line, but the closing } does).
In lisps (to my knowledge) it's more normal to have all your closing )s stuck together at the end of the previous line. Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead.
Python formalized the indentation-is-scope paradigm, which means a malicious programmer can't trick you into believing you're looking at a different scope than you are, but my Python code always ends up being a lot taller than I feel it should be, so I have a lingering sense that Python took it a little too far. It turns out that I occasionally appreciate the option you get with lisps to put a short bit of code all on one line and mark the parse tree with parens instead of with line-breaks-plus-indentation.
Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead.
Another handy tool is an editor with rainbow delimiters. This feature changes the colour of each matched pair of parentheses so that they can be seen at a glance.
Seems to me that if you have a clump of )))), you should refactor that into individual functions instead. This would be akin to having a 500 line Java method.
There's really no excuse not to, since refactoring functional code tends to be as simple as taking a node in a tree and moving it out.
Since you use () for more or less everything in Lisp, from declaring functions or records(structs) to calling functions or doing comparisons (<, >, =, etc. are functions in Lisp), you will always get a clump of ) at the end of something.
In Java, if you have a class containing a function which itself contains an if-statement containing multiple lines. You will have three lines at the end of the file containing nothing but }. There is no reason you couldn't clump those together and thus end the last statement in your function with }}}.
The difference is, in Java it's good practise to give each } a seperate line. In Lisp, most people prefer to clump their ) together.
Also, you could easily end a function with alot of )))))).
I imagine yogthos was referring to this, but the clump of `)))))` itself is only a symptom of the real problem: that you've got too much nested logic.
`)))))` at the end of a function doesn't matter. The function's over. Even if you needed to make a change that would break the clump up, you generally navigate it from the opening side.
But if it's at the end of a fat true-expression of an if-statement, you probably have to add a `;; x is false` comment to remind yourself what's going on by the time you get to the false-expression.
Just like in Python when you get to an `else:` yet you have to scroll up just to find the matching `if` because somebody decided to roll their own SOAP client on the true path.
As we're already off-topic, I'll chip in with my view.
I personally hate the opening { stuck on the end of the line in C/Java. For me it goes on a line on its own pretty much every time.
I've lost count of the amount of times there's been a subtle bug introduced by incorrect scope as a result of a { being missed from the end of a line and not being noticed due to the code being formatted as if it was there.
With a { sat on its own, directly lining up with the }, that's pretty much impossible to miss.
I've always put it on the end of the line and found it easier to parse visually than when its on a separate line. To me that just seems like a waste of space.
How do you quickly check to make sure that all of your closing braces match up with the relevant opening brace, rather than, for example, an if statement that you forgot to put the { on?
Hopefully the formatting will work for this -
For example,
if (isOK)
for(thing in list
{
doSomething(thing);
}
}
Screams out to me at very first glance that I've missed a {,
whereas
if (isOK)
for(thing in list){
doSomething(thing);
}
}
needs me to scan to the end of each conditional line to see where the problem is.
For me, it's the other way around.. the second instance is easier to spot because the visual cue of associating the block with the statement is more obvious if you assume beforehand that each statement has to end with the brace.
As for closing braces, I associate them with the start of the statement itself (the if and for) and not the opening brace, assuming the indentation is consistent.
I'm not arguing one is necessarily more right than the other - just that I prefer the way considered less right by consensus.
I get the impression that the { on the end of the line is the one that most people prefer, so I think you're more with consensus than me.
The argument about taking up less space, I get - although given the massive monitors that we all have these days, that seems far less of a consideration than when I used to work on 20-line terminals.
What I just genuinely struggle to understand is how it's clearer (and if I managed to understand, it might help me parse other peoples' code).
Let me turn my code into something that actually has a matching amount of braces
while (readLine){
if (isOK)
loadSomething();
for(thing in list){
doSomething(thing);
}
}
versus
while(readLine)
{
if (isOK)
loadSomething();
for(thing in list)
{
doSomething(thing);
}
}
If I was quickly parsing the first code, I'd assume everything was fine and that the first } was closing the if(ok). To spot the defect, I have to scroll up, find the matching control statement and check to the end of its line to see whether it had a {.
With second one, even the most rudamentary scan tells me that there's an opening and a closing brace that don't match up. All I have to do is scroll up and see whether there's a matching { at the same indentation.
I'm not saying you're wrong - if you are, most people who write coding articles, or code that I have to look at, are also wrong. But I would love to know what it is that other people can see easier/quicker in the first code than they can in the second in.
Some reduction in lines of code may have come if they had rewritten it in Java too, due to better understanding the problem space after having written the first version.
Java has a fair amount of boilerplate type code other than getters and setters. For example forcing you to use "classes" for everything, even when you really want a procedural or functional API. Some people try to write semi functional style programs anyway using object oriented "patterns", which can sometimes lead to really just a lof more lines of code too. Also explicit type annotations take up space. And the Java style of using exceptions can be quite verbose. There may also be better designed libraries to do a lot of things for you automatically in clojure. Macros are very helpful too. But I'm just making all of this up, haven't actually seen the codebases to compare them.
Its very possible they would have gotten significant (though possibly not equivalent) reduction if they reprogrammed it directly in Java. An old codebase often has a lot of useless cruft and over-abstractions and so on, and you do better each time you code something. The experimental control is not the old code base, it is translating the same code base back to itself with similar engineering effort. No wonder few people actually try to gather that sort of data.
There is a lot of ceremony in Java. If I want a single polymorphic function I'll create an interface and a bunch of classes to implement it, instantiate those classes and pass them to where they need to be.
In a functional program, I'll just create the function and pass it around anonymously if needed. The function will in itself will likely be simpler than the Java equivalent and stateless.
There will also be less DTO type class code for carrying around dumb collections of data, replaced with simple records and lists.
I'm not at the stage where functional programming has really clicked for me, but after writing some Clojure and Haskell recently, the sheer amount of code you have to type in in Java to express yourself is really starting to bug me!
I think it's also in the mentality as in it's perfectly fine in lisp to use simple lists as data structure, whereas we mostly use wrapper objects around primitive or collections in Java.
For instance, you'll often see a "Person" class definition with getters, setters around the .adds collection function. The idea is to encapsulate the collections as much as possible so that if it changes down the road, nothing else is affected but that simple method. In lisp, however, you'd probably just (let (people '("tom" "john"))) and be done with it. I'm obviously exaggerating, but hopefully you get my point. It's similar in Python where I use dicts all the time, but feel bad using it in Java or C++ without abstracting or encapsulating it.
It's not so much a syntax issue (even though clojure syntax is less verbose than Java) but mostly the mentality behind the idioms IMHO.
The misconception list basically summarizes the interactions I have with people who know I use Clojure. It's really interesting that these are still around, especially considering that lisp hasn't been new for a long time.
As an aside, I loved the chrono trigger reference! I was actually listening to some of its music as I viewed the slides.
I know this is old hat for the FP community, but it's worth mentioning that the dot product would be normally be written as (reduce + (map * xs ys)) (or something similar) for the sake of readers scared by the loop acc pattern.
It's really interesting that these are still around, especially considering that lisp hasn't been new for a long time.
People want to believe the world is simple, and that learning one PL is enough. Once you know Java, why learn another one? Either they're all the same, or they're so radically different they must be weak in some critical way.
When you grok Lisp, and realize that it is actually practical, you realize that you've only scratched the surface of the PL problem... and that's scary for a lot of people, especially as there's a tendency to overestimate the barriers between languages. (The Internet runs on components written on all sorts of different languages.)
it's worth mentioning that the dot product would be normally be written as (reduce + (map xs ys))*
Yes, I had that thought, too.
In fact, that formulation is too slow for, say, large-scale ML. I've been getting a lot of mileage out of Prismatic's hiphip library... and I really look forward to, about 5 years from now, when Clojure is as mature for numeric programming and ML as Python is with Numpy, Scipy, Numba et al.
(Disclaimer: I am a python that found a new home in the scheme world)
Clojure is a Lisp. Lisp based languages are great.
However:
One of the problems with clojure - or rather with its home or ecosystem - is the fact that it requires a very high activation energy to get going. Grasp the concept of functionial programming, cc/call, macros, totally different syntax than your previous imperative language of choice, and last but not the giant blob that is called Java platform with all the traps regarding project management, IDEs, building, compiling, deploying etc. I know that there are great tools such an Leiningen or LightTable, but they only make the whole process of transition more manageable.
Now, I know that Java is a huge benefit for all that want to make use of its gazillion tools. But if You are not into Java, then Clojure might give you some hickups.
In comparison to Clojure, Python also allows for some functional programming, gives the eager disciple an easy going start. Most of libraries are written in Python (or exist in a fast, a C-esque implementation) and are very easy to grasp. I can not say that for the Java enviroment Clojure has been born into.
I frown at java lingo error tracebacks in the REPL, and to bring a external Java library into play is even more problematic. Any thoughts on this how to remedy this problem without getting involved into Java?
I wish there would be some kind of Index of how to use certain Java libraries from Clojure, or an index of Clojure wrapping packages. It might make life a bit easier.
Finally, I claim that for now, I am quite happy with my choice to migrate from Clojure to Chicken Scheme/picolisp. You can get quite a bunch of Clojure data types from the good package repository. It compiles without setting your hair on fire. And it is really simple, as the language basics are small.
I will keep an eye on Clojure. but I must say that Julia is as appealing to me when it comes to the choice of future language.
Agree on the Java ecosystem. I deployed a toy app using Tomcat a couple of months ago, and now I have no idea how to update it. I lost the "tutorial" I found and can't remember what it was (stupid mistake), but still, the fact that I had to resort to a pretty unknown tutorial to deploy something shows how desperate I was.
I'm looking for another language with similar philosophy to Clojure's, but with a better ecosystem.
http-kit lets you run your own app server like Node.js does instead of sticking it in a servlet container. You just call run-server and you've got a working web app. So yeah, basically a Jetty replacement, but it's less tied into the Java ecosystem and more into Clojure.
Oh ok. Not familiar with Node.js but if it means getting out of Java then definitely worth a deeper look. Thanks. Have any links to quickstart type tutorials? Something that allows someone to deploy a (toy) webapp?
Clojure has a simple scratch->toyapp->deploy workflow. It's just that to find it as a newcomer, you either:
A. Luck out and find it immediately.
B. Get directed to it by someone more experienced.
C. Chase a suboptimal solution because you don't know better. You simply don't know there's an easier workflow. You don't know you can eval code within an editor. You don't know there are tools like paredit or how they'd help you.
For toy webapps, it's good to know these incantations:
lein new compojure myapp (creates webapp skeleton)
lein ring server (starts embedded server for dev)
I don't get it. Scala has all the features, it's fast AND it's easier to read (no matter how much the presentation tries to play down the () issue, it's there.)
no matter how much the presentation tries to play down the () issue, it's there.
I'm getting tired of statements like this. I've been using Clojure for a couple of years now. I got over the () "issue" after a couple of weeks, once I discovered Counterclockwise for Eclipse (and, eventually, Emacs paredit or smartparens). You just don't notice them after a short period.
In general, editors that "understand" Lisp-like syntax are more powerful than other code editors. Being able to select forms - anything enclosed in a (), {} or [] - with a key combination and move them easily around the hierarchy (also see barfage & slurpage in the paredit cheatsheet: http://www.emacswiki.org/emacs/PareditCheatsheet) makes editing code faster than other languages.
As for the "easy to read" argument, I now find Clojure easier to read than, say, Java because there's generally less of it.
Agreed. I remember hearing the exact same about significant whitespace in Python. Its just one of those things that, after a little bit of use, simply fades away into the background and stops being an issue. Many people even grow to like significant whitespace in Python and parens in Lisps.
Clojure is dynamically typed, homoiconic, and places a lot of emphasis on immutability and managing state. Scala has different design goals. Idiomatic Clojure code is quite different, structurally, from idiomatic Scala. You should try it, and believe it when everyone says that the parentheses fade after about a week.
There are great reasons to use Scala (specifically Akka, static typing), but syntax is a bullshit reason. A decent programmer can adapt to normal languages like ruby, scala or lisp in about a week of use. Wacky languages like perl or k might take a bit longer.
Also, syntactically Scala is a lot more complicated than lisp. It's a bit closer to Java, but that doesn't make it simpler. Just more familiar.
frankly, in the world, people are using scala instead.
however, i have to differ with you completely regarding scala being easier to read. as someone who regularly works with scala code i find it incredibly difficult to read in comparison to clojure. Lisp syntax is different from what most people are used to but is very simple; scala looks more like the syntax that people are used to, but is deceptively quite different, and is quite complex.
Can't agree with this more. All the plus pointed touts for FP are available in Scala without the necessary mental context-switch that is required to grok the parenthesis extravaganza. Why Clojure instead of Scala?
The 'easier to read' part is valid maybe for the first week of working with a Lisp (commenting as a part of a team which switched from mostly-Java to only-Clojure several months ago).
Pros for Clojure:
* Faster code -> result cycle. No need for compilation (except for AOT-compiling releases).
* Absence of static typing helps in some of the domains, e.g. web applications. We have successfully used Prismatic's Schema[1] in places where a stable data interchange format was needed.
* Binary compatibility. No need to recompile libraries for different Clojure versions.
* Community oriented towards simplification. People mostly try to mimic Rich Hickey and Clojure.core development approach.
* Orientation towards small libraries. I haven't yet had a need to use a framework written in Clojure (although Pedestal[2] seems interesting).
How complete is core.typed? Does it have a good story for scala type features I've come to depend on (covariance, some kind of typeclass-like functionality, higher kinds)?
Well, first of all Clojure is a substantially simpler language than Scala, which counts for something.
Secondly, I believe that Clojure does a better job of presenting the developer with a new way of thinking about problems, which (as alluded to further up the thread) can be a big source of productivity gain and inspiration.
I'm not sure the precedence of syntax makes it simpler, though. In the end, it's just syntax, but the primary posit that's been put forth is that Clojure is simpler to use than languages such as Scala. And Polish notation, while existent, is not the common form of expression in mathematics.
I find it "absurd" in that it requires adaptation to a form of syntax that's more niche than anything. I don't see that adding to the argument of simplicity in this particular scenario.
I'd guess that, for you, Clojure _wouldn't_ have the same kind of efficiency that some others see. Lisp, in general, appeals to people who think "A is really the same thing as B. Cool!" They see treating all functions the same as a really worthwhile thing, in and of itself. Others, such as yourself, aren't really excited by that. It's a style thing.
Fanatical generalization can be a really effective trait in some circumstances. A small team of generalizers working on a reasonably-defined problem (such as a re-implementation) can turn out amazingly simple and compact code, since they tend to factor out common things.
On the other hand, a generalizer might not be as tactically-focused as someone else. A tactical person might be more inclined to make a surgical change, whereas a generalizer might wind up with a "toolbox enhancement".
A tactical person is going to get more mileage out of tools that already take care of a lot of common cases, and do them in common ways. A generalizer will do better with tools that are more "meta".
All this from an assertion about the simplicity of Clojure syntax?
Because this is just syntax, "A 'looks' the same as B. Cool!" is a more definitive statement. After all, it's just getting compiled/translated to JVM byte code anyway.
Efficiency is an entirely different conversation and tangential to the one we've been having regarding syntax and simplicity. Nobody has asked to this point, but I have no problem with Clojure's syntax structure. I find it very compact, I like the explicitness of parentheses, and like that it works in the JVM. (Disclaimer: I've not deployed any Clojure-based code in a production environment, so my own experience is limited.)
I also find that it's syntax has its own quirks, much like every other language. And while some things 'look' the same in Clojure, other things don't -- and that's perfectly ok. It's relatively consistent enough that the syntax logically points to its idioms.
But I stand by my original comment: it's only as simple as the observer makes it out to be. I don't find the syntax of Clojure more or less simple than many other languages.
The fact that <, >, +, - and other operators are just functions brings alot of benefits.
- They are variadic functions, they work on a range of inputs.
- Their names are valid identifiers, which allows for function names like list->vector
- Precense is fairly self-evident. 2 * 2 / 5 - 1 vs (- (* 2 (/ 2 5)) 1). You could add parenthesees to the first, and most would, Lisp just forces you to.
- They are consistent, a function is the first element of a list, no matter what
- Closely related to the above point, they make writing macroes easier. Imagine if macroes had to take infix operators into account. Certain macroes would have to scan the syntax they receive for the presense of operators, or you would have to create special macroes for operators...
- You would need special syntax for passing operators as values to other functions. ((dummy) < 5) ;; What should happen here if dummy returns another function instead of something orderable?
It's only absurd since we're so used to assuming that (some) math functions are special and deserve special notation. The notation for almost every other function you're going to use is f(x,0), why should the function '>' get a different notation?
I agree that it is uncommon and take getting used to, but it does have the advantage of being consistent, and consistency and lack of special cases is a part of simplicity.
> why should the function '>' get a different notation?
I think that the concept of standards is worthy of consideration. While '>' is an operator in scala and other languages, it's also an operator in basic mathematics. And in basic math, we're taught "if x is greater than zero" using the notation of "if x > 0" (parentheses notwithstanding.)
It's only consistent in terms of how clojure presents it, but it's inconsistent with treatment elsewhere. Consistency does go to simplicity, but if it takes some "getting used to", it can't be all that simple.
>It's only consistent in terms of how clojure presents it, but it's inconsistent with treatment elsewhere.
Well, engineers using scientific calculators handled polish notation (and reverse polish notation, like 3 4 + for 3 + 4) just fine for half a century or so...
check:
(> x y z 0)
vs:
if( x > y && y > z && z > 0)
Now the next question is: where do I need that for? Just let it soak in for a while: it is surprising how your solutions change if your tool works differently.
I've been busy with Scala for some time now, but the OO aspect also dominates the FP part. And that OO part is huge: traditional classes, case classes, traits (which are occasionally used standalone) static functionality not in the class but factored out in an object, structural typing just to name some.
There are many aspects I like of Scala, but simplicity is not among them. In teams you will have to have strong communication if you do not want to limit yourself to a subset of the language ("no fp" for instance)
Those are great examples, and I agree -- the compactness far exceeds something like Scala.
Although I think I could create a function in Scala that permits the same thing, yes? Mostly the arbitrary operands or range reference imply recursion & iteration, which I believe I could implement easily. (Check me if I'm wrong here, I might be missing something.)
I wouldn't have suggested that, except that Clojure (as I understand it) also encourages that one can "expand" the language. I would see adding this capability as a Scala function in the same light.
Which is basically the same as (foo a b) except the ( if one identifier later.
I've often seen stuff like this in languages without operator overloading: a.add(b.multiply(c) d.subtract(5)) and nobody complains about that, but a lot of people are hung up about (add a (multiply b c) (subtract d 5)) which is almost the exact same thing. Of course, in Lisp-like languages, you would be able to redefine functions, s it would actually look like this: (+ a (* b c) (- d 5))
I don't get why people have such big complaints about syntax. Whatever you are used to is easier to read. Obviously! But its not hard and does not take long to get used to this.
So once you are used to it, Lisp is simpler because it has one single rule that is followed by every language construct: (function params) - in other languages you have all kinds of flow control statements with different rules (if, else if, while, for, switch.. all have different rules). Then theres function calls and method calls. Precedence. That's a lot to have to remember.
I'm just going to copy-and-paste this old comment:
When evaluating language tradeoffs the scala community values power more and the clojure community values simplicity more.
Scala wants to provide you with the most powerful language possible. Clojure wants to make it easy for you to mold your own language.
Scala language tooling works at compile-time and analyses the language statically. Clojure language tooling connects to a running environment and uses introspection. Scala has much better static checking and clojure has much better introspection. While both languages have a repl, the clojure community puts much more emphasis on live, interactive programming.
Clojures syntactic extension mechanisms are much simpler, being based on sexps. Scalas are powerful (eg can access the lexical environment) but much more heavy-weight.
Clojure has run-time eval. This makes it possible to do eg domain-specific jit (eg compiling DSLs -> clojure code -> java bytecode on the fly). Scala has a run-time interpreter which I haven't used but is reputed to be slow.
I haven't yet explored non-jvm backends for scala but I suspect that clojure will eventually dominate on that front for two reasons: 1) when clojure-in-clojure is finished porting clojure to a new platform will be far easier than scala and 2) clojure was designed from the beginning to be a parasitic language and delegates many features to the underlying platform.
Scala breaks backwards compatibility much more often than clojure. Whether this is a pro or con is up for debate. There are lots of small niggling things in clojure I would like to change but on the other hand maintaining scala code across compiler updates is a pain in the ass.
I got really into Scala a couple of years ago, and then worked on a big Scala project. One of the problems is that if you throw average Java developers into it, and don't give them the time to learn the language properly-- these were smart engineers, it was a lack-of-time problem rather than an ability issue-- you get a lot of Scala-in-Java. The code is just as unreadable, "enterprisey", and unattractive as it would be in Java.
I would use Scala for a personal project where I felt I needed compiler-time typing: I was writing something for the ages, or where bugs might be truly intolerable (say, high-frequency trading) but code would remain small. I wouldn't recommend it for a huge project, though. (Then again, I generally wouldn't recommend huge projects.)
Scala tends to fall down on large projects, especially if the developers aren't skilled. Cyclical dependencies (yes, I know they shouldn't exist, but they will if you have a team of 25+ engineers and a wide skill spectrum) defeat incremental compilation, which matters when your compile speed is ~500LoC/second (Scala's compiler is very powerful). Also, SBT is a nightmare. I haven't used Scala's macro system but I can't imagine it being as clean as Clojure's.
I like Scala on its terms alone, even if I've seen it go wrong on large codebases. (Java has the same issues, but it has tools that rescue you. It's an ugly hackish thing, to rely on tools to deal with code messes; but they're there.) I like static typing (at least, the theory that comes from it.) However, I tend to think that what most programmers really want when they say they want "static typing" is something more like Prismatic's Schema: runtime checks that can be turned on or off (performance) as needed.
I think Clojure's easier to read and to write. It just looks harder, because people exaggerate the parentheses problem, which is not much of one at all. If anything, the only thing that's somewhat hard to explain to newcomers is quoting (e.g. why '(1 2 3) returns (1 2 3) at REPL but (1 2 3) is an error). The quotes were the hardest thing for me to get right when I started in Lisp.
It's not really accurate to think of core.async as mutable state at all, because by default channels are only queues in the sense of implicit waiting go blocks forming a queue.
Unless you use a buffer implementation, channels have no "placeness", you're not mutating/adding to a queue by parking a go block.
core.async has state (when buffers are used, in particular) but state is not exactly the same thing as mutability. Producers and consumers don't have unconstrained access to channels, in the way that users of a mutable cell (a Clojure atom, a ref cell, or just a variable in many languages) can clobber it at will.
Likewise (although I didn't get into this) state doesn't necessarily violate referential transparency. Memoization uses state as an optimization, but doesn't interfere with RT.
edit: well after actually reading your comment, I see now what you said :). Yes that seems like a good reason for companies to adopt Clojure, I hope they do.
Ah, yeah, I really wanted to go to Conj, but only found out about it last minute. I'm in San Francisco.
It's not that there are no companies using Clojure, it's more that the companies where I do want to work, don't want to use it.
When I mention it in an interview or something they kind of give me a funny look like they hope I am not one of those LISP people. I mean I can see where they're coming from. It's a big risk to let someone write in a language you don't know much about, and that you don't know if you'll be able to hire for in the future.
And at the places where they are feeling a bit adventurous, everyone seems to be leaning towards Go :(
Oh well, I can still use it for all my side projects until it gains more popularity I suppose. It really is such fun to use. I only started 2 months ago and I have been having the best time of it.
Adjectives, to me, are more unresolved but I think Clojure's protocols appear to be close to the right answer.
Adjectives are properties or classes of things, like Numeric or Closeable (with-open pattern) or Monad. The Haskell solution is type classes. The Java solution is OOP. The Erlang solution (fairly similar) is the actor model, which is the inspiration for OOP's principle of locally interpreted functions/methods (as opposed to global function definitions). OCaml's is functors, which are extremely powerful but hard for most people to use.
Perhaps protocols/interfaces are like restrictive adjectival phrases, and metadata like non-restrictive.
Just in case you don't know: Restrictive clauses limit the possible meaning of a preceding subject, whereas non-restrictive clauses tell something about a preceding subject, but do not limit its meaning.
Example of restrictive use: The suspect in the lineup who has red hair committed the crime.
Example of non-restrictive use: The suspect in the lineup, who owns a red car, committed the crime.
Interesting to see you mention Supply Chain Optimization. Other than myself, I know of only one other person that actively uses it for the task. Is there some hidden community that I'm not aware of?
I use Clojure for a lot of analysis, modeling, protoyping and optimization for supply chains (I work in Amazon's Supply Chain). The presentation mentioned that Clojure was used for SCO, and I was curious jf there was some community of supply chain users of Clojure somewhere because AFAIK I'm one of two people that use it for SCO.
We use it at my company, Weft -- we're building Waze for Cargo. Shoot me an email (marc@weft.io) and we'll try to get a Clojure for SCO group going! There are at least a few of us hacking away on that kind of stuff.
Related to this is the Erann Gat "How I lost my faith in Lisp" story[1]. He was also a googler but going the other way. Of course, this is quite old (before Clojure). But the points remains...
If this were generally true (of any technology X) the dollars would be flowing that way in an unstoppable torrent, and anyone using anything else would be the subject of ridicule and/or pity. It would be like writing custom websites in C.
Since that isn't happening, I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people. So yeah, citation(s) please.