"The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]
This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.
Yes, this is actually one of the rare moments where I find my past background in competitive programming helpful. There's a habit I've picked up of constantly running through techniques to speed up whenever I'm coding. Not necessarily the most efficient way to code, but it's a habit difficult to change.
Which data structures did you use? In Python, I tend to rely on dict, list, and set for 90% or more of my code. I wouldn't want to rely on structures written in pure Python.
Nothing exotic. One of the changes for example was replacing a list of list with a set of tuples, which greatly sped up checking if an object was in the collection. Another change was using a generator comprehension and an included itertool function rather than hand rolled nested for loops.
Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops.
We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.
Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is
"Could this be solved with matrix multiplications and summing?"
Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy.
I recently swapped out something that was running at about 100 rows calculated/second to about half a million in about 0.2s.
.keys() returns a list (in python2) so if you write
for k in dict.keys():
...
then python first builds a list of all the keys, loops through them and then throws away the list. If the dict is large, this can be quite expensive. The correct way is to either use .iterkeys() which returns an iterator which generates the keys one at a time, or simply iterate directly over the dict, saving you need to first copy all the keys into a list you'll just throw away.
This has been 'fixed' in python3 and .keys() now returns an iterable view of the keys, and if you actually want a list of the keys you have to explicit and write list(dict.keys())
Just to clarify your python 2 note: In python 3 keys() returns an iterator, so there is no penalty (i.e. iterkeys was dropped, keys assumed iterkeys interface).
The equivalent python2 behaviour can be obtained using list(somedict.keys())
Let me just add Numba to the list. It compiles numpy-using code to native via llvm and can remove temporary arrays in the process.
I tried a very simple toy program the other day and while I had to write some things slightly un-pythonically (it can't deal with syntax like a[:] = b+c yet), it performed practically as good as hand-written C code.
If your code falls within the subset it supports :) I've not yet got it to run our code in nopython mode - I think the latest problem is a function expecting a function as an argument, but the error messages aren't helpful or enlightening, even with DEBUG turned on.
All the points the author makes through the post are interesting, and Python is definitely great for protoyping, but I think the initial premise is false:
"people don't tend to think of [Python] as a high performance language; for that you would want a compiled language -- ideally C or C++ but Java would do."
Java is compiled to bytecode, but it isn't a "compiled language" since that bytecode has to be interpreted by the JVM.
All the good libraries the author mention are probably implemented in C or FORTRAN, and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code.
"...and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code."
True. But this assumes that time is not a constraint.
I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X hours, you stop both and see how far they've gotten.
I think the argument then is that you might find that the C programmer hasn't yet solved the problem, whereas the Python programmer might have a solution + a deeper understanding of where the slow parts are, and started optimizing those.
So in the end the C programmer may win, but perhaps it's not about winning in this sense but more about how fast you can get to a point where you can move on to the next thing.
I've tried to do a test like this and the results surprised me. While the testing conditions weren't perfect, I tracked how much time it took to port Fortran code to C++, and how much time it took to write the C++/CUDA optimized version. I would have expected writing all the optimized CUDA kernels would add significant time to the project, but in reality it was something like 1.3x for a solution 15x faster.
I think as developers we often underestimate the amount of work we need to put into the non coding part of development for larger projects, like testing, debugging, optimization, and design. When these components become important then despite the local productivity gains of using rapid development languages like python or Julia, it would overall be a wash if you're writing the system from scratch in a language like C.
Absolutely. For this kind of one-shot scientific computing, the only way the C programmer can win is if the Python programmer is sitting on their hands for weeks waiting for their program to run.
Unless (1) the relevant library happens to be written in C and there are no python bindings and the problem is simple OR (2) there is an existing solution which is 95% complete in C and one needs to write from scratch in python. I've never come across situation (1) with a new project. Situation (2) is quite common.
Java(well, Sun's Hotspot) is most definitely compiled, but mostly by the JIT at runtime. The interpreter only runs for the the first warmup, usually 10,000 method calls and the JIT kicks in and translates the method to machine code. The tricky part is it's not simple to control exactly how and when it does its job, although it usually works very well.
The major implementations of the JVM are jited, not interpreted. Yes, you get great performance with jitting, tho not necessarily without unpredictable GC pauses and without a loss of energy efficiency from having to run a jit alongside your actual code.
GC pauses have little to nothing to do JIT process itself. You can get JVMs w/ read barriers and soft-realtime guarantees.[0][1]
Please note, gc and jit are quite a different thing, even if the gc needs help from the jit.
I agree with his conclusion that Python has great tools to help performance, but the GIL makes it impossible to effectively make use of separate cores, so I have to wonder just how badly the Java version was written. Properly written high-performance Java (store all data in contiguous arrays, do away with all the OO crap, reuse objects like crazy) should perform as fast, or faster than optimized Python, and you can scale it to as many threads as you like.
Python has MPI4PY and Multiprocessing. Not sure why you need threads in the first place. Also, many matrix operations autoscale to all cores when compiled against Atlas.
But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of converting to/from numpy formats.
(And while I personally love Scala, there's nothing magic about it in this regard. There's no reason a language can't offer Python-like expressiveness and Java-like performance, and many modern languages do)
Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI.
I consider it an accident of history that Numpy, Scipy, Pandas, Scikits got written for Python and not Lua. Thanks to Luajit I think Lua would have been a better choice. Now that cause has been taken up by Torch and Julia.
JVM by itself is terrible for reaching close to the FLOPS that the CPU is capable of. Try sparse matrix multiply with it and see it for yourself.
What make sparse matrices harder to JIT is proving that the loop bounds will not be exceeded, so all accesses re bound checked. In any case in my experience array bounds checking and escape analysis never gave the boost that theory and JVM fans promise. So even normal matrix multiply will trail behind. That said Hotspot JVM is possibly one of the most optimized VMs we have got.
A structural problem of JVM is that its runtime semantics is over-specified, there is very little room for the JIT to
do its stuff. For example function arguments are evaluated right to left, there goes an opportunity for parallelism.
For normal dense matrix laid out as a double[] and accessed directly as i* N_ROW + j probably won't get its loop check elided. For double[][] I would _think_ that happens more easily.
But how are sparse matrices then generally laid out? A naive approach would be some hash map, perhaps with some locality, in which I don't see JIT problems.
There are some defacto standard formats such as CRS, CSC, list of tuples etc. Layout of the third should be obvious and it is not used much for cases where speed matters because one loses locality in this layout. For the other two they are laid out column after column (or row by row), row ids, and offsets to indicate the start and end of columns (rows).
Thanks, this helped. Using CSC and CRS is probably quite problematic with JVM bounds check elimination (or the lack of it). So if they would need to be used on JVM, I think it would be wise to drop the safety and use _sun.misc.Unsafe_ for unchecked array access.
Correct me if I am wrong but Spark does not use JNI for Python. It uses py4j to allow Python programs to access Java Objects. All the core functionality is implemented in Scala/Java. py4j uses sockets to communicate. See https://www.py4j.org/about.html
That's my understanding as well. This is the reason why you are caught between the rock and a hard place on the Hadoop stack. Java code wont be efficient in terms of FLOPs and a popular and often effective escape hatch: code number crunching parts in C, C++ or Fortran, is also not very effective / convenient. Particularly so if it requires going back and forth over the bridge frequently because crossing the bridge has significant overhead. So typically you have to move majority of the core into the high performant language of choice and what remains is glue. If gluing is what I want to do, there are other languages that can give Java stiff competition. The core capability thing going for Java in this domain is HDFS, and its not that great a file system for big data.
When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+ you lose by implementing numeric code in pure Python. So Scala gives you a 10x faster cycle during early development. Maybe it's fast enough that you don't need that last factor of 2, or maybe just JNIing a few core operations is enough (you can also do some tricks with the advanced type system; with NumPy if it looks like Python iteration then it is, whereas with something like Breeze you can potentially have something that looks like Scala iteration but won't actually copy the data back and forth). But in the cases where you do need to push things right down into Fortran, Scala is just as good a glue language as Python is.
> When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+
Not so in my experience. Write a matrix multiply in pure Java, and pure C, C++, Fortran. The difference would easily be north of 5X, typically more.
Regarding JNI, please see the root of the comment tree. Rarely have I seen more stinky garbage. Its hopeless if you have to go back and forth across the bridge. If that be so I might as well be on the other side of the bridge.
JVM is fantastic, but if you are doing number crunching and performance matters, then JVM is a wrong choice.
It seems you misread the article as saying Python created a trade-off. Perhaps the author shouldn't have tried to create a surprise ending. Regardless, the conclusion was that the Python implementation was both easier to code and more performant than the Java implementation.
From the article: "Easy development comes at a cost of course. That initial experimental implementation was terribly slow, taking thirty seconds or more to cluster only a few thousand points." Even if the final implementation in mostly-Python is fast, having a slow feedback cycle during early development can be a huge cost.
Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination.
In contrast, if you work in Java, you are trying to use the same tool for both jobs, and you may well fall between 2 stools. And I say that as a typical Java-head.
My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?
As for the number crunching, I think Rust would be a better choice here. Good memory management is its USP and that can have significant performance benefits.
> My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?
Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fast and break things" mentality. The Node ecosystem is built on that mentality, and while outlier developers at the front line of the JS community are able to be extraordinarily productive in JS/Node, developers building user-facing programs just get bogged down in breaking changes in dependencies, buggy and poorly-designed 0.x libraries, untraceable framework code that prioritizes configuration over convention, and intractable errors caused by a broken type system (this final issue is significantly improved in ES6).
Source: Worked in Python 2.x for a few years, worked in Node for a few years, currently work in Python 3.x and Node (general design is to limit Node to compiling and testing the browser-based part of our product). JS/Node is maybe 20% of our code, 20% of our features, 50% of our dev time, and 80% of our bugs.
Agreed. It is often true that different parts of a problem are best solved in different languages. That's one reason we built Beaker and released it as open source: http://BeakerNotebook.com. Its polyglot architecture and autotranslation makes working with multiple languages easy.
This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works.
Except, when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible.
So, use Python, but use it only on Linux.
(Okay, if you absolutely must do it in Windows: Use Anaconda).
FUD. I've been using Python on windows for years and between Anaconda and Christoph Gohlke's python packages and I've yet to run into something that didn't just work.
Yeah with Anaconda it's not as bad. But no-one told me that's what I needed if I wanted all the science packages to work on Windows... I'd never even heard of Anaconda before this. It took all of my blood and tears for weeks before I got everything fixed. So I guess what's wrong here is the lack of documentation.
Granted, it does not say "and by the way, when you do that you get tons of other great things like pandas, matplotlib, etc". But they are very clear about not trying to do this yourself.
Interestingly though, Windows version of Numpy on Anaconda is quite a lot faster than what you'll get with Ubuntu, like 2x faster easily. I think it has to do with the fact that AVX instructions are used on the windows linalg routines that it binds to. If you buy MKL of course you get much more speed again.
This is a problem for many languages that have external libs requiring local compilation (C, C++, etc.), given many of them are principally developed on Linux, with Windows compatibility being an afterthought. Its a problem not unique to Python.
> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast.
What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io, it's felt a step backwards.
I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree and KCachegrind. I've found the cProfile output confusing when it crosses the Python-C barrier for numpy, sklearn etc.
After using the profiler to look at the right piece of code, an important part of the process when optimising with Cython is:
cython -a module.pyx && open module.html
the -a command produces an annotated html file, which lets you see how Cython is compiling your Python into C. There's a colour coding to give you an indication of which bits are fully typed and being translated directly, and which parts are hitting the CPython API at run-time (and hence will be slow).
Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run into issues is when implementing algorithms in which case I usually isolate the slow function and turn it into a Cython module.
Recently have been playing around with https://github.com/jboy/nim-pymod which seems like a much better solution.
> cProfile takes a while to learn how to use well.
I bet :) I have only scratched the surface, that's for sure.
> What didn't you like about line_profiler?
In this case I was trying to profile the overall simulation codebase to find the slow spots (rather than guess) - a simulation that takes 60 minutes to run. line_profiler wasn't great at giving digestible results from that - and I haven't worked out how to write all output (across multiple modules) to file, without specifying the file in each decorator.
I've started to break the codebase down into 'tests' to measure each algorithm separately, and will give line_profiler another go then.
Re memory_profiler, while the mprof command showed all peaks, the line by line output only showed the result after executing each line - so when 4GB of RAM disappeared in a skimage call, only to be released at the end - it wasn't reflected in the output. Which is tricy when trying to reduce overall memory usage.
Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere.
From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable type system would be a huge improvement.
Because the Python ecosystem is huge, with real scientists writing real libraries to get stuff done.
The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).
C'mon zippers are not that hard, and really useful.
Lets say you want to do processing of some xml file.
Normally you'd walk the tree and do manipulations in place.
With zippers however, you can inspect every intermediary tree result, you can rethink you problem so that you walk the tree once to extract interesting information, then compute a changeset for the tree, maybe merge it with a differently computed changeset, and then apply the union of them.
I build a ocr system in clojure on zippers and it was a lot of fun. You could for example extract a list of all the words, with line wraps removed, then do the correction on that view/lense of your data, and reapply the changes without having to worry to reintroduce the pesky linewraps, because they were never removed from the original document.
> The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).
There are "Real World" Haskell resources, in fact there is even a book named Real World Haskell[0], though I recommend Haskell Book[1] these days since it takes an approach of teaching from first principles.
Haskell tends to attract people who like programming languages as programming languages. It encourages a style that's completely different from what you're taught in your basic Fortran/C/Java/Matlab course. If, like a lot of scientist, you've done a little imperative programming already, Python isn't much of a step; Haskell, at least as it is presented usually, is another bag entirely.
This means you have a much larger audience, and therefore it's more likely that someone will already have written a tool of your domain in Python.
The diff between Haskell and Python is not functional programming, I program functionally in Python, it is the type system.
This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just executing subexpressions and seeing results and the shape of the tensors interactively is more useful than thinking really hard how the operations transform the shape.
If you're using persistent datastructures like pyrsistent I could agree, but vanilla python is very unsuited for functional programming. Even JS has better lambdas.
> This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just executing subexpressions and seeing results and the shape of the tensors interactively is more useful than thinking really hard how the operations transform the shape.
If it is true that the type system, perhaps static type systems in general, make you think hard about things that ultimately don't matter much... why not go for the most dynamic and interactive language possible? Clojure perhaps.
I've been toying with the idea that either fully dynamic and as interactive as possible or full dependent typing is the generally right answer.
By right, I mean what you should reach for with most programming tasks if you aren't going to put effort into selecting one tailored to your task.
The only functional langauges to gain much popularity here are OCaml and F#.
Haskell tends not to be a go-to choice here because it... well... there are a lot of reasons. It's a tough language to learn compared to its competitors because it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them.
> it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them.
Err... can you qualify this? Based on my knowledge of Haskell this isn't a fair assessment at all.
Based on my knowledge, it's very fair. How many language extensions are available vs placed in common use? Where is the guidance on which to use? You WILL use at least 3 I can think of off hand very commonly.
And how many failed lens libraries preceeded the current (quite good) dominant library that still show up on google and even get included in? How many people still run versions of Yesod using now-unfavored abstractions?
The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs. But these developments do not magically erase people who have navigated the prior 5 years of Haskell where a lot of these technologies got established.
I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.
Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging." You will know these people when you find them. Compared to the Clojure community or the F# community or the Erlang community which are also forward thinking, interesting and skilled... it looks positively hostile. You may not care about that, but I refuse to endorse or use technologies who's community leadership is so toxic. Why, I've stopped using Linux in anything but legacy apps because I completely despair of that project from ever rising above is puerile and cliquish roots.
Alright, I'll give you that there are a ton of GHC extensions after getting a list of them, all 106 as of GHC 7.10[0].
This does make me wonder why having so many extensions was never an issue for me.
> The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs
Agreed.
> I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.
Fair enough.
> Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging."
I've seen this personality type in other languages, but not Haskell yet. I too abhor the "negging" type, as well as the subculture that term arose from.
The large number of high quality scientific and computational libraries that are easily available for Python (most things are a "conda install" or "pip install" away) plus the Jupyter (literate programming) notebook (although that now supports other languages) currently make a very compelling case for using Python instead of Haskell and most other functional languages.
Probably because functional languages don't actually at excel scientific computing relative to Python, C#, Java, and JavaScript. :D
Edit: I kid, but most of the languages I just mentioned have very fast native compilers, easy ways of invoking low-level interop, functional-style libraries if you want them, and (except for Python) C-like syntax making it easy to cut-and-paste.
No need to kid. Self-described "FP" languages are a pain in the ass for the type of programming scientists do. I don't even know what kind of programming those languages aren't a pain in the ass for.
"I don't even know what kind of programming those languages aren't a pain in the ass for."
If we limit the discussion to ML languages, they are generally very nice e.g. for datastructure transforms (including interpreters and compilers). Static typing with type inference makes implementations short, and, if written in good style, are correct when they compile.
There are various other reasons why one would not want to use them. For me it's platform stability and availability - while I enjoy functional languages in principle - I yet seem to choose C++ more often than not for my hobby projects.
I would say learning self-described fp-languages have made me a better programmer who values static typing more than before.
"I would say learning self-described fp-languages have made me a better programmer who values static typing more than before."
FP paradigm has indeed didactic value and the experience gained in a functional language may translate into a plus of productivity ...in other languages. When you have to get things done, the functional language's embedded arm-twisting (well, mind-twisting actually) doesn't help! That's when you'll either abandon the (idea of) "pure" functional language for a multi-paradigm language (that will allow you to keep a mainly functional style with limited deviations when the situation warrants it) or you'll keep pushing yourself and throw the blame on you for not being good enough with a language that wasn't designed from very beginning for much anything but the FP standpoint (for the sake of it).
I could make the same assertion about "self-described OOP languages" or "self-described procedural languages" with just as much evidence. Once you've done it enough, all programming feels like a pain in the ass.
I see FP languages being able to excel at the kind of projects with large, maintainable codebases that evolve over a decade and are shared between dozens of developers. Service oriented architectures serve the same goal -- isolating behavior and state locally, and being explicit about mutation and state changes over time.
Using Scala, I've found the ability to pass around first class functions, and the default immutability of variables, enables me to make profound algorithmic optimizations I wouldn't undertake in C++ et al. Then, if the computation generates too much garbage or goes slowly in a loop, I can then take shortcuts, like using mutable local variables, to make that section go faster.
Actually, if you squint a lot, R (or S) is a functional language. And it shows, crazily powerful for something, absolutely horrible for something else.
"R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functions, and even return them as the result of a function."
Yeah, lots of squinting needed, or a terribly shallow understanding of FP to say that. Merely providing first-class functions and map/reduce idioms does not a functional language make. Python has those, so is it also FP really? R is as much a FP language as Haskell or OCaml are imperative "at heart."
Sorry to STRONGLY disagree with you 100% but R clearly is a functional language and it might just be that you are unfamiliar with R or have not spent time with R.
R describes itself as a Functional programming Language:
The language syntax has a superficial similarity with C, but the semantics are of the FPL (functional programming language) variety with stronger affinities with Lisp and APL. In particular, it allows “computing on the language”, which in turn makes it possible to write functions that take expressions as input, something that is often useful for statistical modeling and graphics.
https://cran.r-project.org/doc/manuals/r-release/R-lang.html
Academic Evidence: "The Journal Trends in Functional Programming" also list R as a functional programming language.
R is an environment and functional programming language for statistical data analysis and visualization. Largely unknown to the functional programming community, it is popular and influential in many empirical sciences.
The issue might be it isn't Haskel so it is not 100% functional but there are more Functional Programming Languages then Haskel.
PS This is why so many programs in R are poorly written because people try to force OO into R when they could do things Functional. That is also why my eyes bleed when I see people looping through everything in their R scripts.
Like many modern languages, R supports multiple programming paradigms, including sequential, object-oriented, and functional, but it should take more than a couple of functional features to declare something a full-blown FPL. A duck-billed platypus is not a duck.
Yes, you can write R code in a functional style in places when convenient, but one floor down it's still running that loop you didn't want to see. It's a language based on mutable values and sequential execution and looping, not pure function calls and recursion. The syntax itself is very much sequential, unlike stereotypical FPLs. Even the lexical scoping design relies fundamentally on mutable environments. So yes, people do seem to enjoy proudly declaring R a FPL citing a few features, but as I said, those features make R as much a FPL as Python.
Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures, which is what you need to efficiently implement many numerical algorithms. But I'm a fan of Haskell's [REPA](https://hackage.haskell.org/package/repa) which provides multidimensional arrays with complete control over which operations are computed immediately and which are deferred and can be combined later with other operations. This allows many computations to be expressed in a high-level array-oriented way without incurring the cost of creating huge temporaries. C++ template libraries like Eigen can do this, but give much less control to the user on what is deferred.
Despite the Haskell program using the elegant facilities provided by REPA, the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.
I agree that the Julia looks nicer, despite the fact I usually dislike so much nesting.
> the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.
It has to be written differently more because it's using the state monad. You can write monadic actions that resemble pure code, for instance:
> Why isn't Haskell, or any other functional language, popular for this sort of thing?
Because some of the scientists doing this are bellow average programmers. Look at the author singing praises to Python while benefiting from C, Fortran and Cython. He doesn't understand that what actually matters is what's under the hood, and you want him to learn functional programming and rewrite his algorithms?
That seems very unfair. I have no reason to think the author doesn't understand that Python libraries are built on carefully written C and Fortran, and based on the article they seem to be a perfectly competent programmer.
Have you missed the title? He gives undue credit to one of the most unfit languages for HPC ever. Python is not made to be efficiently implemented, yet the author cheerfully writes "I choose to use Python for performance critical scientific computing code".
That's the unfair part right there: lying to people about the proper uses of Python.
I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking from a company perspective, the drawback python has is, strangely, the lack of paid for alternatives. It's not that people in companies don't trust open source (hadoop is becoming big here too), but one wonders if the developers will be able to find the support they need in case any issue arise from a free library.
I have first-hand experience with a BI-ish system, squarely targeted at the enterprise and doing quite well there, that we wrote using Django and a whole list of open source components.
We did run into some resistance initially, because our stack is almost the opposite in every way of what our enterprise colleagues are used to. However, our development velocity, especially around analytical features and just in general, has made a most gratifying impact.
(I have to add that the 5-man dev team we have working on this is stellar. It's hard to determine scientifically what the interaction is between team quality and the choice of a Python-oriented software stack. See Paul Graham's essays for more discussion on that point.)
In terms of support: There are many highly professional often boutique software agencies that can support Django systems, if you're not around. To my mind this is even better than the normal commercial support you get from a different vendor for each different component in your commercial enterprise system.
It would be interesting to hear about your experience, did you do a write up somewhere or could I email you with few questions?
My project would be to put different data sources together + to allow users to upload their own structured data via Excel (think financial estimates). The current system has about 450 users, the next might have much more depending if it gets extended to other divisions.
one wonders if the developers will be able to find the support they need in case any issue arise from a free library.
Continuum Analytics is founded by the creator of Numpy and employs many leading python developers, and they offer support contracts for basically the entire Python/Numpy data analysis stack.
I would like to second the suggestion for F#. It has a nice balance in that it straddles the line between being an OSS ecosystem and an enterprise ecosystem with paid libraries. FsLab and Deedle got mentioned already but the F# website has a lot of great resources too: http://fsharp.org/guides/data-science/index.html
A very beginner Java programmer here. It's a nicely organized notebook, great demo, but: seems like a lot of effort was put into optimizing the python efforts, and none for Java. Isn't that an unfair comparison?
My real question is, is it so much easier to do this excercise in Python than Java, assuming equal proficiency in either case?
Java has no operator overloading. Many Java developers vehemently oppose addition of operator overloading into the language, as if it were the root of all evil. The lack of that feature results in convoluted function calls where a clear and concise math expression would suffice. Consequently, not many people choose Java for math tasks, and not many people write math libraries for Java.
> it became necessary to see how our python version compared to the high performance reference implementation in Java.
It sounds like the Java version was also optimized - though it's hard to say to what degree. I think in general if you're doing a lot of numerical computing and matrix operations, the basic java builtins are not going to cut it anyway, and you're going to end up using something like ND4j or JBLAS to get comparable performance to something like numpy, in which case I have hard time imagining it could possibly be /easier/ than numpy.
Granted, the cython stuff is more "advanced", and you probably would struggle a bit (or just would not care for the task) if you were a researcher with a surface understanding of what's really going on, but it's no big deal for a programmer. The other thing is that the defaults usually work well enough that you don't really have to pull out the big guns most of the time. You can also decide to do your optimization progressively. "Let me rewrite this part in cython" is a much quicker win than "let me rewrite all of this to run on top of platform X" or something.
Also it definitely does depend on the kind of problem. For example, the JVM does have awesome tools for distributed computing and streams like http://akka.io/, vs python is more lacking.
In conclusion though, I've been always very pleasantly surprised by how far I can get away with by just dumping larger and larger datasets (current record is in the hundreds of GB for me) into the python / pandas / scipy / numpy stack, and how much of a pleasure it is to use compared to anything else. To toss that away, I'd want to have tried and failed at a problem with the python stack first.
Agreed. Python is an excellent tool in that respect. Batteries included helps. Being able to access fast C routines help. Compile to C projects like Numba and Cython also help. And of course, ipython (Jupyter) notebooks for exploration.
The interesting insight from the article is that python might be a good language for learning algorithms. The fast development time allows you to write a complete program (albeit clunky) without the pre-optimizing you might be tempted to do in other languages.
My current setup for "scientific computing" is RStudio and CSV files for quickly running typical stats-tests (a couple of t-tests and a tost + krippendorff's alpha the last couple of month) and python+libraries for anything that resembles "building stuff" (mostly scikit-learn to build some classifiers).
I mostly use R "as a consumer" i.e. I basically use RStudio whenever my colleagues fire up SPSS.
That combination works fairly well. I'd recommend it to anyone who enters academia in any field that involves statistics who doesn't want to use the typical proprietary tools (I've also tried PSPP and it works ok for basic tasks but lacks a lot of functionality. If all you want to do is run a quick t-test or ANOVA it's a decent tool).
The "notebook" he refers to is the web page itself. It's a format that's becoming popular among Python / data science folks: a web-based, often interactive layout with code and results embedded in it. See "Jupyter Notebook", the viewer for which is what you're viewing as you read this.
If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing".
The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!".
I agree some of the tooling and niceties are beyond a doubt best in breed with Python, but it's disingenuous to equate this to "writing HPC code in Python". If you had written a RPython to Verilog translator that produced an FPGA of your algorithm would you call that "using Python"?
> If Numpy, Pandas, etc. were wrappable from JavaScript
But they aren't, of course.
For science, the fact that so many great libraries are available and they all use Numpy arrays as base datatype (from image processing to GIS to machine learning etc etc) is a real strength of Python and I doubt another lanuage is going to achieve that any time soon.
Python in science is more than mere wrapping: it gives you new level(s) of abstractions, unrivalled in any other generic programming language AFAIK. If it were mere wrapping, other languages would have been able to catch up fairly quickly after all.
Big parts of numpy (and to a lesser degree pandas) are not in python, but lots of libraries on top of it are mostly python (scikit learn, scikits image, etc.). It is no more wrapping than lisp is wrapping the C VM.
Claiming that using wrapped libraries written in another programming language (LAPACK, anyone?) is an inauthentic usage of the language (here, Python) is pedantic and unhelpful. C is just a wrapper for assembly, then, right?
Sometimes it's not - sometimes LAPACK is written in C. Sometimes your C is just a wrapper around FORTRAN and people should take this fact seriously: modern FORTRAN is often a better option than C for numerical workloads.
His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.
By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive?
I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.
Well, the word "bizarre" has a commonly understood meaning, but you somehow decided to use it to mean something completely different. That's a bit bizarre :P
But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC.
In reality, the reason why he "still" uses Python is that the libraries are fast enough for HPSC. But that's not what people see when reading posts like this.
The message they see is that "Python is fast", not that some of its 3rd-party libraries are fast.
The question is: what language should I use for HPC.
An answer is Python.
If you use C for HPC (the way you mean it) it will be slow as a dog. Because HPC is done with BLAS, LAPACK, etc. If you do HPC in C you are calling into BLAS and LAPACK. They are Fortran libraries. Anything else is dog slow in comparison (a slight stretch, but not by much).
So, the "real" answer is Fortran. But not a lot of people are sitting at their desk writing Fortran for scientific computing these days. They are writing Python, C, or C++ for the most part. All of which use libraries that call into the Fortran libraries.
That's all this article is saying, and we all understand it. No one is wringing their hands about what language the libraries are written in unless you are trying to write one of those libraries.
Working professionals use Python for scientific software because it excels at data munging, making reports, and very high speed computation. Everyone gets that you need to add some libraries beyond the libraries provided from the base install.
First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter.
Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easier and safer than doing it by hand (e.g., generating correct reference counting code). All of this is a direct benefit of Python and its scientific ecosystem.
> First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter.
Sure, but that's how people construe the post, which I think the author knows too.
The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC", but that wouldn't have generated nearly as much interest as "Why I still use Python for HPSC".
People want to see others say good things about their favourite language, which in this case is Python.
I knew where the article was going (with the libraries), but still wanted to read it because even I wanted to see someone compliment Python itself, because it used to be my "primary language".
Python is a good language, but to the extent you're not using those highly optimized "C-libraries", it just doesn't perform well, and it's not suited for concurrency. That's a part of why I switched to Clojure.
Just because people construe the post that way doesn't mean they're right. Truth is not democratic.
You can make any title more accurate by making it longer and more specific, but picking a good title is a trade off against other things as well. A charitable reader would not necessarily insist on the narrow interpretation of Python the language you insist on.
And your title is not more accurate, because it's not just the existence of certain libraries, it's also the way they can be easily glued together and incrementally improved. Compare this to say Java, where you'd have to use JNI to use those same libraries, which is much more cumbersome to the point that most people prefer to write everything in Java.
> People want to see others say good things about their favourite language
To be honest I only think this only describes people that enjoy internet debates on programming languages. If what you are interested in is getting work done, you don't care about the merits of a particular tool in isolation but about end results. Hypothetically, you could find the "perfect" language but if it doesn't have users or libraries, it's of no use, and therefore the insistence of discussing only this or that language is pointless.
To be fair, the article doesn't say "Python is fast." It says "Python is slow," and then goes on to say something like "But if most of your heavy-duty calculation involves calling prewritten C routines, you can do your development work in Python without getting killed on performance."
Language discussions always seem to go down the same dismal tube in the end: "Is C > Python, or is Python > C?"
The point I think a lot of people are missing is that we've always used optimized libraries no matter what language we've used. C devs would use hand optimized assembly when speed was critical. There's no perfect language that just does everything.
I'm currently working on bringing BLAS [1] and LAPACK [2] to JavaScript. And here is a matrix/vector library with some of the BLAS functions implemented [3].
Numpy can basically be considered to be part of the standard library for Python so I'd say using it still qualifies as "using Python". Pandas is newer but is graduating to that level of ubiquity too.
If you want to import, export, tabulate, edit, plot and analyze CSV data in browser-side Javascript, here's something I wrote to paste numericJS, jqPlot, lzstring, and various html5 storage and file handling features together.
"The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]