APL, (which is the array language I had some exposure to), had three
interesting characteristics:
1. It implements an algebra: Operators take arrays as input and yield
arrays as output.
2. The special characters bound to those array operators.
3. It is functional.
#2 definitely gave APL programs a distinct and elegant personality,
but probably didn't help with widespread adoption.
But #1 and #3 are powerful ideas, and have showed up in other
forms. SETL was a set-oriented language from the late 70s/early 80s. I
took compiler courses from the guy behind it, R. B. K. Dewar (of
Spitbol fame). Fantastic teacher. SETL programs were concise and
powerful like APL programs, exactly because of #1 and #3.
Later in my grad school career, I got into relational algebra, which
also has these characteristics. My advisor was T. H. Merrett who
engaged on a many-years, quixotic quest to explore relational algebra
as the basis of a general-purpose programming language. I didn't buy
into all of that, but as a database guy, I find relational algebra to
be powerful and useful on a day-to-day basis.
Finally, I really like functional stream approaches as a great way to
bridge set-at-a-time programming (for lack of a better term) with
ordinary, low-level programming. Java 8 streams and lambdas is a great
(if flawed) example of this approach. Even Map/Reduce is, under the
vast amounts of complexity.
I like this approach so much that I built a command-line tool, osh
(http://github.com/geophile/osh) based on it. The idea there is the
Unix idea of composing simple commands using pipes, except in osh, the
pipes carry streams of Python objects. And the commands that operate
on the objects in these streams are Python. E.g., to print the pids
and commands of processes whose commandline contains "emacs":
I read your post, thought 'hmm sort of powershelly with a bit of awk', clicked on your Github project page and saw Powershell was listed as the first 'Software with Similar Goals to Osh'.
I've been searching for the 'panacea' of shells (and/or auxilary shell tools/hacks that dup() fd's 0,1,2 to enhance existing shells) that hits that same 'sweet spot' you describe. As such, over the last ~15 years I've been through a litany of setups, ranging from:
- the standard "bash/zsh/fish" approach (where you extend the shell) to
- the "scsh/ipython/eshell" approach (where you bring an inferior shell's functionality into a language) to,
- the screen/tmux approach (where you take a shell and then layer functionality over it). I.e., for directory navigation,
I'd written my own f-recency+bookmark system that would hook 'cd <tab>' and generate a pane sort of like Midnight Commander to nav around
I'm not sure where I'm going with this other than, I feel your pain and I'd imagine tons of other people do/did as well. Powershell is painfully slow and RAM heavy but the ability to add custom properties(!), providers, access the registry, and manipulate all of these objects as you'd like. Your project definitely looks like an interesting take on things as well. At least we're making some progress, I suppose ;)
===
(!) This is incredibly powerful since you can take a path, like C:\users\foo\downloads\video\, take file item, and then have Powershell invoke an executable to extend functionality out. If Windows doesn't have "Length" or "Encoder" as a property on the file out-of-the-box, you can just use an auxilary tool (say, ffprobe), "mapcar" the exec to the list-of-files, grep out the Length: field, and bam, that file now has Length. ``ls|where Length -gt 15'' ends up being pretty magical.
osh gets extensibility by reading a file on startup (~/.oshrc by default). That file is Python, and contains both configuration (e.g. database login info, ssh login info), and functions that can be used in osh commands.
"Dewar, a professor emeritus of computer science at New York University, believes that U.S. colleges are turning out programmers who are - there's no nice way to say this - essentially incompetent."
"A lot of it is, `Let's make this all more fun.' You know, `Math is not fun, let's reduce math requirements. Algorithms are not fun, let's get rid of them. Ewww - graphic libraries, they're fun. Let's have people mess with libraries. And [forget] all this business about `command line' - we'll have people use nice visual interfaces where they can point and click and do fancy graphic stuff and have fun."
I am both an APL user (k) and a spitbol user but still not a Python user. Regarding the later, I think I suffer from the Dewar perspective[1]: it's too easy and too focused on libraries. Perhaps this is misplaced, but for myself I feel k and spitbol are more "educational" than Python.
1. In truth, I have no idea if Dewar saw Python as he did Java and PHP.
That's questionable. APL doesn't treat functions as first-class values. Its higher-order functions are limited to "operators", i.e. second-order and no higher.
I don't recall SETL, and relational algebra doesn't have functions.
I meant functional in the sense that the 1st-class objects (arrays, sets, relations) are not mutable. (My memory of APL and SETL is fuzzy, so maybe there is some way to alter state, but that certainly isn't the emphasis.)
I had the good? fortune of programming in A+ (a derivative of APL) at Morgan Stanley in the mid 90s. Not sure if it is used there anymore. For someone coming from a background in object-oriented languages like C++ and Java (only just introduced in 1995,) A+ was utterly alien and incredibly powerful. A single line could accomplish stuff that would take 50-100 lines of C++. But that single line looked like Egyptian hieroglyphics! To even read or write code in A+ required a special font addition to XEmacs to work as the glyphs would not render in any other font. I never fully understood the art of programming in A+ in the year or so I had to work on the product that was written it it, but I was amazed at the things the experts in my team could achieve with it.
It is indeed still in use at Morgan Stanley in the Interest Rate Derivatives Technology group. Apparently there were plans to decommission the A+ code as far back as the early 2000s, but much of it is still in use and, from what I can tell, the users still have a fondness for it even in the face of replacement pieces done in Java/Scala/C#. Several A+ devs retired within the past few years, but they have been able to bring on new blood, which was surprising to me. I didn't do much A+ coding in that group, but it always seemed like a breath of fresh air working in A+/XEmacs where you could easily directly inspect the running program to either produce new results or investigate existing ones. It was certainly much nicer than working on the modern Scala project that would often take an hour+ just to rebuild if you happened to add some logging.
About 10 years ago, A+ was just used as the communications protocol for fixed income. Nothing (that I knew of) used it, but C++ ceremoniously created A+ objects, sent them over the wire to another C++ process that decoded them.
Around that time, we moved much of the mortgage models over to kdb and q, which was fun. Lots of weird dark little corners in that language. (List of dicts? Nope! It's a table now! 'sum x' drop NAs, while 0+/x was NA if any value was NA, but maybe only for ints, not floats, etc. Ugh.)
It is still there and open source [0] and I guess you can say k (not kdb or q!) are it's successors in some ways. Definitely in the way they are all very much APLs, with or without the glyphs. If you read the APL book by Iverson and then go to implementations of APL, A+, J, K, you'll notice differences, but there are all identical enough to pick them up fast once you are proficient at one of them. kdb+/q are different beasts and that is were the issues start (and the big bucks are to be found); debugging (performance) issues related to critical software written on top of k.
Do you have any experience with kdb+/q? I was considering taking a job offer where I'd have the opportunity to learn kdb+/q but I don't have any experience with it and I can't find a huge amount of information about it online.
Despite my griping, I liked kdb+ and q. As a database query language, q is awesome, if you are doing a lot of time-series analysis (running sums, etc., nothing fancy). As a language, the table type in q is very nice, a bit like what pandas or data.table wish they could be.
For a while there was a freely downloadable version to try out, and you can look at "q for mortals", http://code.kx.com/q4m3/, to get some flavor.
They just made the 64 bit version free for experimenting with. It is annoying because it only will start up and run if you are connected to the internet, but it fully works. The 32 bit version can be downloaded as well and that fully works, with or without internet connection.
The key thing is "Notation as a tool of thought" What would mathematics or music be without notation? It might be hard to believe but there was once a time mathematics didn't have notation for "+, - * /" or even the decimal point "." Mathematics was solved by writing words, it was slow, it was painful and it didn't progress as fast. Likewise with music, sure we can write music by writing do re mi fa so la ti, but really?
Yet, this is how we program today with words, writing english words that don't mean what they claim to be. SomethingManager, SomethingDelegator, etc. Bah! The very ideal behind APL is to notations, to represent code with symbols which are non ambiguous and always mean the same thing.
There's a lot that we have lost. It's still worth taking a look at. For those interested, J (www.jsoftware.com) is a free version with lots of great learning resources.
As somebody who started programming long after I learned to touchtype, I feel like programming uses a lot more typographical fauna than normal language - granted, not as much as maths - but I'm pretty sure half of the maths notation is as it is because it looks cool.
Further, when you have a SomethingManager, is it really representable by a general symbol? What symbol would that be?
With many languages adopting LINQ object querying, I find it more and more suitable to start using math's set operators for them. Many languages have almost identical behavioral sequence operators (.find in js, .Where in C#, etc) but math sort-of standardizes them all with a sound theoretical background. I'm not entirely sure if they all map 1-to-1, but at least most will.
const items = ∅; // empty set
if (myItem ∈ items) { ... } // is element of
const union = items ∪ otherSet;
assert items ≌ ∅; // items are all equal
The computer science formalisation of these operations is known as “relational algebra”, which is based on set algebra.
That said, the assertion that all elements in a set be equal doesn’t really make sense — or rather, it’d be written as “|S| < 2” by virtue of the fact that a set is defined as containing distinct elements; consequently, in a set with more than one element, all elements are not equal. If you want to use mathematical formalisms for set operations, you’d instead have to work on sets of tuples and test for a predicate: “∀x ∈ S: p(x)”.
What exactly do you mean by "items ≌ ∅"? The only thing I read in that is "is isomorphic to the empty set", which with all reasonable definitions of isomorphic is the same as that 'items' is empty, which isn't the same as "items are all equal".
Typing these characters is quite annoying though. The characters for most programming languages are readily visible on a 108 key keyboard, which means they’re easy to remember and teach.
It's depressing that this is such a common argument against moving away from the ASCII character set. We all use operating systems that allow us to define key sequences for these characters. And it's really easy to remember them after a week or so, I mean, how often do you actually look at the letters on your keyboard anyways?
The main issue is that computers don't come with a standard default layer for math and APL characters anymore so it's a huge leap of faith for someone to start to use these characters.
> Typing these characters is quite annoying though
This depends on the implementation. In Dyalog APL for example, the special characters (glyphs) are effected using fairly simple two-key sequences. Mnemonics are used to help learning. I found it fairly straightforward, surprisingly so.
This a great example of how 'naming things' is by far and away the hardest problem in computer science. If you're going to call something 'normal form' or 'manifesto' it's best if there's exactly one of them. By the third one, others start wondering how normal the first form could have been and whether it might not have been more prudent to try a communique or memorandum before going full manifesto.
While I understand your point, it is my understanding that the first two "manifestos" weren't by Date and Darwin. I think Cobb's original article introducing the very idea of the relational model is one of them, and I forget the other.
They are rather making an egotistical (but IMHO deserved) point that their work is the 3rd revolutionary call to action in the history of relational database systems.
Yes, it's not entirely without uses to have a third manifesto. There was, for instance, a NoSQL manifesto a few years ago. Relational purists could reasonably point out this is a clear sign of the immaturity of the new technology as they were on their third manifesto nearly two decades previously.
I found some of those purists very annoying. The real world is often messy and organic, and trying to force excessive order on it just makes for confusion and/or silly extra busy-work.
The arary languages have always struck me as the extreme end of this. The semantics are very dense: each operator does a lot. Intermediate results aren't granted names. The few people who can follow this find it pleasant as the "extraneous" information is removed. Most people find it hard to adapt to. And it doesn't readily suit a lot of business logic.
Possibly the one context where arrays-as-language-primitive has really taken off is the 3D processing world, first with OpenGL's matrix stack computation and later with shader programming. Note that shader programming is the other way round: you don't start with a screen object and apply functions to it mapped across all the pixels (fragments), you write a small program as if for one pixel. Even if the end result is executed in a highly parallel fashion.
I don't like that blog post on "semantic compression". He's rediscovered refactoring but doesn't think so:
> This is a very bottom-up programming methodology, a pseudo-variant of which has recently gained the monicker “refactoring”, even though that is a ridiculous term for a number of reasons that are not worth belaboring at the moment. I also think that the formal “refactoring” stuff missed the main point, but that’s also not worth belaboring. Point being, they are sort-of related, and hopefully you will understand the similarities and differences more over the course of this article series.
Um, what? Refactoring is called refactoring because that's what it is: You're pulling out a "factor" of the program and reifying it into a symbol which then replaces the factored code. "semantic compression" is a great name for it, but it's been the done thing in Forth, for example, for many decades.
It's just like factoring polynomials.
This is a textbook example of what I call "half-smart": He's smart enough to write this article, but not smart enough not to.
(Also that site disables reader mode. Boo! Why would you do that?)
I would argue that most programmers are fine with this. Unix pipes are a very similar point-free way to program that we're all comfortable with. If people came to APL thinking of it as something similar to pipes as opposed to something similar to C they might have an easier time with it.
It's funny to call this point free when OOP notation like ps().grep(pid).kill(9).or(halt); comes down to the same thing. Just rambling because I'm confused. lol.
You're right, method (OOP) notation lets you compose/pipeline functions. Pipes enable the same thing. So does F#'s pipeline (|>) operator, Haskell's composition (.) operator, Clojure's threading macro, concatenation in stack-based languages like Forth, etc... It's not a concept unique to OOP, which is why GP referred to 'point-free programming', which describes the general idea.
> Possibly the one context where arrays-as-language-primitive has really taken off is the 3D processing world, first with OpenGL's matrix stack computation and later with shader programming.
There are many more applications than 3D, most notably APL and all its descendants applications in analysis work, logistics, stats and so on.
I remember APL as part of learning how to program at my high school in the early 1970s.
I could touch type APL on the IBM 2741 text terminals (basically Selectric typewriters with serial interface.). And could compose simpler APL codes at that speed.
The principal weakness was APLs narrow view of data type. Text was a 3rd class citizen. There was no substring searching in the early IBM implementation. As a result one had to:
Form a cross product of text with the substring, testing for equality of each character
Rotate each row of the cross product by n steps where n is the row number
(This aligns the tests along columns)
Perform AND reduction along the columns to find the full matches.
None of this was intuitive and no I didn't invent that method.
Rotate each row of the cross product by n steps where n is the row number
q)(til n:count b) rotate' a=/:b
000000000000010000000000b
000000000000010000000000b
000000000000010000001000b
000000001000010000001000b
Perform AND reduction along the columns to find the full matches.
q)(&/) (til n:count b) rotate' a=/:b
000000000000010000000000b
And there we have it. A naive string search in an array language.
Now some interesting advantages fall out of rediscovering this:
- It's obviously parallelisable: The "obvious" (iterative) solution in other languages isn't, and without some difficulty it isn't clear where the work can be split up.
- It uses "outer product" with compare = instead of multiplication × -- an experienced programmer might forget how deeply satisfying it is when first learning how to compose operators
- With some thought it gives a programmer ideas on how they can do string search even faster. Fast string search in an iterative language doesn't look anything like the naive solution.
b⍷a (b find-in a) is useful enough that an experienced programmer should expect a "modern" language to implement it (in q it's called "ss", other languages of course call it different things) but it is far from necessary, and we might cheat ourselves of something. After all, when we are experienced enough to see these things as obvious, other things become obvious as well.
When I was in high school in the 70s I got to attend a week of programming classes at U Waterloo. I had never programmed before. My first programming language was APL. We sat for hours in front of IBM typewriters with strange symbols, trying to write simple programs. The noise from a roomfull of typewriters still rang in my ears at night.
I didn't learn much, sadly, but it sure piqued my interest in programming.
APL is a great language, but it shouldn't be the first one you meet.
F-Script was an interesting array language object combo that was originally written on OPENSTEP. It fit great with Cocoa later and was quite nice to program in.
F-Script was really interesting. It was basically array programming welded onto Smalltalk welded onto Cocoa. It had a whole Smalltalky object browser, and the array programming features felt surprisingly natural.
Thanks for posting this. Very interesting. Smalltalk actually had some array-language like elements (Alan Kay knew a good idea when he saw one), but this takes it another level.
They kind of got replaced by "table oriented" query languages, like SQL. Arrays can get very unruly to coordinate, manage, and/or comprehend. Tables tend to ensure a bit more structure and can "do" most of the same things as arrays if you normalize properly.
SQL is probably not the ideal "table processing" language, but it's good enough and a strong enough standard such that competitors have yet to unseat it (although I'd like to see more competition, such as Tutorial-D/Rel, SMEQL, etc. fight it out in the market-place.)
I have used Fortran 90+, Python with Numpy, R, and (rarely) Matlab/Octave, all of which allow operations on whole arrays and array sections. Would they be considered array languages? What is qualitatively different about APL, J, and K?
Consider the lone factorial problem: Every array programmer will do some kind of the product of, one plus array of all ints up to x
apl: ×/⍳X
q: prd 1+til x
j: */1+i.x
k: */1+!x
Notice how similar they are?
Now, how do you solve this problem in Fortran? In Python? In R? In Matlab? What's the first tool in your toolbox? Is it to iterate over the values?
FUNCTION FACT(N)
INTEGER N,I,FACT
FACT=1
DO 10 I=1,N
10 FACT=FACT*I
END
def fact(n):
result = 1
for i in range(1, n+1):
result *= i
return result
function b=fact(a)
b=1;
for i=1:a
b=b*i;
end
function fact = iter_fact(n)
fact = 1;
for i = 2:n
fact = fact * i;
endfor
endfunction
fact <- function(n) {
f = 1
for (i in 2:n) f <- f * i
f
}
If the native programmer's first impulse looks something like that, then it's not an array language because the programmer isn't thinking in terms of arrays.
I wonder if some of the popularity of "functional" in imperative languages is that it lets you put this sort of stuff together without needing an "array language". In python it's pretty easy to
(albeit with some imports, and note range is [begin, end) so that's 4!, not 5!). It might not be the first thing a Python programmer does, but it's certainly reasonable. Haskell of course is
Prelude> foldl (*) 1 [1..5]
120
(inclusive this time, hence 5!) which is only slightly more verbose than the "array" languages, but does have the advantage of clearly specifying what your base case is if the list is empty.
Perhaps array languages are just getting subsumed as a special case of "functional programming", be it either the relatively weakly-guaranteed FP that gets embedded into otherwise OO/imperative languages or Haskell's stronger-guarentees FP.
It's worth noting that the behavior is totally different if you pass higher dimensional matrices. The Fortran, Python, Haskell, etc. code will all break or fail to typecheck, whereas the APL/J/K implicitly maps over them. Understanding this topic ("rank") is a major source of confusion for new users and power for existing users, as lots of different mapping regimes can be obtained without a lot of additional operators.
The rank operator in J (and probably Dyalog APL) allows you to treat a 3D matrix as an array of 2D matrices, a 2D matrix of arrays, a single item or a bunch of unit-sized boxes. This concept generalizes to higher dimensions. I don't think this aspect of array programming has gotten as much airtime as it deserves, probably because it is complex, but this is where the semantics of array languages and conventional languages really differ.
It seems to me that mathematical higher dimensional matrices are, in the broad scheme of programming languages, a corner case. Certainly a big one and an important one, but not one that was ever likely to drive general purpose languages.
Indeed, I'd argue that if that was your primary use case you are and always were better off with a relatively custom language (relative to programming as a whole), because the needs are so different and the wins so big using an environment set up to support those needs that you'll want that in the end. You can fuzz the line with things like NumPy, because all these lines are fuzzy, but dedicated support will be a big win in the end.
I think it's kind of a chicken-and-egg question, personally. Compared to array languages, most other programming languages are, in a sense, tree-oriented. Does their prevalence mean that our problems are mostly tree-structured? Or is our propensity to see problems as tree-structured a result of brainwashing by our tools? "A prisoner falls in love with his chains" as Dijkstra said.
But I wouldn't call Julia an array programming language. Not everything is an array, there are scalars too, and vectors (vs matrices) are very special arrays that exhibit the expected duality properites (since 0.6)
It's kind of crazy to think of Matlab as not an array programming language since the array is literally the only data type.
Here is a Fortran 95 program to compute the factorial of 0 through 5. The factorial can be computed as the product of an array created on the fly, without an explicit loop.
program xfact
integer :: i
print*,fact([(i,i=0,5)])
contains
elemental function fact(n)
integer, intent(in) :: n
integer :: i,fact
fact = product([(i,i=1,n)])
end function fact
end program xfact
program xfact
integer::i
print *, fact ([(i, i = 0, 5)])
contains
elemental function fact (n)
integer, intent (in)::n
integer::i
fact = product ([(i, i = 1, n)])
end function fact
end program xfact
I like that definition! Not least because it means Futhark qualifies as an array language: i32.product (map (1+) (iota n)). I have also heard the more fundamentalist definition that if you have to say 'map', then it is not an array language.
R is definitely an array programming language, but with the accent on statistics (historically, though you can do a lot more with it than that).
Fortran and Python are much more general purpose languages that also happen to have array operations to speed things up.
Matlab/Octave are more math oriented than any of the above and are far more than just programming languages, they are more like interactive notebooks, the IDE for Matlab (and for Octave too now, though I haven't worked much with it) is so closely coupled with the language that the language has no stand-alone right to existence.
APL, J and K are all more closely related to each other than to any of the others, the closest of the others would be R.
Minor aside: when I worked at Scilab I couldn't bear having to interact with a GUI so I got used to working either in complete CLI mode or sometimes in the partial CLI mode if I wanted to use a specific GUI feature. I only needed it for interactive plots or interfaces (iirc), however if I had been using xcos I would have needed it. So I don't know about Matlab / octave but I don't think they're fundamentally linked to their GUI; at least not in the sense of the programming language.
APL came before all of those. J/K are decendants of APL. Those languages that you used are copying APL features. Some of it is not too bad, but they don't have the entire power.
In J for example, you could factor your code out. The way you factor mathematics. Their power lies in their notation.
Outside of the notation, If you have a multidimensional array, you can operate on ranks of it. You can write loopless code, you don't need for, do, while loops. The compounding and combining of verbs allows for a very functional style of programming that you can never realize in the languages you mentioned. You have to understand APL was designed for describing algorithms before it got turned into a computer language.
No. Is like claim "C" is a OO language. Is more correct to say it have "array capabilities".
To be considered an "X" language it must that "X" be the most primitive/idiomatic/natural/default way to think and program on that. And also a primitive of the language!
You can do OO on SQL, but that is hardly the idea!
Similar, with a array language you see everything as arrays and as array manipulation and rarely deviate from that. This is the same on other languages.
Even in multi-paradigm languages like python or C# maybe you put a little functional here or linq there but that portion of the code seriously look "alien"
I think I'll disagree mildly. In "slow" languages like MATLAB and Python/NumPy, often the most performant way to do computation is array primitives. So much so that the style becomes idiomatic over, say, for loops.
I think that it's intuitive for people working with large datasets/numerical data, and not otherwise. You can cheerfully write loops in Python for most problems, except the set of problems solved by Numpy.
Mind you, I cut my teeth on R, so needing to use loops in other languages confused the hell out of me at first.
J (i and i assume APL/K to an extent) is called an arry language because arrays are, essentially, a primitive data type. There is no special syntax for arrays, either for creation or handling. The verbs in J just work with arrays, and in most cases, work better with arrays than if you were to manually break it down.
1. It implements an algebra: Operators take arrays as input and yield arrays as output.
2. The special characters bound to those array operators.
3. It is functional.
#2 definitely gave APL programs a distinct and elegant personality, but probably didn't help with widespread adoption.
But #1 and #3 are powerful ideas, and have showed up in other forms. SETL was a set-oriented language from the late 70s/early 80s. I took compiler courses from the guy behind it, R. B. K. Dewar (of Spitbol fame). Fantastic teacher. SETL programs were concise and powerful like APL programs, exactly because of #1 and #3.
Later in my grad school career, I got into relational algebra, which also has these characteristics. My advisor was T. H. Merrett who engaged on a many-years, quixotic quest to explore relational algebra as the basis of a general-purpose programming language. I didn't buy into all of that, but as a database guy, I find relational algebra to be powerful and useful on a day-to-day basis.
Finally, I really like functional stream approaches as a great way to bridge set-at-a-time programming (for lack of a better term) with ordinary, low-level programming. Java 8 streams and lambdas is a great (if flawed) example of this approach. Even Map/Reduce is, under the vast amounts of complexity.
I like this approach so much that I built a command-line tool, osh (http://github.com/geophile/osh) based on it. The idea there is the Unix idea of composing simple commands using pipes, except in osh, the pipes carry streams of Python objects. And the commands that operate on the objects in these streams are Python. E.g., to print the pids and commands of processes whose commandline contains "emacs":