The two things missing from Cascalog that would take it from great to godlike are 1) an easy way to use the distributed cache and 2) a way to run Cascalog jobs on the cluster without the compilation/hadoop jar cycle. I don't know if #2 is even possible but it would be ridiculously powerful.
Could you elaborate on #1 please? Wouldn't a distributed cache defeat the purpose of data locality of Hadoop? Regardless, I guess one could write a tap to Avout to enable this?
Sorry, just saw this reply. Hadoop comes with a distributed cache that is generally used for small files -- a common example would be doing a large join against a small table that would fit in memory. For example if you wanted to filter out stopwords or something, the currently accepted way is to put this stopword list into the resources/ directory of your JAR, which is not really optimal for data that might change frequently.
