Hacker News new | past | comments | ask | show | jobs | submit login

You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice.

Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.




MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc.

Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.


> in favour of Hive, Spark

Well, Hive and Spark create MapReduce jobs to satisfy your queries: it's still in the background, but you don't have to think about it (as much).


Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.


Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago.

You've seen people writing pages and pages of Java code to do ETL ?


I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions.

The rest of the world does not particularly care about SV fashion trends.


I'm very curious to know where you got this strange notion that MapReduce is "a technology/paradigm that stopped being used a decade ago". I barely knew what MapReduce was in 2010, and didn't touch my first Hadoop cluster until late 2012. Every place I have worked since has had a Hadoop cluster or two.


I'm on a team of a Fortune500 who uses a combination of Java and C# to do ETL. AMA.

Edit: We're also a fairly greenfield team, with a product in beta that's less than 4 years old. So like, the leaders of the team knowingly crafted us in this particular direction.


Back in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: