I work in the Analytics space and been mostly on Java and I am so glad that other people feel the same.
At this point, people have become afraid of suggesting something other than Spark.
I see something written in Rust to be much better at problems like this.
I love the JVM but it works well with transactional workloads and starts showing its age when its dealing with analytical loads.
The worst thing is then people start doing weak references and weird off the heap processing usually by a senior engineer but really defeats the purpose of the JVM
I guess your company is running on Java and running something else would cost a lot in training, recruiting, understanding, etc. But down the line, defeating the JVM will be understood only by the guy who did it... Then that guy will leave... Then the newcomers will rewrite the thing in Spark 'cos it will feel safer. Rinse-repeat...
(I'm totally speculating, but your story seems so true that it inspired me :-)
Some of it is what you mentioned about training and hiring costs that but mostly its this creation of the narrative that it will scale someday in the future. This is usually done by that engineer(s) and they are good at selling so a different opinion is ignored or frowned upon by the leadership.
I have now seen this anti pattern in multiple places now
Analytical loads deal with very large datasets in the order of terabytes even after you compress them. These workloads dont change much so keeping them in the heap eventually results in long processing pauses because JVM tries to recover this memory.
However, in most cases, this data is not meant to be garbage collected. For transactions, once you have persisted the data, it should be garbage collected so the pattern works.
There are a lot of other aspects that I can probably think of but the above one is the most important in my mind.
