On: The end of big data

[ramraj07]

Yeah, we moved away from spark for a reason (you kinda have to babysit it or it’ll crash on anything actually big data). Snowflake took care of things much better than that. Their lake offering is like the worst of both worlds.

[disgruntledphd2 → ramraj07]

Spark allows you to distributed model training though, which is really nice when you need it.