On: The end of big data

[nus07]

That was a fantastic answer.Likewise learned a lot. As someone stuck doing old school RDBMS Sql and Python for APIs , what’s the best resource for me to learn about databricks and spark ?

[tacon → nus07]

This books has an excellent reputation for the foundations of data-intensive software:

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

https://www.amazon.com/Designing-Data-Intensive-Applications...

[alexott → tacon]

It could be a good second step, imho…

[alexott → nus07]

For short intro I would point to “Learning Spark, 2ed” - it was available for free from Databricks site (maybe still available, look via Google) - it gives intro into the Spark and Delta Lake. The “Data Analysis with Python and PySpark” is a bit deeper, but doesn’t cover Delta.