We need to re-think how to make data _useful_. The fact that the value hasn't materialized after decades of attempts, billions of dollars, and lots of tools and technology points to the fact that our core assumptions and patterns are wrong.
This post doesn't go far enough. It challenges the assumption that everyone's data is "big data" or that every company's data will eventually grow to be big data. I agree that "big data" was the wrong model. We also need to challenge that all data should be stored in one place (warehouse, lake, lakehouse). We need to challenge that one tool can be used for every data need. We need to challenge how we build systems both from a technology and people standpoint. We need to embrace that the problems and needs of companies _are always changing_.
We are living with conceptual inertia. Many of our patterns are an evolution from the 70's and 80's and the first relational databases. It's time to rethink how we "do data" from first principles.
The problem is that no tool alone can make data useful. It requires human ingenuity to come up with a theory, gather the required data, then test and verify the theory.
We've gotten to a point where the first and last step get skipped. Business leaders see other companies doing interesting things with data, so the answer must be "gather all the data"! Internal teams end up focused on gathering the data without the context of how it might be used.
We need to train data teams to not focus on the data as the product. Instead, they should be responsible for driving business actions. Gathering and cleaning the data should just a byproduct of that activity.
This post doesn't go far enough. It challenges the assumption that everyone's data is "big data" or that every company's data will eventually grow to be big data. I agree that "big data" was the wrong model. We also need to challenge that all data should be stored in one place (warehouse, lake, lakehouse). We need to challenge that one tool can be used for every data need. We need to challenge how we build systems both from a technology and people standpoint. We need to embrace that the problems and needs of companies _are always changing_.
We are living with conceptual inertia. Many of our patterns are an evolution from the 70's and 80's and the first relational databases. It's time to rethink how we "do data" from first principles.