It’s a parquet file, with some json as metadata to define the version boundaries.
That’s all.
You too can have transactions if you implement your access layer appropriately.
It’s a useful format, but let’s note pretend it’s any more magic than it actually is. I’ve also not noticed any improved performance above what partitioning would give you.
Delta metadata based min/max statistics, Parquet row groups and min/max in-file, combined with the right combination of bucketing and sorting, allowed us to actually ignore all partitioning and drastically improve performance on PB scale. The metadata can be a very useful way to do indexing/allows you to skip a lot of things.
It's not magic, and you can understand what the _delta_log is doing fairly easily, but I can testify that the performance improvement over partitioning can be achieved.
>You too can have transactions if you implement your access layer appropriately.
Yes, we can all "just" make it ourself, still nice when others make it.
Personally I very much like that it's conseptually easy and that it builds on an open source format (parquet). I also expect there to be a dragons, both small and large, in actually getting the acid stuff right,so I am happy we can fight them together.
I think it is just API layer on top of existing storage systems like s3 and hdfs: https://docs.delta.io/latest/delta-storage.html#amazon-s3