I have been working for years on a system designed to do much of what is described in their 'Level 2' layer. It is a single system that can effectively manage unstructured (i.e. file) data, semi-structured data (NoSql), and highly structured data (RDBMS).
I don't have the resources available to me like this team, so there are still a lot of features on my TODO list that would enable some of the things they are looking for, but it can do many things now. It can manage 200M+ files and find subsets of them in sub-second speed. It can build relational tables with hundreds of millions of rows and thousands of columns and perform queries faster than many conventional DBs.
As many people know, CAP stands for Consistency, Availability, and Partitioning which applies to distributed systems. You get to pick two. I chose the AP part with eventual consistency. 'Eventual' is the term with variability. Some systems are faster than others in bringing the distributed nodes up-to-date with data changes across the system. I believe I have some algorithms that complement the structure of the data to make the window of inconsistency as small as possible.
The system is also designed for full ACID compliance, but there is still some work to be done in this area.
I don't have the resources available to me like this team, so there are still a lot of features on my TODO list that would enable some of the things they are looking for, but it can do many things now. It can manage 200M+ files and find subsets of them in sub-second speed. It can build relational tables with hundreds of millions of rows and thousands of columns and perform queries faster than many conventional DBs.
www.Didgets.com is where you can download the beta software. Demo videos at https://www.youtube.com/channel/UC-L1oTcH0ocMXShifCt4JQQ