i think your third paragraph is more or less false for apps over a certain size, think enterprise-y stuff. it is certainly false for the class of apps that i work on - big pharma regulatory compliance software, very complex data model, hundreds of tables etc. Our ORM can express higher level queries and a wider set of queries than raw SQL - you can express a query in one line (one thought) that compiles down to quite a few nested SQL expressions. Finally, the problems of orm stem from the fundamental nature of SQL, so dropping into raw sql couldn't possibly fix them. You need something like Datomic or CQRS/ES to remove the object/relational impedance mismatch at a fundamental level. (This is analogous to why Git rocks compared to SVN; Git solves the problem of centralized mutable state at a fundamental level which opens the door for a better model of the problem and more powerful abstractions.)
I hear that these sorts of things use EAV [1], which is traditionally something that ORMs do not handle well. The rationale of EAV over another model is that entities have MANY optional attributes and you'd have horrible data sparseness (without EAV).
But you say you use an ORM:
> Our ORM can express higher level queries and a wider set of queries than raw SQL
Are the anecdotes I hear about using EAV in these sorts of applications right or is the problem domain so big that there's room for EAV and non-EAV and nobody's wrong?
Whereas traditional EAV is a triple - entity, attribute, value - Datomic is a quad. Datomic adds a timestamp and treats every value as an immutable fact at a point in time. Which is exceptionally valuable for highly regulated systems like finance, pharma and healthcare.
yeah, Datomic is EAV (technically EAVT, it's event sourced so there is a notion of time and reads/writes need be acid with transactions and stuff) and EAVT can express even higher level queries than an ORM can, mostly because you can cache the indexes locally so you can do consistent read queries in your app process (Datomic's query engine is a library that runs inside your app, if cache is hot reads don't touch network... like git). From my understanding it doesn't have much to do with sparseness, though you certainly can support sparse objects performantly with EAVT.
EAVT/Datomic is a better fit for apps with complex structured data than an ORM over SQL, but migrating there is no trivial feat. Nor is convincing my customers in their due diligence phase, as they have surely been burned before by some kid pushing MongoDB, but they understand SQL and know the product can be successful with SQL, they know they can go get Oracle consultants to save their ass in 10 years when my company is sold, etc.
