Please help me understand this better. So HStoreField is not exactly Json field type of postgres but rather a subset of it, i.e just string->string mapping without any nesting. You cant even have key->array mapping?
Basically yes. hstore and json have entirely separate interfaces in Postgres, but the storage properties of hstore are a subset of json as you say. json/jsonb is generally more powerful, and will supplant hstore in a lot of new projects.
However hstore has been a part of Postgres since version 8.2 released in 2006, and I think was around as a third-party project before that, so it's used by a lot of projects.
It's also easier to manipulate hstore values then json values within the database. hstore's simplicity allows operators for combining and subtracting hstore values[0] which json doesn't have. Due to its restrictions, an hstore can also be converted to and from a record type (think database row). JSON fields on the other hand will generally be populated by your app and then only queried by the database.
It humors and saddens me that Django is a web framework with an ORM bolted on. SQLAlchemy has had a postgres driver that supports HSTORE and JSON/JSONB for ages.
Likewise, Django has had third-party hstore and jsonb fields for ages, but I take your point. SQLAlchemy is beautiful.
The Django ORM has taken great strides lately. Another big improvement in 1.8 is the expansion of database expressions – you can now combine arbitrary database functions and columns as you can in SQLAlchemy. This is a big step towards a cleaner, logically consistent API, and makes extensibility (like that offered in the new contrib.postgres) much easier.
Django's ORM also does things that SQLAlchemy's doesn't. Even at its highest level SQLAlchemy requires explicit joins AFAIK.
That said the Django team is aware of SQLAlchemy's arguable superiority and there have been numerous proposals to integrate it somehow. The most likely plan would be to use SQLAlchemy core to replace the low-level SQL generation. That way you could take a Django queryset and get its SQLAlchemy representation, then continue building from there.
SQLAlchemy is awesome, but it's an order of magnitude more complex to setup and configure than the Django ORM. With Django you get the integration with migrations, a system to spin up and tear down test databases, less boilerplate code required during model configuration (like not needing to specify the name of the database table), etc...
From the quick searching I've done, I think that Django predates SQLAlchemy by nearly a year (July 2005 and May 2006 respectively).
Alchemy has a lot more power behind it, but the Django ORM is (arguably) easier to use for the majority of web use cases.
I had a hand in opening up the Django ORM with the introduction/formalisation of Query Expressions (https://docs.djangoproject.com/en/1.8/releases/1.8/#query-ex...), and others are doing significant work in cleaning up the ORM internals so that more complex features can be layered on in core and by users.
Why does it sadden you? It seems like the majority of use-cases for database interaction (at least with the relational ones, likes PostgreSQL or MySQL) occur in web frameworks. The coupling makes a lot of sense to me, especially given the opinionated nature of programming in Python.
HStoreField is going to be a significant improvement over the JSONFields we have been using...
[1] https://docs.djangoproject.com/en/1.8/ref/contrib/postgres/