I do the exact same thing with multiple sites, using Asual's address plugin (which I love, btw... it also integrates with G Analytics perfectly). The only difficulty I have is getting people to use the non-hashed links instead of just copying what's in the address bar. Google treats all of the hashed links as the same link (everything up to the hash mark).
Such deep linking also needs to be combined with a progressive enhancement approach: each deep (hash-based) link must have a "non-deep" equivalent.
On my site, for example, both of these URLs point to the same "state":
http://alikewise.com/Profile/FounderGuy/Book/42
http://alikewise.com/Profile/FounderGuy#/Book/42
The first one is SEO-friendly, and also friendly in non-JS situations. The latter allows for a nice ajax experience.