IMO, part of why developers didn't use it was the API design made using it too painful. They took most of the existing file APIs and added new *Transacted APIs alongside them, e.g. CreateFile begot CreateFileTransacted, etc. So to add filesystem transactions to your existing code, you had to change all the file API calls.
I think, if they had made transactions work with the existing file API–such as by storing the hTransaction in thread-local storage, with an API to get/set the transaction association of the current thread, and having the existing non-transactional APIs behave transactionally in that case–it might have saw more adoption by developers.
I think I remember the API being marked as deprecated, though. Or is my memory playing tricks on me again?