Additionally the XHTML content can just be a single large file instead of one file per chapter/section. Paginating and rendering the large single file is going to be more effort than the same on a smaller file. This is all on top of the pitfalls and variability you mention.
Yup, great point. Especially if you've used some tool to convert from another file, like a PDF, into an EPUB, you can easily end up with the entire book in a single XHTML file, which, again, can be pretty heavy for a browser to parse and format! I also have no idea whether Calibre et al actually use native web views, or have their own renderers, which are almost certainly less performant than native web views!