Getting otf fonts working, getting images working and properly positioned, getting alternating left-right headers to behave themselves, especially at a chapter start. Every new thing involved going back to ctan (I had high hopes for memoir) or searching online in the hopes that someone took the time to write down how they did it. Most of the time you'd just find "how to make your math formulas look pretty" or "how to typeset your thesis"
You'd think that typesetting a novel would be a gimme, but in the current ecosystem (tex or otherwise) it's a shambles.
OTF is dead easy if you use XeTeX (specifically the xelatex command). The fancyhdr should give enough control over headers. Yeah, floating images can be annoying.
I use it mainly for technical documentation (product manuals, etc.), so I've never tried the memoir class. But what I find myself doing mostly these days is makinng my own classes, usually based on article because it seems to me to have the most 'neutral' defaults.
It's a very steep learning curve, but I've found it an extremely valuable technology to be able to use.
Asking questions on the TeX stack overflow is a pretty good way to get help also.
I agree with the last point: asking questions on tex.stackexchange is a GREAT way to get help. Sometimes you run into an "impossible" problem and some wizard will show up and fix it in minutes.
I'm typesetting a series of novels with LaTeX and memoir and I share your pain. I didn't have much trouble with fonts (fontspec is pretty awesome compared to the old TeX way), but things like typesetting on a grid, that was difficult.
I'm now reading on ConTeXt and it seems to be a much more adequate TeX format. For one thing, it supports grid typesetting! LaTeX's philosophy is to separate style and content, leaving the style to "someone else". It seems that ConTeXt's philosophy is to separate style and content, but still make it easy for the user to choose the style. It's also more integrated: comprehensive core, few third-party packages, meaning less compatibility issues (but less choice too). I'll definitely try it for my next project.
The main idea is to have every line of text fall on an evenly spaced grid. Here's an example where the middle paragraph is not grid aligned: https://i.stack.imgur.com/XskJu.png
Generally you also want the baseline of titles to fall on the grid. Possibly also the formulas, figures etc.
LaTeX is pretty bad at this: by default it inserts stretchable vertical space between paragraphs, and around things like bullet lists, centered text and formulas.
Stretchable space is a good thing when you have a lot of elements beside simple paragraphs: it gives TeX flexibility to produce a nice page layout. For example the optimal spacing around equations might not be a multiple of the inter-line space. And you might still want to have the last line land precisely at the bottom of the page, so there must be a stretchable space somewhere. This flexibility also helps avoiding widows and orphans[1].
On the other hand I think non-grid-aligned text looks terrible for novels, especially when the page is thin and you can see through the paper the text on the other side (it's much less noticeable if the lines on both sides are perfectly aligned). Grid typesetting is also nice when you have multiple columns of text on the same page: it looks odd if the lines of one column are not aligned with those in the next column.
Hm. Font stuff I'm not surprised, fonts can be tricky especially in Latex, you're definitely right. Bit surprised images were difficult. I guess I haven't had to deal much with margin and header issues. fancyhdr can certainly be confusing. Did you try "lyx"? I guess for what it does help with, it might not actually make any of these particular issues easier..
Of course, it's all about using a good template. Usually I find there are tons of resources out there, but it does often involve finding someone with the same problem and asking the right question.
You'd think that typesetting a novel would be a gimme, but in the current ecosystem (tex or otherwise) it's a shambles.