Internally a lot of editors don't nest inline styles like that. The formatting is more like a style attribute on a <span> tag, listing all style properties. So you still only have two positions, previous formatting to the left, new formatting to the right.
If I were implementing this, I would ignore the middle one. There's virtually nothing visible to the user that there is a middle one OR that <b> is actually placed before <i>. I feel handling this edgecase causes more UX problems than not.
The approach I took with my own hacky editor component was to highlight the styles that currently apply at the caret position via toggle buttons for B, I, etc. This is something word processors have been doing for decades, although I recognise it's far from a perfect solution.
For example, “<b><i>text” has three insertion points shown with the pipe character: “|<b>|<i>|text”