"Use a 3D engine with Z-buffering" does seem like the right solution to these problems today. However, even if you don't, taking inspiration from one seems like another way to solve this class of problems, including the case of the three mutually overlapping boxes in the last example: independently decide the "front" box to draw for each screen coordinate. For each box, record which boxes appear "in front" of it, non-recursively. (So, A > B, B > C, C > A.) When rendering, only consider boxes that would render at the given screen coordinate in the absence of occlusion. (So, in the given example of mutual overlap, any given screen coordinate only needs to consider two boxes.) Then, render the front box at that screen coordinate.
If you have any translucency (water, plants/trees, objects), then render the frontmost opaque box followed by other translucent boxes in front of it, in back-to-front order.
(That approach would completely break in the presence of concave shapes, which could allow a single box to appear both in front of and behind another at the same screen coordinate.)
Depending on your shapes (e.g. boxes), you wouldn't need to do that for every screen coordinate; you could make the same decision for entire regions, bounded by the screen-projected coordinates of the vertexes.
However, once you reached any significant degree of complexity, you'd probably still be better off implementing a full 3D engine in software if you didn't have 3D hardware. Z-buffering isn't particularly complex to implement. (More complexity comes in when trying to pre-cull some geometry without rendering it at all, to avoid rendering over the same screen pixel many times.)
Thanks for expanding on my musings! Two thoughts to add:
1. Pretty much every 3-D graphics engine worth using supports importing and batch-rendering meshes that are actually made up of multiple sub-meshes. So you can fix the circular occlusion problem and the concave object problem with pretty much no performance loss by just subdividing those offending shapes during the art phase. This simplifies rendering because the engine doesn't need to do any dynamic mesh subdivision.
2. Pretty much every 3-D graphics engine worth using supports depth sorting for translucency, so you effectively get that for free too.
I should have clarified that "3-D" by itself doesn't necessarily get rid of your problem. However, there's very little reason to build your own graphics engine anyway, and modern 3-D graphics engines abstract away the occlusion problem for free.
I suppose I had a third point, a response to your last paragraph: it seems hardly worth it to basically implement your own software rasterizer with a RAM based Z-buffer if getting the dedicated hardware to do it is just a matter of pulling in a 3-D graphics engine. And doing it on dedicated hardware is going to be much faster. Granted you have to learn how to use the 3-D graphics engine and you'll have to do a bunch of work through shaders, but if you're doing graphics stuff, this is staple, bread and butter knowledge. Kind of like how web frontend developers would be expected to be able to write CSS and to be fluent in at least one JS framework.
> 1. Pretty much every 3-D graphics engine worth using supports importing and batch-rendering meshes that are actually made up of multiple sub-meshes. So you can fix the circular occlusion problem and the concave object problem with pretty much no performance loss by just subdividing those offending shapes during the art phase. This simplifies rendering because the engine doesn't need to do any dynamic mesh subdivision.
True, but this problem also just goes away if you represent objects as meshes of triangles, because individual triangles aren't concave.
> it seems hardly worth it to basically implement your own software rasterizer with a RAM based Z-buffer if getting the dedicated hardware to do it is just a matter of pulling in a 3-D graphics engine.
Agreed completely, if you have that hardware available. I could imagine constrained-computing scenarios (either as a challenge, e.g. "standalone minecraft clone in 4k of code", or a deeply embedded environment) where you wouldn't have a 3D engine available and you might want to implement a simplified rendering engine for a highly constrained rendering problem.
But yes, for almost every scenario, you want to just use the GPU.
I'm going into pedantic territory, but since we're on the topic of drawing order for translucent meshes... might as well.
You don't actually avoid the problem entirely when you're using just triangles because it's still possible to construct a "A overlaps B overlaps C overlaps A" cycle of occlusion using three triangles. It's more or less the same corner case as a cyclic occlusion problem with convex pole-shaped meshes.
For this reason it seems like graphics engines tend to just batch-render entire sub-meshes at a time because rendering entire meshes presents little additional complexity over rendering individual polygons (and is more performant). It's a leaky abstraction but the leaks are pretty rare.
If you have any translucency (water, plants/trees, objects), then render the frontmost opaque box followed by other translucent boxes in front of it, in back-to-front order.
(That approach would completely break in the presence of concave shapes, which could allow a single box to appear both in front of and behind another at the same screen coordinate.)
Depending on your shapes (e.g. boxes), you wouldn't need to do that for every screen coordinate; you could make the same decision for entire regions, bounded by the screen-projected coordinates of the vertexes.
However, once you reached any significant degree of complexity, you'd probably still be better off implementing a full 3D engine in software if you didn't have 3D hardware. Z-buffering isn't particularly complex to implement. (More complexity comes in when trying to pre-cull some geometry without rendering it at all, to avoid rendering over the same screen pixel many times.)