While it is probably true that the provided library is incompatible with sleep modes, it is also true that ESP sleep modes are said not to work right. Just from the data sheet, the regular sleep mode draws quite a lot more current than it should. Apparently, getting it to wake up reliably from a deep sleep mode, instead, where power draw would be reasonable, is more tricky or unreliable than is documented.
If you want a true low-power system, you probably need to use STM or something else.
It has a second chip (AXP202) that is responsible for battery /power and has the button directly wired to it. That's why I believe deep sleep should be possible. If it needs to reboot instead, fine I can accept a ~second delay. But again, the supplied firmware boots 5 seconds and I've no idea why and how to rectify it.
If you want a true low-power system, you probably need to use STM or something else.