If you're worried about copying the code, I could see having the self hosted artifact be compiled or otherwise obfuscated would probably deter most people who wanted to reverse engineer it, since it would take significantly more time and effort to open up a compiled app than something just made in JS for example.
If you're worried about copying the API design, well, the implementation is the hard part, not necessarily the API design, which the Google vs Oracle lawsuit also showed, interestingly enough.
EmailEngine has all the code public [1], even though it is not open-source but is source-available. Some core parts I even published under the MIT license, like the IMAP client library I built from scratch to serve the special requirements EmailEngine has for IMAP access [2]
My thinking has always been that those who try to hack the license validation stuff and replace the missing build pipeline were never going to be my customers in the first place, so every second I would spend on them is a wasted effort.
If you're worried about copying the API design, well, the implementation is the hard part, not necessarily the API design, which the Google vs Oracle lawsuit also showed, interestingly enough.