I don't know the details of the system you've created, but you could use a 'restrictive' open source license like the GPL, and offer a dual licensing strategy, where people who want to use it in commercial systems pay to get a compatible license. That only works if it's a library though, rather than an independent system.
Another idea might be to find a partner first, and let that person handle most of the business, and just concentrate on the technical aspects of it.
The system calls several GPL based components (SPASS, aspell, ...) but does not extend any thing under GPL.
I was thinking that if I open-source it I would put it under a very open license (e.g. BSD) and assume/hope that anyone that extended it would make contact with me for help/advice.
Another idea might be to find a partner first, and let that person handle most of the business, and just concentrate on the technical aspects of it.
Good luck!