Yes. In the absence of copyright/patent/trademark, everyone can use every bit of knowledge they come across.
The "natural right" is the right to use information. Copyrights, patents, and trademark are constitutional limitations of this natural right, hoping to encourage more creation and sharing of knowledge to increase the total amount available to all.
Trademarks don't really affect that, that's why I don't like the term IP. Trademarks simply say you can't present yourself as others, not that you can't use any kind of information.
The "natural right" is the right to use information. Copyrights, patents, and trademark are constitutional limitations of this natural right, hoping to encourage more creation and sharing of knowledge to increase the total amount available to all.