Yeah, I guess considering that there's no installation process for actions that's kind of a necessity.
Maybe as the technology matures Google will start getting smarter about being able to infer which provider I want to use for particular types of actions, but I recognise that's likely to be a rather difficult problem to solve.
Somewhat understandably the direct actions are limited to certain domains that Google supports. It isn't a free for all.