I agree. for my use case, once a user is authenticated, you get his roles and subscription. There's a limited number of features or actions for each object type, and a limited number of object types. So you can get the set of rules in the client to manage UI, and apply the same set of rules on the backend in the API.
In this use case the authz calculation time will be the same with a million users and a billion objects.