I'm not sure whether you're taking issue with leisure time generally or the tendency for people to participate in leisure that you believe is inefficient, but both are orthogonal to the question of which type of agricultural system is more efficient.
In particular, if your solution is simply "spend less time on [your current] leisure activities!", it can be applied to either aggricultural system. In the case of specialized agriculture, it simply means spending more time on the individual's specialization or really anything with a greater opportunity value than growing one's own food (if your goal is to maximize earnings, it's tough to do worse than spending an hour per day growing your own food to save ~$5).
In particular, if your solution is simply "spend less time on [your current] leisure activities!", it can be applied to either aggricultural system. In the case of specialized agriculture, it simply means spending more time on the individual's specialization or really anything with a greater opportunity value than growing one's own food (if your goal is to maximize earnings, it's tough to do worse than spending an hour per day growing your own food to save ~$5).