This is not true. You are ignoring the most important factor of all, which is whether the company's share price is above or below its "intrinsic value". Of course, intrinsic value is a fundamentally fuzzy concept, but in some cases the share price is obviously out of line relative to historical measures of valuation (price to tangible book value, TEV/EBITDA, price to sales ratio, etc.). If there is no good fundamental reason for the lowered valuation (loss of patent protection, entrance of new competition, secular changes in technology, etc.), then it could create value for remaining shareholders if the number of shares is reduced and that capital is replaced with debt (or simply spending excess cash on the balance sheet). If the share price is above a conservative estimate of intrinsic value, then a company can easily destroy shareholder value by buying back shares. In that case, they might be better off issuing more shares or doing an acquisition for stock to take advantage of their overvalued currency. This sort of capital management is one of the most important ways a management team and board of directors can create value over time. For a good historical example, look at the case of Teledyne ( http://www.capitalideasonline.com/articles/index.php?id=2725 )
It's true that intrinsic value is not usually well defined (except in simple cases, where a company is going to be liquidated and has clearly defined assets and liabilities that can be precisely measures, such as cash/marketable securities), but that doesn't mean that a thoughtful investor can't determine a reasonable range of likely values that errs on the side of conservatism. For example, by zero-valuing assets that are hard to quantify, such as an undeveloped plot of land. Or if there is an annuity-like cash flow stream from a long-term contract, one can discount those cash flows to the present using a conservative (high) discount rate. In cases where the share price is way below a conservative downside-case valuation, it's not necessary to know with precision what the intrinsic value might be.
Individual investors can define a proxy for intrinsic value. But, overall investors will come up with different numbers. Thus investor A may think a buyback was a net gain, while investor B may think a buyback was a net loss.
