Apart from the tragedy of the commons problem, it's also hard because we don't know if the benefits will outweigh the costs. As far as I've heard, the science doesn't answer that accurately enough to actually make a decision. We might be better off with global warming than with a premature end to oil and coal.
Bear in mind that global warming isn't the end of the world, it's just a change. Humans will certainly adapt and carry on. The problem is the costs might be inconveniently high - or they might be tolerable.
you are likely referencing a report that concluded the costs for stopping climate change were higher than the costs of dealing with the consequences. The report that I read on that topic came to its conclusions by valuing people's lives in dollar figures and putting those lives at less value for anyone in developing countries.
So, yeh, if you think poor climate refuges have no value then you are right we should take no action.
The problem is that those holding onto power have politicized climate change so that those not paying close enough attention wrongly conclude that we don't need to take action.
Bear in mind that global warming isn't the end of the world, it's just a change. Humans will certainly adapt and carry on. The problem is the costs might be inconveniently high - or they might be tolerable.