I've always thought that a huge limiting factor is that we can only really observe from our current point in time. We are time limited which is a big hindrance just as it would be to only observe things form a single physical position (which is also sort of true - but at least we can send probes and whatnot out there).
