Bruce Schneier and others[1] have done the math on brute forcing 256 bit keys: even with a perfectly efficient computer using the least amount of energy possible, you would have to deplete the entire energy content of the Sun to just iterate over a 225 bit keyspace once, let alone do anything meaningful with those keys.
It's estimated there are 10^80 atoms [1] in the visible universe, so 2^256 is definitely a huge number. I didn't realize 256 bit brute force was nigh feasible with only a solar system.
I'm a bit surprised the quantum algorithm only gives a polynomial speedup.
It's all down to probabilities, yes our hypothetical attacker could guess your key correctly the first time or within a few years but the chances are so tiny it approaches zero for practical purposes on practical timescales.
