For me the difference would be: if the write past the array has been detected before it happens (via a range check) it would be a regular error which which can be handled, while a panic would be "oops somebody else has written past the end of the array" (e.g. a hitting a canary check), which should abort immediately because at that point it's no longer guaranteed that any recovery code would even work or just make things worse (e.g. trying to flush data from memory back to disk, but that data might have been corrupted too).
You have a good point and lock poisoning had the same reasoning, but wasn't really popular [1]. (In hindsight the poisoning itself was a great idea but should have been decoupled from locks.) Compared to other languages with similar constructs, Rust panic is probably regarded as less fatal because an incorrect but safe Rust code tends to have a limited reach.
Ah, and it should be also noted that some non-fatal signals were also delivered only via panic. The best-known example is a memory allocation failure, which is recoverable in Rust but needed unwinding for a long time. Nowadays you have an unsafe but non-unwinding alternative.
