On random numbers, I found the ideas behind "Parallel Random Numbers: As Easy as 1, 2, 3" [0] (which is implemented in JAX) interesting.
Their generator is a mapping between keys and random numbers which get rid of the sequential dependency in the generator making it very easy to run in parallel at a GPU scale.
I've been diving into parallel reproducible PRNG [1] in the end I choose to have a PRNG based on xoshiro and seeding process that could take loop iteration variables [2] for reproducibility:
[0]: http://www.thesalmons.org/john/random123/papers/random123sc1...