Ah, thanks for clarifying! I think someone already called this out above, but the benefit of the approach I describe is that you only need one sample per pixel. Four samples per pixel starts to feel pretty slow on my machine, and I suspect you'd need more than that to make the averaging approach look good.
One option might be to render, say, two samples per frame, and keep the previous samples when the light source doesn’t move. The shadows would be noisy while the light is moving, but quickly look smoother when it stops.
