I think this was well prioritized; they struggled with the issue at times, found a temporary workaround, but when that workaround stöd being efficient and the bug hit them everyday, they decided to track down the source. Then they reported upstream, it was reproduced, and someone patched it, and rolled out new, fixed kernels.
That is a perfect example of how things works and should work. They contributed to the community. I think it was a great prioritization.
I'm certain there were lots of other people hitting this bug and killing processes or rebooting to get around it. The troubleshooting and reporting done here, silently saved a lot of of other people a lot of efforts - now and in the future. I don't think they were after it to be heroes; they just shared their story, which I'm sure will encourage others to maybe do the same one day.
[Author here] Your comment resonates with me. Indeed, we had been working around the issue for a number of years before we decided to tackle it head-on and even then we set a deadline for debugging and had a fallback plan as well: if we didn't manage to figure something out in a couple of days, we would switch transports or ditch rsync altogether.
In the end I believe we struck a good balance between time spent and result achieved: we gathered enough information for someone more familiar with the code to identify and fix the root cause without the need for a reproducer. We could have spent more time trying to patch it ourselves (and to be honest I would probably have gone down that route 10 years ago), but it would be higher risk in terms of both, time invested and patch quality.
Finally, I'm always encouraging our teams to contribute upstream whenever possible, for three reasons:
a) minimizing the delta vs upstream pays off the moment you upgrade without having to rebase a bunch of local patches
b) doing a good write-up and getting feedback on a fix/patch/PR from people who are more familiar with the code will help you understand the problem at hand better (and usually makes you a better engineer)
c) everyone gets to benefit from it, the same way we benefit from patches submitted by others
That is a perfect example of how things works and should work. They contributed to the community. I think it was a great prioritization.
I'm certain there were lots of other people hitting this bug and killing processes or rebooting to get around it. The troubleshooting and reporting done here, silently saved a lot of of other people a lot of efforts - now and in the future. I don't think they were after it to be heroes; they just shared their story, which I'm sure will encourage others to maybe do the same one day.