Hacker News new | past | comments | ask | show | jobs | submit login

It already went away when I tried to print out the address of the variable, so that I could watch it in the debugger (because, in order to take the address of it, it had to become a stack variable).




In the end, do you remember what tools you used to confirmed that R11 was overwritten? The tools and the path to the root cause are also quite interesting.


I first looked at R11 because of the assembly output. There are flags that you can give g++ to produce the assembly output when it compiles. That showed me that the variable was in R11, and where it wound up on the stack in the g() function body.

From there, it was a question of how g() was smashing the stack. (I hadn't looked at that before, because I assumed that it had to be f() smashing the stack in order to change the variable.) Well, the next thing on the stack was the structure for mesgrecv. If too much got read into it, it would overwrite the stored copy of R11. That led me to look very carefully at the mesgrecv call. Checking the parameters against the man page showed up the unexpected (at least to me) requirements for the size parameter.

I never "verified" that the stored copy of R11 was being overwritten, except by changing the size parameter and noting that the loop in f() never terminated any longer.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: