from Hacker News

My Favorite Debug Ever

by strzalek on 1/4/16, 10:26 PM with 22 comments

  • by mrich on 1/5/16, 11:56 AM

    I prefer using AddressSanitizer for finding bugs like these now. In addition to heap overflows it can also find stack and global overflows.

    As a second option I'm using valgrind which can find uninitialized memory (and doesn't require recompile of all libs as MemorySanitizer does).

  • by timclassic on 1/5/16, 3:06 PM

    ElectricFence is great, and I recall having a similar feeling to the OP when I first used it because many of the described issues were also new to me at the time. It's a great experience :)

    If you're on Solaris or a Solaris-derived OS, you also have the libumem and watchmalloc libraries that can help you out. I have used libumem to great effect in the past.

    It's been a while, but I anecdotally recall that Solaris is more stingy with its mallocs than Linux. I used to compile and run my C projects on Solaris as a first pass in my search for memory errors.

  • by teddyh on 1/5/16, 12:23 PM

    Is using GDB and Electric Fence to find a normal buffer overflow bug really interesting enough to warrant a blog post these days, let alone a HN story?

    I guess it might be interesting for all the people who, like the author, are Java and PHP programmers and have no experience with C?

  • by vardump on 1/5/16, 3:57 PM

    Usually the toughest things to debug is when stack is overwritten. Worse, if it's stack and heap. Not my favorite things to debug...

    Here's what I usually do in such cases:

    Call stack is unreliable at that point. You'll see puzzling things like function calls with impossible parameters, etc. If things just don't make any sense, it's better to map all code paths that can lead to the crashing EIP/RIP (hopefully valid pointer to the instruction that caused the crash). Check EIP/RIP if it's in some rep movsd (= potentially inlined memcpy, check ECX (RCX) rep counter, EDI (RDI) rep pointer), or if the execution is in some runtime library code such as memset, memcpy, etc. similar. The next thing is to make sense of the call stack manually, if there are portions not overwritten, but what stack walk couldn't resolve. Of course it pays to take a look around stack otherwise as well, for signs of overwrite and contents of the overwrite.

    It's also possible a pointer to stack object leaked at some point and the crash occurs at completely different part of code than where it actually segfaulted. Or some runtime structure was corrupted, like heap. Sometimes you can find those by just inspecting and guessing struct/object shape and values near stack pointer ESP (RSP).

    If the bug can be reproduced, memory breakpoints, logging (especially if multithreaded, but watch out for blocking I/O from logging), tools for debugging memory corruption (valgrind, compiler paranoid mode, etc.) etc, even mapping some pages unreadable and unwritable. It can take a while to find the actual bug.

    If reproduction is not possible, good luck. Better spend some quality time with memory hex view, disassembler, trying to locate registers and stack values that might contain pointers, etc. It might take a while to find the issue...

  • by adricnet on 1/5/16, 3:26 PM

    Thanks for sharing this! I could wish for some tool use (gdb) screenshots, though really the inspiration to learn is valuable enough.

    A next read in this vein might be Bug Hunter's Diary by Tobias Klein: https://www.nostarch.com/bughunter

  • by 64bitbrain on 1/5/16, 4:46 PM

    Great article! I had a similar experience, a Java programmer turned Linux Kernel programmer. The first think I remember doing was debugging a kernel panic using crash. It is not doubt challenging for Java/PHP programmers to debug C coredumps/KP, when there is pointer reference involved. But at the same time I learned a lot, specially memory management.
  • by mschuster91 on 1/5/16, 3:12 PM

    I wonder how many of the young so called "programmers" would be able to solve this. It's a pity that one can call himself a "developer" without having the tiniest idea how a computer actually runs, what a process, a thread or a pointer is...