from Hacker News

The Sad State of Debug Performance in C++

by SuperV1234 on 9/26/22, 5:01 AM with 108 comments

by _gabe_ on 9/26/22, 4:30 PM
I ran into these problems with std::unordered_map. While developing my game, I would always run it in debug mode because I need to debug it. Loading a game world was taking around 12 seconds, and it was beginning to grate on me because of the increased iteration times. So I decided to profile it and see what was taking so long. I compiled in release mode to begin debugging, and it magically took the world load time down to 2 seconds.
I wondered what code I had written that caused such a massive performance hit in debug mode. So I went to profile my code in a debug build to find out. Lo and behold, something like 90% of the CPU time was wasted doing lookups from std::unordered_map because of debug iterators. I tried everything I could to just turn debug iterators off, and eventually gave up and switched to robin_hood::unordered_map which runs great in debug and release. Now I have an unwarranted aversion to the std lib, even though it really is great so long as you're running it in release mode.
by Deganta on 9/26/22, 2:25 PM
The problem is that C++ is full of so called "zero-cost-abstractions" that are only zero cost if you ignore:
1. Compile time
2. Non-optimized builds
Stuffing everything into the Standard Library using lots of template magic does has its upside, but debug performance is one of the big downsides.
Another one are really clunky interfaces, like std::variant. This should really have been a language feature, not a library features.
by PaulDavisThe1st on 9/26/22, 6:40 PM
I've been using C++ since 1993. I still don't understand how I've written all this code and never once found myself using, or wishing I could use std::move.
Presumably my code has been less than optimally efficient or something, because it seems that most people talking about "modern" C++ view it as absolutely central to the language.
by glandium on 9/26/22, 9:21 AM
> Even if -Og was ubiquitous, it is still suboptimal to -O0: it can still inline code a bit too aggressively for an effective debugging session.
IMO, the problem is not so much the inlining, but the sad state of the optimization passes losing track of many things and producing useless debug info.
by BenFrantzDale on 9/26/22, 10:22 AM
It’s always nice to understand more about why game developers do the weird things they do. I always buried and run in “RelWithDebInfo”, then if I really need it, I’ll throw `#pragma clang optimize off` around the code that the debugger can’t see into.
by humanrebar on 9/26/22, 1:30 PM
Have any multibillion dollar game (etc.) companies invested any resources into debug build modes in the open source compilers they develop with? Seems like it should be trivial to tie team productivity rates to quality of the debug experience.
by alternatetwo on 9/26/22, 9:46 AM
MSVC has /Ob1 [1] which only inlines specifically marked functions, or functions defined in a header. Shouldn't this inline all those trivial calls yet keep most debug info if you disable all the other optimizations? I haven't experimented with it yet *or what I rather mean is use Debug builds, but enable /Ob1 in them.
[1]: https://learn.microsoft.com/en-us/cpp/build/reference/ob-inl...
by staticassertion on 9/26/22, 5:44 PM
With our rust project we actually compile with `01` because the cost of that optimization pass is less than the cost to our tests running 100x slower. There is definitely a need for a 'more optimal' debug mode, and it is one of those things that feels zero cost until it isn't. Similarly, I try to at least consider compile times when writing code these days.
by andrewmcwatters on 9/26/22, 9:27 PM
You know, at least in C++ you get some really poorly performing standard containers. Every time I reach for C, it drives me nuts that the best you get is GLib.
And BOTH pale in comparison to newer language standard libraries like Go's.
by phaedrus on 9/29/22, 4:21 PM
This dovetails with something I was thinking about recently in the context of Reproducible Builds (or the lack thereof). For most of its history Computer Science (at least as practiced by industry) has gone all-in on "functionally equivalent" transformations, and done almost nothing in the space of "output invariant" guarantees.
In my day job we sometimes we need to reproduce the build of a firmware ROM or executable, sometimes decades after the engineer who last built it left. Getting a match is easier (or even possible) for older tech only because of the relative unsophistication of the compilers used - even then it's only reproducible "by accident," and not because the compiler vendor made any guarantees about X language construct reliably produces Y machine code and data layout.
But we need that! For getting accurate baselines. For security verification. And there's no reason in principle we should have to forego updating compilers, IDEs, and OS environments in the indirect hope of not disturbing anything. Those are two separate things: if we had a through-line of higher level language construct -> semantically defined transformation (irregardless of optimization settings) -> machine code, vendors could continue to update their IDEs and compilers while just making sure they still respect the invariants.
C++'s so-called zero-cost abstractions are poor substitute for this: header (library) writers and C++ gurus write code as if they worked like this "guaranteed output transform" I describe, but no compiler actually has to respect it (nevermind that the fine details of what the transformation actually is isn't nailed down) and it differs between Debug and Release build which is particularly bad for game development as TFA makes clear.
by 323 on 9/26/22, 1:52 PM
I worked on a CAD/CAM product, debug mode was literally unusable, it was 100-1000 times slower than optimized mode.
by halayli on 9/26/22, 12:52 PM
The std::move -> call issue has been fixed in clang-15.
by olliej on 9/26/22, 7:41 PM
The urge to have basic language features not be part of the language, but instead be part of the standard library is responsible for a lot of the less than stellar parts of of C++. Std::move/forward are particularly obvious cases, but you get similar accessing members of tuples, etc.
Wanting to keep things in the standard library rather than the language itself means that you have to compile large amounts of template hell for a wide array of basic things which hurts build time even in debug modes, and then as this article says you end up with no-op operations that become calls. You also can’t easily debug any of this because you end up with absurd layers of template nonsense for what is again basic functionality.
You can compile with -O1, but that then inlines things that aren’t part of the standard library that makes debugging of the actually relevant code annoying (this is what the debug llvm and clang debug builds do), as it inlines your own code and also means that you lose variables all over the place.
by bluGill on 9/26/22, 2:13 PM
I'm with Kernigham and Pike: the debugger is for getting a stack trace out of core dumps. Printf is for debugging.
by chubot on 9/26/22, 6:42 PM
It's also annoying for tools -- like debuggers, and uftrace (user space function tracing), which will start showing all these tiny functions that you don't really care about
I prefer to avoid all the operator overloading and smart pointers, etc.
by StellarScience on 9/26/22, 3:19 PM
MSVC lacks -Og because it instead treats inlining, optimization, and inclusion of debug information as separate, orthogonal choices.
These days I rarely build a whole MSVC project as "Debug" without optimization. Instead I enable both optimization and debug information, so I always get call stacks and line-by-line stepping ability. When I end up debugging some file where the optimizations inhibit debuggability, I'll recompile just that compilation unit or library without optimization or inlining.
That said, I agree with the gist of the article that both "zero cost abstraction" and debuggability need to be constantly improved.
by not2b on 9/26/22, 7:40 PM
The author needs to learn about the -Og switch, which can be used together with debugging. It can be thought of as "do those optimizations that don't interfere with debugging". I verified with Godbolt that this suffices to eliminate the call overhead.