Over the years I've become more and more convinced that debugging has done a lot to develop my problem solving skills and analytical thinking in general. It's like being a full-time modern times Sherlock Holmes with the crime scene and tools neatly at your disposal.
Usually breakpoints and stepping through code solves things pretty easily but some bugs can be real stumpers.
Here's some of the tools I tend to use:
1. What has changed? What could have caused this bug to appear?
2. Verify fundamentals: Is it actually running the code I think it does. If I change something that should change something, does it change? If I break something does it break?
3. Verify Input: Make sure it's valid
4. Verify output
5. Timing issues?
6. Slowly remove code "around" bug until it works again
7. Or start back at something you know works and add back code until it breaks.
8. Break out the functionality into an isolated environment so you can test it separately so you'll know if the problem is within the feature or a side-effect of something else
9. If your code makes it too complicated to see and find the bug refactor until it isn't.
10. Go to sleep / Go on a walk. Let your subconscious process it
and finally, lot's of bugs are made self evident by well structured and refactored code, so don't get too busy working on the symptoms rather than the root cause.
Those should have been 0 and 0.1 in his list. Agree with all 12 points. My little grain of sand: use all the tools available to avoid bugs: a good IDE and/or code linters/inspectors can work wonders on a codebase. I've seen people still programming with "just a text editor" (not Sublime, vi, emacs or any powerfull text editor, I'm talking little less than Notepad with syntax highlighting) telling you "it's just I don't like IDEs" when asked about it.
I'd even dare to say that it is better to shove Vi/Emacs experience into a real IDE rather than the other way around.
I really like Vim, I use it for almost everything, however when I am working on a large C++ codebase I tend to use an IDE that has full support for visual debugging, code navigation and so on. I am always amazed when some more experienced programmers than me spend valuable time grepping files and using GDB on command line.
> I am always amazed when some more experienced programmers than me spend valuable time grepping files and using GDB on the command line.
I don't know about Vim, but Emacs works very well as a visual debugger, for quite a couple of languages. It looks like this for C, for example: http://www.inet.net.nz/~nickrob/gdb-ui.png (it's Emacs debugging itself in this screenshot).
In general both Vim and Emacs can become IDEs that are on par with other offerings. For Python development, I worked with Komodo, PyCharm, Vim and Emacs. With jedi[1] both editors get context sensitive autocompletion (for some strange reason called "IntelliSense" or something by some) and "find definition", "rename identifier" etc. With Rope, they both get nice refactoring support. With Magit/Fugitive, you get the prettiest and most functional GUI for git. With Speedbar/TagBar[2] you get classes outline. With yasnippet/vim-snipmate you get configurable, programmable snippets/templates. And so on and on - all that on top of largely superior editing model that is being developed and improved for 30 years.
I worked with Komodo, PyCharm and a couple of other IDEs, then switched to Vim and I didn't feel that I'm missing something. Then I switched to Emacs because I wanted an editor that I could easily customize as much as I'd like, and VimL just didn't cut it. There is nothing comparable to Emacs in terms of extensibility, LightTable and Atom may get there with time, but I suspect it will take quite a few years. No other IDE even tries to approach this level of extensibility and customizability.
While you can indeed transform Emacs/Vim to a good IDE, there are lot of (experienced) people who do not. Anecdotally I have seen quite a lot of posts boasting use of 'vanilla' editors without plugins.
Regarding your GDB screenshot, that is certainly good, I would argue though that using pure text (with some images included) is limiting oneself.
This being said, I really hope that NeoVim will go far enough and bring good enough integration of Node.js and javascript in general (my current work uses these technologies). So far I have been quite disappointed by stuff like tern[1]
Sometimes (especially when concurrency is involved) this is not feasible, yet debugging is still possible. These are the cases that really test a developer's skills.
Regarding 7, I think you should explicitly mention "git bisect" and similar tools. (Or, are you doing that step by hand?!)
For me, when I run out of ideas, "git bisect" is the most efficient debugging tool. On success, it allows me to pin down the exact commit that introduced the bug. It is very fast to do (binary search), and on success it narrows the bug down to a relatively small amount code to check.
Of course, this only works if you have a clean code history: small commits and always a working state after each commit. That is, no intermediate crap commits, and no monster commits doing 3 things at once.
I am sorry, but it is contrproductive. It is better to invest time into better code, which will find and tell you about problem, because such code can be reused many times by whole team.
Instead of playing Sherlock role, think: why your code allowed you to make that bug? Maybe you need to improve your code, add more test cases (or improve existing), add beter comments, add more examples, add more flexibility in configuration (so you can test program subsystems in isolation or in mock environment), use better tools, use toos for automatic verification of code, etc.
Your code is your debugger. If you will write properly, you will need not to run debugger at all, because code will explain problem to you (or to future user or administrator of your program).
Sometimes that is true; if your architecture is suboptimal then yes you can trim away huge classes of bugs with a high-level change. But as with all aspects of software development, the truth is a question of nuance and judgement.
The way your assertion can go wrong is when you don't recognize the errors that your rewritten code will add. At a high level we always envisage perfect code, and existing code is always seemingly full of warts, but no perfect coding plan ever survives contact with reality. Those warts are bug fixes and optimizations which make the thing work. Don't be too quick to rewrite lest you create bigger problems than you started with.
IMHO, precious moment when bug is found in the code AND it is not obvious, so you need to run debugger, is perfect time to improve your code. It indicates that code is complicated, or not well covered, or poorly commented (because bug is not noticed at peer review). You are going to test this piece of code anyway, because you need to check is bug fixed, so why not to spare some time to make code better and battle-test your improvement: if you catch bug in the process, then you improved code, if not: keep going.
And no, I am not talking about asserts. Assertions are only small part of real debugger.
Typical tools I use in my own programs are: good logging and tracing, which are able to explain what happens inside program without spamming me, some test cases for critical parts and for fixed bugs. Often, I will also split large program or script into smaller independent parts, with their own options, configuration, loggind and test cases. It's all.
I've found that it is often impossible to debug multi-threaded code in a debugger. Hitting a breakpoint and single-stepping through code on one thread smashes any other reads that are operating and expecting the thread that is stopped to respond in a reasonable timeline. It can also cover up race conditions, and just generally dork up anything that is time-related.
If you have a race condition in a multi threaded program, a simple printf is enough to screw the timings and make the bug disappear, it's not at all better than breakpoints and stepping in a debugger.
In my experience, the only way to produce safe multi threaded code is to isolate the threading parts and synchronization and stress test them early on. At this point I use a combination of prints, random delays and simple asserts on invariants that the code should hold. Most issues are reproduced within about 10 seconds of 100% CPU core utilization. Some nasty corner cases may require minutes of grinding away.
Multi threaded debugging is an unsolved problem. There are no universal solutions to this problem. Conventional tools often fail so badly that it is best to write and test your multi threading / synchronization code in isolation, and then applying it to the actual workload which is tested in a single thread.
Another option is trying to build a formally verified model ahead of time (e.g. using Spin - www.spinroot.com) and then write the actual program after the model has been verified.
> If you have a race condition in a multi threaded program, a simple printf is enough to screw the timings and make the bug disappear, it's not at all better than breakpoints and stepping in a debugger.
If you have a race condition in a multi threaded program, a simple printf may be enough to change the timings and make the bug disappear. But a breakpoint and stepping in a debugger is guaranteed to completely change the timings (unless your timings are on the order of minutes).
In concurrent software development, the third way (analysis) is the only way. You need to know (or find out) which resources can be accessed concurrently, and develop a theory of how this could lead to the observed error.
I remember stepping through many programs when I started programming as a teenager. It was quite fascinating to see everything in slow-motion.
Over the years, I had less and less use for intense debugging sessions, but I've been reading and writing logs all the time.
Today, I see server side code without logging statements, I immediately ask myself, how do the authors debug this thing?
Related:
> If you want more effective programmers, you will discover that they should not waste their time debugging - they should not introduce the bugs to start with.
Had to google the quote about wasting your time debugging to see that it was from "The Humble Programmer". I completely disagree with it! There will always be bugs. It's good to strive to avoid making mistakes, but however hard you try, there will be bugs. It is counter-productive to pretend there won't be any.
It's good to strive to avoid making mistakes, but however hard you try, there will be bugs.
That's basically what the quote is implying - you should not be throwing things together and hoping it will work after debugging, but carefully designing to avoid bugs, so there won't be much if any debugging required.
An analogy to this can be found in the aviation industry: it's known that pilots do make errors and planes do crash, but such errors are treated as unacceptable and they try their hardest to minimise them. It's been a very successful strategy.
All too often, the "carefully designing" becomes some sort of masochistic ritual where all errors are attempted to be piped through a single tool in the dev chain. Static typing is popular nowdays. As are high level functional strategies.
It is great when these work. Or when you have a high level team that doesn't abuse the system at all and gets things done using these strategies. I have seen too many times where the result is a system that even the creator does not fully understand. Worse, what should be simple changes turn into giant modifications of a Rube Goldberg machine.
Just because it can be done badly is not a reason for abandoning it. To continue the aviation analogy, the solution there was to ensure that the people designing airplanes have had fairly rigorous training and that they have an adequate understanding of the relevant theory.
Also if you run on production scale it is worth investing in log tooling.
It's so much easier if you can search what has happened instead of guessing. Also customer experience is much better if you tell them that is fixed instead of asking for details :-).
Disclaimer: I work for log management company (Sumo Logic).
They are good for being faster to debug in completely different ways. Logs do not provide you of exploratory inspection nor interaction with the objects present in a given point of the stack (a specific closure). On the other hand, logs are valuable when you want to see the chronology of what's happening, specially when there are async or parallel things going on
I've seen developers bang away at a problem for days and not be able to describe how it "works" in a code review. They just tried every crazy thing they could think of until it eventually returned results that looked correct.
Don't be that kind of developer. For one, all the other developers are going to make you maintain that code for the rest of your life. Two, never be too proud to ask for help. We can't know how to solve every problem on our own. It's not possible.
I agree, understanding certainly can't be emphasised enough - I believe that a good programmer must always be able to understand enough to "mentally execute" - manually stepping through code, possibly with pencil and paper or a whiteboard, and making sense of the results at each step.
There are some who argue against this, and their argument is effectively "if the machine can do it for you, why should you need to know how to do it mentally" --- it fails miserably when debugging, precisely the time when the machine can't do it.
I've heard it phrased thus: "If you don't understand precisely what you need to do, what makes you think you can tell a computer how to do it?"
I think a pencil and paper are a programmers best friend. I find it helps me work through complex logic easier when I'm debugging. I find it helps me write cleaner code when I'm tasked with something challenging. I keep a notebook next to me when coding or debugging. I try to write down my thought process when writing new code. When debugging, I write the recreate steps (if there aren't any already), and any insteresting notes about the bug while I'm tracking it down. I write down anything I think the 'fix' may impact (I also email the team about those). Basically anything and everything.
I have a similar quip that not many in my office find popular. Basically, if you can't manually test something, what makes you think you can automated the test?
But how can I mentally execute code if the whole point of the code is to interact with a third-party OS facility or application that's closed-source and therefore opaque? In that case, my mental execution of the code will depend on a mental model of the third-party component that's probably incomplete. Might as well just let the machine run the real thing.
It does get a little more difficult when execution flows into code you didn't write, but you can still validate your post/pre-conditions at those boundaries.
> In that case, my mental execution of the code will depend on a mental model of the third-party component that's probably incomplete.
That's the whole reason why you would want execute the code mentally. If you don't execute the code mentally, you might not realize that your mental model of the third-party component was incomplete to begin with. Mentally executing code is a great way to test your own assumptions.
Your mental model of the code you're reading is probably incomplete too. Just be aware of where you're less confident and then you can use different mechanisms to build that confidence when it's the most logical thing to do (running a minimal test case, or log a function return value, perhaps).
This scales up to about 10-15 lines of code before it becomes impractical in most modern imperative languages. Functional languages due to their higher level constructs get better mileage out of the technique but there is still a small practical limit to the technique.
It also doesn't work if your team isn't co-located.
> For one, all the other developers are going to make you maintain that code for the rest of your life.
Ah, if only that were the case. In reality, developers move between jobs, and bad code always becomes someone else's responsibility eventually.
Instead, it's better to understand that as a professional you should be - if not proud of your results - proud of making the best of a bad situation for whoever will inherit your codebase, be that your future self or a colleague.
Also a couple of tips for the beginning programmer:
* You've almost certainly not found a bug in the compiler or libraries or language runtime (somewhat more possible if you're using alpha, beta, or dev versions of something).
* Your CPU, memory, or other hardware are not defective.
* You are not experiencing cosmic rays flipping bits randomly in your data.
* The problem is most likely to be a mistake in your code.
I have worked with programmers who are quick to jump at the most unlikely explanations for bugs, and it's a very timewasting way to work.
> * You've almost certainly not found a bug in the compiler or libraries or language runtime (somewhat more possible if you're using alpha, beta, or dev versions of something).
This should be in bold type on every page of Stackoverflow.
While it's great to see more attention brought to debugging, some of this is just (for lack of a better word) insane. e.g.:
For searching in space, most programmers have done binary-search with commented code: comment out or bypass half of your codebase. Do that recursively until you nail down where (at which file, which function, which lines) the bug lives.
No, most programmers actually haven't done this for the simple reason that it's highly unlikely to work: most codebases with half of their functionality removed don't actually function. That this kind of lunacy is being asserted as authoritative is galling enough, but it gets worse:
When binary search doesn't work, you need more creative approaches. Consider brainstorming to enumerate even the wildest possibilities: for instance, maybe the bug is in some external resource like a library or a remote service; maybe there is version mismatch of libraries; maybe your tool (e.g. IDE) has a problem; maybe it is bit flipping in the hard disk; maybe the date and time library is sensitive to your computer's settings, etc.
This is advocating debugging by superstition, and it represents toxic thinking. When we have defects in software, we need to be strictly empirical in approach:
1. Make observations.
2. Think of interesting questions.
3. Formulate a hypothesis.
4. Develop a testable prediction from the hypothesis.
5. Gather data to test the prediction.
6. Refine, alter or expand the hypothesis.
7. Go to step 4 until defect has been driven to root cause(s)
If this sounds familiar, it is because it is the scientific method[1] to which we can credit much of modern knowledge and civilization. As to how this specifically relates to debugging, I touched on this in my recent DockerCon presentation[2][3]. Bringing attention to debugging is terrific -- but advocating superstition as a methodology is anathema to true understanding.
I listened to your DockerCon talk on debugging. So in a server that can handle multiple concurrent requests (e.g. a web application server), should each unhandled exception abort the whole process (after delivering something like an HTTP 500 to the client) so the developer can do postmortem debugging on a core dump? That would cause all other in-progress requests to be aborted too, unlike simply logging the exception and moving on. But then again, maybe that would increase the motivation to root-cause every failure.
It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse.
To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released.
An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.
Thanks. Now I gotta figure out the best way to do that in Python. Most Python web frameworks and WSGI servers catch all exceptions and keep the process going.
The solution to this problem is simply process isolate, handle each request in a own process and crashing one won't affect the others. Of cause you won't get away with OS level processes.
If it interests you look at Erlang's OTP, it solved this problem years ago and works very well with that.
Doesn't all that stuff you call 'toxic' actually fit in step 5? Lets not get hyperbolic about it. Backing out changes; setting breakpoints and interval-halving through a procedure to find where a value goes wrong; comparing library versions with working build is all good practice.
To be clear: the toxic bit here is the idea that you start with that first. Everything you describe is entirely appropriate when you have a hypothesis in hand, and you are using it as a technique to test it -- but I (strenuously) object to the idea that problems should be debugged by first randomly permuting the codebase.
Binary search is not random permutation. The hypothesis is "problem lies in this half of the codebase". Hey, if your test passes after commenting out some stuff, great.
And if that hypothesis is based on data, great. And if there is no other alternative, fine -- but this technique should be viewed as a last resort (and an unusual one), not a first step.
Wait a minute. First of all, the data is simply "there is a reproducible bug in the program". The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. If you have a better idea of the location, then by all means.
Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away.
Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately.
I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you.
Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.
Part of the scientific method is that you generate hypotheses based on the data you have, not that you generate random hypotheses. It's a directed way of thinking.
> The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search.
In your example, you used this as an assumption, not a hypothesis. And it's not quite right: the assumption you made is that the _presence_ of a small portion of code is responsible for the defect. That's very different than saying that the code is more broadly responsible for the defect. In my experience, very few bugs are caused by the mere presence of some code.
> When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away.
Those are fine solutions for those very specific, very simple problems. Given the problem space, the assumption that the error is caused directly by the presence of some input is well-founded.
Sorry, I'm kind of confused here. Why is the approach I'm defending considered random and not based on data? Is the generation of random hypotheses in general considered unscientific? What about fuzz testing or pharmaceutical R&D? What is the precise difference between hypotheses and assumptions in the context of the scientific method? What is the difference between the presence of a small portion of code being responsible and the code being more broadly responsible? Why the emphasis on presence?
Because it seems you're making an assumption about the problem being reproducible and identifiable by running half the code. I've never found this to be the case. Ever. Its a strange idea. Where does this come from? This isn't a hypothesis based on data, it is an assumption. And a bizarre one.
I can. It's long and tedious, because a lot of it is very simple things you learn very early on writing and maintaining software.
You seem to be arguing that your example demonstrates the fact that you can localize a bug in the code with binary search; specifically that the bug must exist in either a() or b(), and that running them independently is both possible and will determine the single location of the bug.
This is not true. It is only true in the case that one assumes bugs must have single locations, and that those locations can be found with binary search over what amount to incomplete programs. In other words, you're assuming the prior, begging the question, etc.
It's tempting to say "real code isn't like this contrived example" but in fact this contrived example is the best possible demonstration of why blind binary search is a poor strategy. Let us say the 'bug' exists in a(). You seem to be assuming this is the necessary consequence:
main () { a(); b(); } -- segfaults
main () { a(); /* b(); / } -- segfaults
main () { / a(); / b(); } -- doesn't segfault
But this isn't the only possibility. With the bug existing in a(), you could also see this result:
main () { a(); b(); } -- segfaults
main () { a(); / b(); / } -- segfaults
main () { / a(); / b(); } -- segfaults
You would expect this in situations where b() uses data structures created by a().
We may also see this result, still assuming the bug exists in a():
main () { a(); b(); } -- segfaults
main () { a(); / b(); / } -- doesn't segfault
main () { / a(); / b(); } -- segfaults
If above we learned nothing, here we've actually got a falsehood - our binary search has localized the bug to b(), but it actually exists in a()!
So, in practice, binary search fails to localize a bug in a(). All of these situations can be created by having a() write a global which is relied upon by b() - a() may write to a protected area, b() may have a default value, a() may write nonsense, or pass an integer where b() expects an address - none of this is particularly exotic, they're all the sort of things you get every day when debugging segfaults.
We might now delve into a competing series of contrived examples of a() and b() and argue about their relative prevalence in the world (which none of us are capable of knowing), because if some case is particularly rare, it may make binary search very slightly better than flipping a coin in this case.
Instead, I will point out that this is once again* assuming the prior, and that we have these simple (if shockingly annoying) facts:
1) side effects exist
2) in the presence of side-effects, binary search cannot predict the location of a bug.
And it follows that in the sense that a "scientific" hypothesis has predictive power, then binary search over the codebase is basically the homeopathy of debugging.
If you comment out b, and it segfaults, the segfault is caused by some code in a. Otherwise, the segfault is caused by some code in b. The root cause of the segfault in b may be found in a, but even looking at the stack trace from a debugger will still point you at b. This is nevertheless helpful information.
Further, there's a misunderstanding about the method. You do a binary search on the "percentage" of code viewed as an instruction stream that is allowed to execute, starting from the beginning. I understand that a can have a different length from b. Given more code, you can also search based on timestamps. You never comment out a and allow b to execute without it, this is nonsense. If that's what you thought I meant, I can understand comments about not actually being able to program. I did not explain this clearly, but only because I had forgotten that it was necessary to explain this clearly, because some people actually don't know how to program, even on HN.
And the assertion that // works as a strategy. There's only two things you could possibly comment out, so really the example is not so much "explained poorly" as "conceived poorly". Don't shoot the messenger - it's your example.
As for the further explanation here, allowing a debugger makes the search irrelevant - there's only one piece of information the binary search can tell you: which function causes the segfault, but as you observe, that information's available from the debugger already. And no, it doesn't matter if you reframe things as "a percentage of an instruction stream".
The objection to binary search is not "binary search in the codebase can never tell you anything," it's that it's a method that can't -- not "doesn't" but "can't" -- produce a non-trivial testable hypothesis. Does this mean you can't ever use it? You may have no choice. But it's a terrible place to start, and most of the information it offers is available through other means, often with more precision.
Understanding is the key to everything in software development. In particular, understanding both what you are trying to do and the tools (including language) you are using is necessary for writing code that works.
The former is true even if you are doing exploratory development: in that case, you need to understand what has been found so far and what the next iteration is intended to investigate.
Totally agree with that. In my previous company we were always working with dumps for asserts/crashes and when I started working in my current company I was confused that they mainly use logs.
Now I can't go back to debugging without logs.
Using dumps is like looking at the consequences, looking at logs is looking at the causes.
If often found some bugs in a reporting software, that showed the people wrong numbers for years.
Or if something needs to be re-implemented, say for performance reasons, you get the algorithms and notice that the old version wasn't only slower but also wrong.
When this happens, I start to question if the people really "use" the software in the first place. Or if they just get some wrong sens of safety from it, which calms them, but the underlying system is mostly random and they could simply make up their own numbers.
If debugging is about understanding, then it will be good to use a language that lets you embed the knowledge in the coding. Knowledge should be easier to read from the code leading to a faster understanding, and more bugs are catch by the compiler instead by the end user.
This is a great page (with a video) about what I mean
It's an interesting and helpful article, the downside is that it perpetuates the use of the non-technical language of "bugs" and "debugging." Sure it's accepted jargon, but like most jargon "bug" doesn't really convey much of anything beyond a person's attitude. "Bug" is as unscientific as "weed"...a bug may be the wrong color text in some HTML or as in its origin myth, something preventable with a can of Raid.
Fault/Error/Failure provides a better language for talking about systems and their processes. It provides a diversity of solutions from changing the source code, to handling an error, to just letting it crash...the last is impossible to cast as a solution in the nomenclature of bugs.
There are times in which there the ambiguity of "bug" matches the necessary ambiguity of the context...just as there are times when describing the sun moving across the sky is useful. But the geocentric cosmological model breaks down when we're interested in predicting the location of Jupiter in the night sky.
Just add asserts to your code. Make an assert for any assumption you make no matter how trivial or unlikely to be broken. And don't disable the assert in release versions.
That can, as a very conservative estimate, cut down time spent debugging by 50%.
Life is too short to spend it debugging undefined behaviour.
The second step is using a good tool that helps you make sense of those logs, and includes good debugging information with the logs. A tool like Rollbar (disclaimer: I almost work for Rollbar) makes it super easy to analyze patterns in your errors and logging, find out who experienced the error, and to hear about the bugs before your customer, who may have become used to them.
Anecdote from my previous employer: we had a terrible piece of legacy software that regularly had modal pop-ups warning of errors this or that that doubled the amount of time our clients took to do stuff. They were so used to it that instead of reporting the error, they just dismissed them.
Tangential to the topic: The git bisect image has "good" and "bad" backwards, showing the solving of a bug instead of the production of one. I would hope a commit in between would clearly state it contains the bugfix.
While understanding if of course important, sometimes it's possible to just "zen" the bug. After working with the same codebase for long enough I've been able to see behavior and think, "That's probably down in the login handling code..." and then start there. I don't necessarily have to understand the login code, but it's gnarly, and I've seen similar bugs before from it. I'm right more often then I would expect. When the zen fails it's off to the methods mentioned in the article.
Usually breakpoints and stepping through code solves things pretty easily but some bugs can be real stumpers.
Here's some of the tools I tend to use:
1. What has changed? What could have caused this bug to appear?
2. Verify fundamentals: Is it actually running the code I think it does. If I change something that should change something, does it change? If I break something does it break?
3. Verify Input: Make sure it's valid
4. Verify output
5. Timing issues?
6. Slowly remove code "around" bug until it works again
7. Or start back at something you know works and add back code until it breaks.
8. Break out the functionality into an isolated environment so you can test it separately so you'll know if the problem is within the feature or a side-effect of something else
9. If your code makes it too complicated to see and find the bug refactor until it isn't.
10. Go to sleep / Go on a walk. Let your subconscious process it
and finally, lot's of bugs are made self evident by well structured and refactored code, so don't get too busy working on the symptoms rather than the root cause.