ps, lsof, strace, and /proc are all valuable tools when troubleshooting under fire. They can display something so obvious that you'd never have thought to look for it in a million years, and they can also work through certain tasks quicker than you'll be able to do so yourself.
strace is probably the most often overlooked of these tools, and it's easily one of the most awkward to learn well, but it should always be at the back of your mind for program debugging.
Personally, I generally forget about strace quite a lot, that is until someone wanders over and asks me why I'm trying to debug something in the most awkward way possible, before reminding me that strace is a thing that exists.
