A superb tool for tracing and profiling is the powerful Linux Tracing Toolkit next generation (LTTng) toolset, a Linux Foundation project. LTTng allows you to trace both userspace (applications) and/or the kernel code paths in minute detail. This can tremendously aid you in understanding where performance bottlenecks occur, as well as aiding you in understanding the overall code flow and thus in learning about how the code actually performs its tasks.
In order to learn how to install and use it, I refer you to its very good documentation here: https://lttng.org/docs (try https://lttng.org/download/ for installation for common Linux distributions). It is also highly recommended that you install the Trace Compass GUI: https://www.eclipse.org/tracecompass/. It provides an excellent GUI for examining and interpreting LTTng's output.
As an example (I can't resist!), here's a screenshot of a capture by LTTng being "visualized" by the superb Trace Compass GUI. Here, I show a couple of hardware interrupts (IRQ lines 1 and 130, the interrupt lines for the i8042 and Wi-Fi chipset, respectively, on my native x86_64 system.):
The pink color in the upper part of the preceding screenshot represents the occurrence of a hardware interrupt. Underneath that, in the IRQ vs Time tab (it's only partially visible), the interrupt distribution is seen. (In the distribution graph, the y axis is the time taken; interestingly, the network interrupt handler – in red – seems to take very little time, the i8042 keyboard/mouse controller chip's handler – in blue – takes more time, even exceeding 200 microseconds!)
