Numerical Differentiation
The gradient method uses information from the gradient to determine which direction to follow. This section describes what a gradient is and its characteristics, beginning with a "derivative."
Derivative
For example, let's assume that you ran 2 km in 10 minutes from the start of a full marathon. You can calculate the speed as 2 / 10 = 0.2 [km/minute]. You ran at a speed of 0.2 km per minute.
In this example, we calculated how much the "running distance" changed over "time." Strictly speaking, this calculation indicates the "average speed" for 10 minutes because you ran 2 km in 10 minutes. A derivative indicates the amount of change at "a certain moment." Therefore, by minimizing the time of 10 minutes (the distance in the last 1 minute, the distance in the last 1 second, the distance in the last 0.1 seconds, and so on), you can obtain the amount of change at a certain moment (instantaneous speed...