Gradient Descent

To fit a model you need to minimize a loss — a number that says how wrong it is. Gradient descent does this by computing the gradient (the direction of steepest increase), stepping the opposite way, and repeating. The step size is the learning rate: too large and it overshoots or diverges, too small and it crawls.

Stochastic gradient descent (SGD) estimates the gradient from small random batches, which is what makes training on huge datasets tractable and gives modern deep learning its scale. The same machinery solves convex problems like mean-variance portfolio weights, where the optimum is unique and descent converges cleanly.

Why an agent’s operator should care: nearly every model in the stack — the return predictor, the regime classifier, the sizing policy — is fit by some flavor of gradient descent, and its failure modes (diverging loss, overfit minima, sensitivity to learning rate) propagate into strategy behavior. Sound training discipline, validated with proper cross-validation, is what separates a robust agent from one that memorized noise.