Some variants of gradient dominance conditions motivated by LQR direct policy optimization, and linear neural net feedback

ABSTRACT:

Solutions of optimization problems, including policy optimization in reinforcement learning, typically rely upon some variant of gradient descent. There has been much recent work in the machine learning, control, and optimization communities applying the Polyak-Łojasiewicz Inequality (PŁI) to such problems in order to establish an exponential rate of convergence  (a.k.a. “linear convergence” in the local-iteration language of numerical analysis) of loss functions to their minima under the gradient flow. Often, as is the case of policy iteration for the continuous-time LQR problem, this rate vanishes for large initial conditions, resulting in a mixed globally linear / locally exponential behavior. This is in sharp contrast with the discrete-time LQR problem, where there is global exponential convergence. That gap between CT and DT behaviors motivates the search for various generalized PŁI-like conditions, and this talk will address that topic. Moreover, these generalizations are key to understanding the transient and asymptotic effects of errors in the estimation of the gradient, errors which might arise from adversarial attacks, wrong evaluation by an oracle, early stopping of a simulation, inaccurate and very approximate digital twins, stochastic computations (algorithm "reproducibility"), or learning by sampling from limited data. We will describe an “input to state stability” (ISS) analysis of this issue. We will also discuss convergence and PŁI-like properties of “linear feedforward neural networks” in feedback control.  (Joint work with A.C.B. de Oliveira, L. Cui, Z.P. Jiang, and M. Siami).