Mathematical Optimization for Data Science Group, Saarland University

Welcome to the homepage of the

Mathematical Optimization for Data Science Group

Department of Mathematics and Computer Science, Saarland University, Germany

Home
Teaching
Research
Publications
Job Offers

Saarland
University

Mathematics
and

Computer Science

Automatic Differentiation of Optimization Algorithms with Time-Varying Updates

S. Mehmood and P. Ochs

Abstract:
Numerous Optimization Algorithms have a time-varying update rule thanks to, for instance, a changing step size, momentum parameter or, Hessian approximation. In this paper, we apply unrolled or automatic differentiation to a time-varying iterative process and provide convergence (rate) guarantees for the resulting derivative iterates. We adapt these convergence results and apply them to proximal gradient descent with variable step size and FISTA when solving partly smooth problems. We confirm our findings numerically by solving l1 and l2-regularized linear and logisitc regression respectively. Our theoretical and numerical results show that the convergence rate of the algorithm is reflected in its derivative iterates.



pdf	Bibtex	arXiv

Latest update: 24.10.2024

Citation:
S. Mehmood, P. Ochs:
Automatic Differentiation of Optimization Algorithms with Time-Varying Updates.

Technical Report, ArXiv e-prints, arXiv:2410.15923, 2024.

Bibtex:
@techreport{MO24, title = {Automatic Differentiation of Optimization Algorithms with Time-Varying Updates}, author = {S. Mehmood and P. Ochs}, year = {2024}, journal = {ArXiv e-prints, arXiv:2410.15923}, }

Impressum
Datenschutz