PUB - Publikationen an der Universität Bielefeld

These lecture notes cover theory and algorithms for optimization from an application perspective. With respect to theory we cover basic definitions of optimization problems and their solutions, necessary and sufficient conditions of optimality, convex problems and optimality under convexity, Lagrange- and Wolfe dual forms, as well as Karush-Kuhn-Tucker conditions of optimality. With respect to algorithms we cover analytical optimization; numeric optimization, especially (conjugate) gradient descent, (pseudo-)Newton, trust region, log-barrier, penalty, and projection methods; probabilistic optimization, especially expectation maximization and max-product; linear and quadratic programming; and heuristics, especially the Nelder-Mead algorithm, CMA-ES, Bayesian optimization, hill climbing, simulated annealing, tabu search, branch-and-cut, and ant colony optimization. As such, this document provides a comprehensive overview of the most important optimization techniques for a wide range of application domains as well as their theoretical foundations.