Data-Driven Optimization for Machine Learning Applications

Content:

1. Introduction – Motivation, Data-driven versus Model-driven appraoch, importance of data-driven optimization; overview of optimization problems arising in machine learning applications;

2. Preiminaries – linear algebra; convex sets convex functions; gradient, sub-gradient, hessian matrix;

3. Programming basics (Python, R, Matlab); data loading and preprocessing;

4. Unconstrained optimization for machine learning: regularization-meaning and relevance; regression problems; neural networks and back-propagation of errors; optimization methods for deep learning ;

5. Uncostrained Optimiztion Algorithms; 5A: First-order algorithms – gradient descent, accelerated gradient descent, stochastic gradient descent, conjugate gradient methods, coordinate descent; R and Python implementations; sub-gradient methods (optional); 5B. Second-order algorithms: The Newton Method; quasi-Newton methods; LBFGS; R and Python implementations;

6. Constrained Optimization Methods for Machine Learning – the interior point method; face-recongintion with supprot vector machine using Python, Scikit-Learn and OpenCV ;Matrix factorization methods for pattern recognition- SVD, PCA, non-negative matrix factorization (NMF); Matlab and Python Scikit-Learn implementations; Proximal-Point Algorithms: proximal gradient methods; alternating direction of multupliers (ADMM);

7. Bayesian Optimization methods for Machine Learning;

8. Optimization algorithms in Deep Learning Tools TensorFlow, Kerays, pyTorch