This course teaches an overview of modern optimization methods and algorithms for applications in statistical learning, machine learning and data science. In particular, scalability of algorithms to large datasets will be discussed in theory and in implementation. Theory behind these methods (e.g., optimality conditions and duality theory) will be discussed as well as how to choose the right optimization methods for different statistical learning applications.
By the end of the course, the student must be able to:
• Evaluate the most important algorithms, function classes, and algorithm convergence guarantees.
• Formulate scalable and accurate implementations of the most important optimization algorithms for data science using Python.
• Characterize trade-offs between time, data and accuracy, for machine learning methods.
On the practical side, application of the considered algorithms on real-world databases will be investigated.
Basic knowledge of Linear Algebra and Calculus in one and two variables.
The course is structured into lectures in class, in which the context and methodologies are explained, and computer lab sessions (using Python software), in which the students apply the methodologies to real-world data sets coming from various fields (business analytics, news, medical diagnosis etc).
The exam consists in two parts:
- Oral discussion about applied assignments and case studies (50% of the final grade).
Students may work in small groups or individually.
- Final oral exam (50% of the final grade).
Specifically, the course will cover the following topics:
- Optimality Conditions, Derivative-free Optimization Methods, Gradient Method, Stochastic Gradient Descent Method, Subgradient Method, Netwon and Quasi-Newton Methods, Conjugate Gradient Method, Penalty Function Method.
- Supervised Learning
Support Vector Machines (SVM): Soft and hard Margin Classifiers. Quadratic programming (QP) formulation of the soft/hard maximum margin separators. Kernels methods. Karush-Kuhn-Tucker (KKT) conditions. Dual formulation of the primal QP problem. Wolfe duality theory for QP. Multiclass SVM problems.
- Optimal Decision Trees.
Practical use of learning algorithms. Comparing learning algorithms from the optimization point of view. Use of standard software (Python).
The course material will be provided by means of the e-learning platform of the University of Bergamo.
If the teaching activity will be mixed or in remote mode, changes can be done compared to what stated in the syllabus, to make the course and the exams available also in these modalities.
