STAT 28000/CAAM 28000. Optimization

Department of Statistics
University of Chicago
Spring 2018

This is an introductory course on optimization that will cover the rudiments of unconstrained and constrained optimization of a real-valued multivariate function. The focus is on the settings where this function is, respectively, linear, quadratic, convex, or differentiable. Time permitting, topics such as nonsmooth, integer, vector, and dynamic optimization may be briefly addressed. Materials will include basic duality theory, optimality conditions, and intractability results, as well as algorithms and applications.

Announcements

05/23/18: Review session on Thu, Apr 24, 3:30–4:50, Eckhart 133.

05/23/18: Quiz II on Tue, May 29, 3:30–4:50pm, Eckhart 133.

05/23/18: Lecture notes 17 posted. All lecture notes + appendix combined into a single file.

05/17/18: Lecture notes 16 posted.

05/17/18: Office hours on Mon, May 21, 3:00–5:00, Jones 122B.

05/15/18: Lecture notes 15 posted.

05/10/18: Lecture notes 14 and Homework 4 posted.

05/08/18: Lecture notes 13 posted.

05/03/18: Office hours on Wed, May 9, 1:30–3:30, Jones 122B.

05/03/18: Lecture notes 12 posted.

05/01/18: Lecture notes 11 and Homework 3 posted.

04/19/18: Lecture notes 10 posted.

04/17/18: Lecture notes 9 posted.

04/17/18: Office hours on Wed, Apr 18, 1:30–3:30, Jones 122B.

04/17/18: Review session on Tue, Apr 24, 3:30–4:50, Eckhart 133.

04/16/18: Quiz I on Thu, Apr 26, 3:30–4:50pm, Eckhart 133.

04/13/18: Lecture notes 8 posted.

04/10/18: Lecture notes 7 and Homework 2 posted.

04/06/18: Lecture notes 6 posted.

04/06/18: Office hours on Mon, Apr 9, 2:30–4:30pm.

04/05/18: Lecture notes 5 posted.

04/04/18: Lecture notes 4 posted.

03/29/18: Lecture notes 3 and Homework 1 posted.

03/29/18: Make-up lecture 2 on Wed, Apr 4, 5:00–7:00pm in Saieh 146.

03/29/18: Lecture notes 2 posted.

03/27/18: Make-up lecture 1 on Wed, Mar 28, 5:00–7:00pm in Kent 107.

03/27/18: Lecture notes 1 posted (see email announcement for url).

03/27/18: Check back regularly for announcements.

Lectures

Location: Eckhart Hall, Room 133.

Times: Tue & Thu, 3:30–4:50pm

Course staff

Instructor: Lek-Heng Lim
Office: Jones 122B
lekheng(at)galton.uchicago.edu
Tel: (773) 702-4263
Office hours: Two-hour session the day before problem set is due, Jones 122B

Course Assistant I: Minzhe Wang
Office: Jones 203/204
minzhew(at)uchicago.edu
Office hours: Mon, 6:00–7:00pm, in Jones 304

Course Assistant II: Ken Sze-Wai Wong
Office: Jones 203/204
kenwong(at)uchicago.edu
Office hours: Wed, 7:00–8:00pm, in Jones 308

Grader: Xinyi Ge
xinyige(at)uchicago.edu

Syllabus

Optimization of a univariate real-valued function
Multivariate real-valued functions
Higher-order derivatives of multivariate functions
Multivariate Taylor expansion with remainder in mean-value form and integral form
Unconstrained optimization
Newton, quasi-Newton, conjugate gradient, and steepest descent methods
Constrained optimization
Implicit function theorem and Lagrange multipliers
KKT conditions
Nonlinear optimization
Convex sets and convex functions
Convex optimization
Newton and steepest descent methods revisited
Methods of penalty function, augmented Lagrangian, and barrier function
Linear programming
Basic duality theory
Brief overview of nonsmooth, integer, vector, and dynamic optimization

Problem Sets

Collaborations are permitted but you will need to write up your own solutions and declare your collaborators. The problem sets are designed to get progressively more difficult. You will get at least six days for each problem set.

You are required to implement your own programs for problems that require some amount of simple coding (using Matlab, Mathematica, R, or SciPy).

Problem Set 4 (posted: May 10, due: May 22)

Problem Set 3 (posted: May 1, due: May 10)

Problem Set 2 (posted: Apr 10, due: Apr 19)

Problem Set 1 (posted: Mar 29, due: Apr 10)

Bug report on the problem sets: lekheng(at)galton.uchicago.edu

Supplementary materials

Course homepages from Spring 2017, Spring 2016, Spring 2015.

These are slightly more advanced versions of this course, intended primarily for graduate students: Stat 31015 Winter 2015, Stat 31020 Winter 2009–2012.

Implicit function theorems and Lagrange multipliers.

Grades and quizzes

Grade composition: 60% Problem Sets, 40% Quizzes.

Quizzes: Quiz I on Thu, Apr 26, 3:30–4:50pm, Eckhart 133. Quiz II on Tue, May 29, 3:30–4:50pm, Eckhart 133. Closed book, closed notes, no cheat sheet.

References

We will not use any specific textbook but will use selected material from the following references, all of which would be accessible to undergraduates.

You may download all these books online from an UChicago IP address or via ProxyIt! if you are off-campus.

E. Çinlar and R. J. Vanderbei, Real and Convex Analysis, Springer, 2013.

J. E. Dennis and R. B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, SIAM, 1996.

G. Hurlbert, Linear Optimization: The Simplex Workbook, Springer 2010.

J. Nocedal and S. J. Wright, Numerical Optimization, 2nd Ed, Springer, 2006.

P. Pedregal, Introduction to Optimization, Springer, 2004.