STAT240 - Robust and Nonparametric Statistics

Instructor: Jacob Steinhardt (jsteinhardt@berkeley)
Lectures: T/Th 3:30-5 (Zoom)
Office Hours: F 2-3 (Zoom)

TA: Serena Wang (serenalwang@berkeley)
Office Hours: Th 2:30-3:30 (Zoom)

Syllabus: link

IMPORTANT: If you plan to take the class, sign up here to be added to the class mailing list. We will use this to communicate instead of bCourses.

Prerequisites

No formal requirements, but this class will be fast-paced and assume mathematical maturity.

Lecture Notes

Updated periodically: link (last update 04/01/2021, through Lecture 20)

Please e-mail typos/corrections to me (jsteinhardt@berkeley with a dot edu at the end).

Problem Sets

Problem Set 1 (due February 9th before class [note extension]) tex source
Problem Set 2 (due February 23rd before class) tex source
Problem Set 3 (due March 9th before class) tex source
Problem Set 4 (due April 6th before class) tex source

Schedule

Feedback form for lectures

Lecture 1: Overview and 1D Robust Estimation (video) (tablet)
Lecture 2: Minimum Distance Functionals and Resilience (video) (tablet)
Lecture 3: Concentration Inequalities (video) (tablet)
Lecture 4: Bounding Suprema via Concentration Inequalities (video) (tablet)
Lecture 5: Finite-Sample Analysis via Generalized KS Distance (video) (tablet)
Lecture 6: Finite-Sample Analysis via Expanding the Destination Set (video) (tablet)
Lecture 7: Truncated Moments and Ledoux-Talagrand (video) (tablet)
Lecture 8: Efficient Algorithms: Projecting onto Maximum Eigenvector (video) (tablet)
Lecture 9: Approximation Oracles and Grothendieck’s Inequality (video) (tablet)
Lecture 10: Resilience Beyond Mean Estimation (video) (tablet)
Lecture 11: Resilience For Linear Regression (video) (tablet)
Lecture 12: Efficient Algorithms for Robust Linear Regression (video) (tablet)
Lecture 13: Resilience for Wasserstein Distances (video) (tablet)
Lecture 14: Wasserstein Resilience for Moment Estimation and Linear Regression (video) (tablet)
Lecture 15: Model Mis-specification in Generalized Linear Models (video) (slides) (Python notebook)
Lecture 16: Robust Inference via the Bootstrap (video) (slides) (Python notebook)
Lecture 17: Robust Inference via Partial Specification (video) (tablet)
Lecture 18: Partial Specification and Agnostic Clustering (video) (tablet)
Lecture 19: Nonparametric Regression I (video) (tablet)
Lecture 20: Nonparametric Regression II (video) (tablet)
Lecture 21: Domain Adaptation under Covariate Shift (video) (tablet)
Lecture 22: Doubly-Robust Estimators and Semi-Parametric Estimation (video) (tablet)
Lecture 23: Neural Networks and Pre-training (video) (slides) (Python notebook) Lecture 24: Robustness of Neural Networks (video) (slides) Lecture 25: Scaling Laws for Neural Networks (video) (slides) Lecture 26: Nonparametric Regression III: Generalization and Mercer’s Theorem (video) (tablet) Lecture 27: Nonparametric Regression IV: Random Features and NTK (video) (tablet) Lecture 28: Double Descent (video) (slides)

Supplementary Reading List

These are not necessary to follow the class, but may be interesting further reading.

Jerry Li taught a class related to the first 14 lectures.

Robust Learning: Information Theory and Algorithms (Jacob Steinhardt’s thesis)
Concentration of Measure (lecture notes by Terence Tao)
Alternate reference: Concentration Inequalities (notes by Boucheron, Lugosi, and Bousquet)
Generalized Resilience and Robust Statistics (Zhu, Jiao, Steinhardt)
Principled Approaches to Robust Machine Learning and Beyond (Jerry Li’s thesis)
Probability Bounds (John Duchi; contains exposition on Ledoux-Talagrand)
Approximating the Cut-Norm via Grothendieck’s Inequality (Alon and Naor)
Better Agnostic Clustering via Relaxed Tensor Norms (Kothari and Steinhardt)
Ricci curvature of Markov chains on metric spaces (Ollivier; relation between Poincaré inequalities and Markov chain convergence)
Concentration inequalities: A nonasymptotic theory of independence (Boucheron, Lugosi, and Massart; good general survey that contains discussion of Poincaré inequalities)
Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope (Eric Wong and Zico Kolter)
Training Verified Learners with Learned Verifiers (Krishnamurthy Dvijotham et al.)
Semidefinite relaxations for certifying robustness to adversarial examples (Aditi Raghunathan et al.)