Video Lectures

Description In this first lecture, Professor Strang introduces the linear algebra principles critical for understanding the content of the course.  In particular, matrix-vector multiplication \(Ax\) and the column space of a matrix and the rank. Summary Independent columns = basis for the column space Rank = number of independent columns \(A = CR\) leads to: Row rank equals column rank Related section in textbook: I.1 Instructor: Prof. Gilbert Strang

Description Multiplying and factoring matrices are the topics of this lecture. Professor Strang reviews multiplying columns by rows: \(AB =\) sum of rank one matrices. He also introduces the five most important factorizations. Summary Multiply columns by rows: \(AB =\) sum of rank one matrices Five great factorizations: \(A = LU\) from elimination \(A = QR\) from orthogonalization (Gram-Schmidt) \(S = Q \Lambda Q^{\mathtt{T}}\) from eigenvectors of a symmetric matrix \(S\) \(A = X \Lambda X^{-1}\) diagonalizes \(A\) by the eigenvector matrix \(X\) \(A = U \Sigma V^{\mathtt{T}} =\) (orthogonal)(diagonal)(orthogonal) = Singular Value Decomposition Related section in textbook: I.2 Instructor: Prof. Gilbert Strang

Description This lecture focuses on orthogonal matrices and subspaces. Professor Strang reviews the four fundamental subspaces: column space C(A), row space C(A T), nullspace N(A), left nullspace N(A T). Summary Examples: Rotations Reflections Hadamard matrices Haar wavelets Discrete Fourier Transform (DFT) Complex inner product Related section in textbook: I.5 Instructor: Prof. Gilbert Strang

Description Professor Strang begins this lecture talking about eigenvectors and eigenvalues and why they are useful. Then he moves to a discussion of symmetric matrices, in particular, positive definite matrices. Summary \(Ax =\) eigenvalue times \(x\) \(A^2x =\) (eigenvalue)\(^2\) times \(x\) Write other vectors as combinations of eigenvectors Similar matrix \(B = M^{-1}AM\) has the same eigenvalues as \(A\) Related section in textbook: I.6 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang continues reviewing key matrices, such as positive definite and semidefinite matrices. This lecture concludes his review of the highlights of linear algebra. Summary All eigenvalues of S are positive. Energy x T Sx is positive for x \(\neq 0\). All pivots are positive S = A T A with independent columns in A. All leading determinants are positive 5 EQUIVALENT TESTS. Second derivative matrix is positive definite at a minimum point. Semidefinite allows zero evalues/energy/pivots/determinants. Related section in textbook: I.7 Instructor: Prof. Gilbert Strang

Description Singular Value Decomposition (SVD) is the primary topic of this lecture. Professor Strang explains and illustrates how the SVD separates a matrix into rank one pieces, and that those pieces come in order of importance. Summary Columns of V are orthonormal eigenvectors of A T A. A v = \(\sigma\) u gives orthonormal eigenvectors u of AA T. \(\sigma^2 =\) eigenvalue of A T A = eigenvalue of AA T \( \neq\) 0 A = (rotation)(stretching)(rotation) \(U\Sigma\) V T for every A Related section in textbook: I.8 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang reviews Principal Component Analysis (PCA), which is a major tool in understanding a matrix of data. In particular, he focuses on the Eckart-Young low rank approximation theorem. Summary \(A_k = \sigma_1 u_1 v^{\mathtt{T}}_1 + \cdots + \sigma_k u_k v^{\mathtt{T}}_k\) (larger \(\sigma\)s from \(A = U\Sigma V^{\mathtt{T}}\)) The norm of \(A - A_k\) is below the norm of all other \(A - B_k\). Frobenius norm squared = sum of squares of all entries The idea of Principal Component Analysis (PCA) Related section in textbook: I.9 Instructor: Prof. Gilbert Strang

Description A norm is a way to measure the size of a vector, a matrix, a tensor, or a function. Professor Strang reviews a variety of norms that are important to understand including S-norms, the nuclear norm, and the Frobenius norm. Summary The \(\ell^1\) and \(\ell^2\) and \(\ell^\infty\) norms of vectors The unit ball of vectors with norm \(\leq\) 1 Matrix norm = largest growth factor = max \( \Vert Ax \Vert / \Vert x \Vert\) Orthogonal matrices have \(\Vert Q \Vert_2 = 1\) and \(\Vert Q \Vert^2_F = n\) Related section in textbook: I.11 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang details the four ways to solve least-squares problems. Solving least-squares problems comes in to play in the many applications that rely on data fitting. Summary Solve \(A^{\mathtt{T}} Ax = A^{\mathtt{T}}b\) to minimize \(\Vert Ax - b \Vert^2\) Gram-Schmidt \(A = QR\) leads to \(x = R^{-1} Q^{\mathtt{T}}b\). The pseudoinverse directly multiplies \(b\) to give \(x\). The best \(x\) is the limit of \((A^{\mathtt{T}}A + \delta I)^{-1} A^{\mathtt{T}}b\) as \(\delta \rightarrow 0\). Related section in textbook: II.2 Instructor: Prof. Gilbert Strang

Description The subject of this lecture is the matrix equation \(Ax = b\). Solving for \(x\) presents a number of challenges that must be addressed when doing computations with large matrices. Summary Large condition number \(\Vert A \Vert \ \Vert A^{-1} \Vert\) \(A\) is ill-conditioned and small errors are amplified. Undetermined case \(m < n\) : typical of deep learning Penalty method regularizes a singular problem. Related chapter in textbook: Introduction to Chapter II Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang revisits the ways to solve least squares problems. In particular, he focuses on the Gram-Schmidt process that finds orthogonal vectors. Summary Picture the shortest \(x\) in \(\ell^1\) and \(\ell^2\) and \(\ell^\infty\) norms The \(\ell^1\) norm gives a sparse solution \(x\). Details of Gram-Schmidt orthogonalization and \(A = QR\) Orthogonal vectors in \(Q\) from independent vectors in \(A\) Related section in textbook: I.11 Instructor: Prof. Gilbert Strang

Description Numerical linear algebra is the subject of this lecture and, in particular, how to compute eigenvalues and singular values. This includes discussion of the Hessenberg matrix, a square matrix that is almost (except for one extra diagonal) triangular. Summary \(QR\) method for eigenvalues: Reverse \(A = QR\) to \(A_1 = RQ\) Then reverse \(A_1 = Q_1R_1\) to \(A_2 = R_1Q_1\): Include shifts \(A\)s become triangular with eigenvalues on the diagonal. Krylov spaces and Krylov iterations Related section in textbook: II.1 Instructor: Prof. Gilbert Strang

Description This lecture focuses on randomized linear algebra, specifically on randomized matrix multiplication. This process is useful when working with very large matrices. Professor Strang introduces and describes the basic steps of randomized computations. Summary Sample a few columns of \(A\) and rows of \(B\) Use probabilities proportional to lengths \(\Vert A_i \Vert \, \Vert B_i \Vert\) See the key ideas of probability: Mean and Variance Mean \(= AB\) (correct) and variance to be minimized Related section in textbook: II.4 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang introduces the concept of low rank matrices. He demonstrates how using the Sherman-Morrison-Woodbury formula is useful to efficiently compute how small changes in a matrix affect its inverse. Summary If \(A\) is changed by a rank-one matrix, so is its inverse. Woodbury-Morrison formula for those changes New data in least squares will produce these changes. Avoid recomputing over again with all data Note: Formula in class is correct in the textbook. Related section in textbook: III.1 Instructor: Prof. Gilbert Strang

Description This lecture is about changes in eigenvalues and changes in singular values. When matrices move, their inverses, their eigenvalues, and their singular values change. Professor Strang explores the resulting formulas. Summary Matrices \(A(t)\) depending on \(t / \)Derivative \(= dA/dt\) The eigenvalues have derivative \(y(dA/dt)x\). \(x\) = eigenvector, \(y\) = eigenvector of transpose of \(A\) Eigenvalues from adding rank-one matrix are interlaced. Related section in textbook: III.1-2 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang reviews how to find the derivatives of inverse and singular values. Later in the lecture, he discusses LASSO optimization, the nuclear norm, matrix completion, and compressed sensing. Summary Derivative of \(A^2\) is \(A(dA/dt)+(dA/dt)A\): NOT \(2A(dA/dt)\). The inverse of \(A\) has derivative \(-A^{-1}(dA/dt)A^{-1}\). Derivative of singular values \(= u(dA/dt)v^{\mathtt{T}} \) Interlacing of eigenvalues / Weyl inequalities Related section in textbook: III.1-2 Instructor: Prof. Gilbert Strang

Description Professor Alex Townsend gives this guest lecture answering the question “Why are there so many low rank matrices that appear in computational math?” Working effectively with low rank matrices is critical in image compression applications. Summary Professor Alex Townsends lecture Why do so many matrices have low effective rank? Sylvester test for rapid decay of singular values Image compression: Rank \(k\) needs only \(2kn\) numbers. Flags give many examples / diagonal lines give high rank. Related section in textbook: III.3 Instructor: Prof. Alex Townsend

Description In this lecture, Professor Strang reviews counting the free parameters in a variety of key matrices. He then moves on to finding saddle points from constraints and Lagrange multipliers. Summary Topic 1: Find \(n^2\) parameters in \(L\) and \(U\), \(Q\) and \(R\), ... Find \((m + n - r)r\) parameters in a matrix of rank \(r\) Topic 2: Find saddle points from constraints and Lagrange multipliers Related section in textbook: III.2 Instructor: Prof. Gilbert Strang

Description Professor Strang continues his discussion of saddle points, which are critical for deep learning applications. Later in the lecture, he reviews the Maxmin Principle, a decision rule used in probability and statistics to optimize outcomes. Summary \(xSx/xx\) has a saddle at eigenvalues between lowest / highest. (Max over all \(k\)-dim spaces) of (Min of \(xSx/xx\)) = evalue Sample mean and expected mean Sample variance and \(k\) th eigenvalue variance Related sections in textbook: III.2 and V.1 Instructor: Prof. Gilbert Strang

Description This lecture continues the focus on probability, which is critical for working with large sets of data. Topics include sample mean, expected mean, sample variance, covariance matrices, Chebyshevs inequality, and Markovs inequality. Summary \(E[x] = m =\) average outcome weighted by probabilities \(E\) uses expected outcomes not actual sample outcomes. \(E[(x - m)^2] = E[x^2] - m^2\) is the variance of \(x\). Markovs inequality Prob[\(x \geq a\)] \(\leq\) mean\(/a\) (when all \(x\)s \(\geq\) 0) Chebyshevs inequality Prob[|\(x\) - mean| \(\geq\) \(a\)] \(\leq\) variance\(/a^2\) Related sections in textbook: V.1, V.3 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang discusses optimization, the fundamental algorithm that goes into deep learning. Later in the lecture he reviews the structure of convolutional neural networks (CNN) used in analyzing visual imagery. Summary Three terms of a Taylor series of \(F\)(\(x\)) : many variables \(x\) Downhill direction decided by first partial derivatives of \(F\) at \(x\) Newtons method uses higher derivatives (Hessian at higher cost). Related sections in textbook: VI.1, VI.4 Instructor: Prof. Gilbert Strang

Description Gradient descent is the most common optimization algorithm in deep learning and machine learning. It only takes into account the first derivative when performing updates on parameters—the stepwise process that moves downhill to reach a local minimum. Summary Gradient descent: Downhill from \(x\) to new \(X = x - s (\partial F / \partial x)\) Excellent example: \(F(x,y) = \frac{1}{2} (x^2 + by^2)\) If \(b\) is small we take a zig-zag path toward (0, 0). Each step multiplies by \((b - 1)/(b + 1)\) Remarkable function: logarithm of determinant of \(X\) Related section in textbook: VI.4 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang explains both momentum-based gradient descent and Nesterovs accelerated gradient descent. Summary Study the zig-zag example: Minimize \(F = \frac{1}{2} (x^2 + by^2)\) Add a momentum term / heavy ball remembers its directions. New point \(k\) + 1 comes from TWO old points \(k\) and \(k\) - 1.1 st order becomes 2 nd order or 1 st order system as in ODEs. Convergence rate improves: 1 - \(b\) to 1 - square root of \(b\) ! Related section in textbook: VI.4 Instructor: Prof. Gilbert Strang

Description This lecture focuses on several topics that are specific parts of optimization. These include linear programming (LP), the max-flow min-cut theorem, two-person zero-sum games, and duality. Summary Linear program: Minimize cost subject to \(Ax = b\) and \(x\geq 0\) Inequalities make the problem piecewise linear. Simplex method reduces cost from corner point to corner point. Dual linear program is a maximization: Max = Min! Game: \(X\) chooses rows of payoff matrix, \(Y\) chooses columns. Related sections in textbook: VI.2–VI.3 Instructor: Prof. Gilbert Strang

Description Professor Suvrit Sra gives this guest lecture on stochastic gradient descent (SGD), which randomly selects a minibatch of data at each step. The SGD is still the primary method for training large-scale machine learning systems. Summary Full gradient descent uses all data in each step. Stochastic method uses a minibatch of data (often 1 sample!). Each step is much faster and the descent starts well. Later the points bounce around / time to stop! This method is the favorite for weights in deep learning. Related section in textbook: VI.5 Instructor: Prof. Suvrit Sra

Description This lecture is about the central structure of deep neural networks, which are a major force in machine learning. The aim is to find the function that’s constructed to learn the training data and then apply it to the test data. Summary The net has layers of nodes. Layer zero is the data. We choose matrix of weights from layer to layer. Nonlinear step at each layer! Negative values become zero! We know correct class for the training data. Weights optimized to (usually) output that correct class. Related section in textbook: VII.1 Instructor: Prof. Gilbert Strang

Description In this lecture, Professor Strang presents Professor Sra’s theorem which proves the convergence of stochastic gradient descent (SGD). He then reviews backpropagation, a method to compute derivatives quickly, using the chain rule. Summary Computational graph: Each step in computing \(F(x)\) from the weights Derivative of each step + chain rule gives gradient of \(F\). Reverse mode: Backwards from output to input The key step to optimizing weights is backprop + stoch grad descent. Related section in textbook: VII.3 Instructor: Prof. Gilbert Strang

Description Professor Strang starts this lecture asking the question “Which matrices can be completed to have a rank of 1?” He then provides several examples. In the second part, he introduces convolution and cyclic convolution. Summary Which matrices can be completed to have rank = 1? Perfect answer: No cycles in a certain graph Cyclic permutation \(P\) and circulant matrices \(c_0 I + c_1 P + c_2 P^2 + \cdots\) Start of Fourier analysis for vectors Related section in textbook: IV.8 and IV.2 Instructor: Prof. Gilbert Strang

Description This lecture continues with constant-diagonal circulant matrices. Each lower diagonal continues on an upper diagonal to produce \(n\) equal entries. The eigenvectors are always the columns of the Fourier matrix and computing is fast. Summary Circulants \(C\) have \(n\) constant diagonals (completed cyclically). Cyclic convolution with \(c_0, ..., c_{n-1} =\) multiplication by \(C\) Linear shift invariant: LSI for periodic problems Eigenvectors of every \(C =\) columns of the Fourier matrix Eigenvalues of \(C =\) (Fourier matrix)(column zero of \(C\)) Related section in textbook: IV.2 Instructor: Prof. Gilbert Strang

Description Professor Strang begins the lecture talking about ImageNet, a large visual database used in visual object recognition software research. ImageNet is an example of a convolutional neural network (CNN). The rest of the lecture focuses on convolution. Summary Convolution matrices have \(\leq\) \(n\) parameters (not \(n\) 2). Fewer weights to compute in deep learning Component \(k\) from convolution \(c*d\): Add all \(c(j)d(k-j)\) Convolution Rule: \(F(c*d) = Fc\) times \(Fd\) (component by component) \(F\) = Fourier matrix with \(j\), \(k\) entry \(= \exp (2 \pi i j k /n)\) Related section in textbook: IV.2 Instructor: Prof. Gilbert Strang

Description This lecture focuses on the construction of the learning function \(F\), which is optimized by stochastic gradient descent and applied to the training data to minimize the loss. Professor Strang also begins his review of distance matrices. Summary Each training sample is given by a vector \(v\). Next layer of the net is \(F_1(v)\) = ReLU\((A_1 v + b_1)\). \( w_1 = A_1 v + b_1\) with optimized weights in \(A_1\) and \(b_1\) ReLU(\(w\)) = nonlinear activation function \(= \max (0,w) \) Minimize loss function by optimizing weights \(x\)s = \(A\)s and \(b\)s Distance matrix given between points: Find the points! Related sections in textbook: VII.1 and IV.10 Instructor: Prof. Gilbert Strang

Description This lecture continues the review of distance matrices. Professor Strang then introduces the Procrustes problem, which looks for the orthogonal matrix that swings one set of vectors as nearly as possible onto a second set. Summary Distance problem: Find positions \(x\) from distances between them. Necessary and sufficient: Distances satisfy triangle inequality. Procrustes: Given \(n\) vectors \(x\) and \(n\) vectors \(y\). Find the orthogonal matrix \(Q\) so that \(Qx\)s are closest to \(y\)s. Related sections in textbook: IV.9 and IV.10 Instructor: Prof. Gilbert Strang

Description The topic of this lecture is clustering for graphs, meaning finding sets of “related” vertices in graphs. The challenge is finding good algorithms to optimize cluster quality. Professor Strang reviews some possibilities. Summary Two ways to separate graph nodes into clusters k-means: Choose clusters, choose centroids, choose clusters, ... Fiedler vector: Eigenvector of graph Laplacian: \(+-\) signs give 2 clusters Related sections in textbook: IV.6–IV.7 Instructor: Prof. Gilbert Strang

Description Professor Alan Edelman gives this guest lecture on the Julia Language, which was designed for high-performance computing. He provides an overview of how Julia can be used in machine learning and deep learning applications. Summary Automatic differentiation of each operation in Julia Key to deep learning: Optimizing many weights Speed and simplicity: Computing derivatives and Jacobian matrices Related sections in textbook: III.3 and VII.2 Instructors: Prof. Alan Edelman and Prof. Gilbert Strang