HW0_solutions (1)

.pdf

School

University of California, Los Angeles *

*We aren’t endorsed by this school

Course

M146

Subject

Computer Science

Date

Jan 9, 2024

Type

pdf

Pages

Uploaded by phanta_ch

CS M146: Introduction to Machine Learning UCLA Fall Quarter 2023 Prof. Aditya Grover Release: 28 Sep 2023 Homework 0 Due: 5 Oct 2023, Thursday, before 11:59 pm Except for the programming questions, you should be able to solve these problems by hand. Problem 1 (Multivariate Calculus) Consider y = x sin( z ) e − x . What is the partial derivative of y with respect to x ? Solution: ∂y ∂x = sin( z ) ( − xe − x + e − x ) = (1 − x ) sin( z ) e − x Problem 2 (Linear Algebra) (a) Consider the matrix X and the vectors y and z below: X = 2 4 1 3 y = 1 3 z = 2 3 i. What is the inner product y T z ? Solution: y T z = 1 · 2 + 3 · 3 = 11 ii. What is the product X y ? Solution: X y = 2 · 1 + 4 · 3 1 · 1 + 3 · 3 = 14 10 iii. Is X invertible? If so, give the inverse; if not, explain why not. Solution: det( X ) = 2 · 3 − 4 · 1 = 2 ̸ = 0 so X is invertible. For a 2 × 2 matrix, A − 1 = a b c d − 1 = 1 det( A ) d − b − c a = 1 ad − bc d − b − c a . Thus, X − 1 = 1 2 3 − 4 − 1 2 . 1

iv. What is the rank of X ? Solution: The rows (or equivalently, columns) of X are linearly independent, so rank( X ) = 2. (b) Vector Norms [4 pts] Draw the regions corresponding to vectors x ∈ R 2 with following norms (you can hand draw or use software for this question): Solution: Each l p -norm corresponds to a unit ball. In R 2 , this corresponds to a disc of diameter 2 centered at the origin. i. || x || 2 ≤ 1 (Recall || x || 2 = q ∑ i x 2 i .) Solution: the unit circle (origin-centered circle with diameter of 2), and its interior ii. || x || 0 ≤ 1 (Recall || x || 0 = ∑ i : x i ̸ =0 1.) Solution: the axes iii. || x || 1 ≤ 1 (Recall || x || 1 = ∑ i | x i | .) Solution: origin-centered diamond with diameter of 2, and its interior iv. || x || ∞ ≤ 1 (Recall || x || ∞ = max i | x i | .) Solution: origin-centered square with side length of 2, and its interior (c) Matrix Decompositions [6 pts] i. Give the definition of the eigenvalues and the eigenvectors of a square matrix. Solution: An eigenvector of a square matrix is a vector that points in a direction that is invariant under the associated linear transformation. In other words—, if v is a vector that is not zero, then it is an eigenvector of a square matrix A if A v is a scalar multiple of v . This condition could be written as the equation A v = λ v , where λ is a number (scalar) known as the eigenvalue associated with the eigenvector v . ii. Find the eigenvalues and eigenvectors of A = 2 1 1 2 . Solution: The eigenvectors v of this transformation satisfy the equation A v = λ v . Rearrange this equation to obtain ( A − λ I ) v = 0, which has a solution only when its determinant | A − λ I | equals zero. Set the determinant to zero to obtain the polynomial equation, p ( λ ) = | A − λ I | = (2 − λ ) 2 − 1 = ( λ 2 − 4 λ + 4) − 1 = λ 2 − 4 λ + 3 = ( λ − 3)( λ − 1) = 0 , known as the characteristic polynomial of the matrix A . In this case, it has the roots λ = 1 and λ = 3. For λ = 1, the equation becomes, ( A − I ) v = 1 1 1 1 v 1 v 2 = 0 0 , which has the solution, v = 1 − 1 . 2

For λ = 3, the equation becomes, ( A − 3 I ) w = − 1 1 1 − 1 w 1 w 2 = 0 0 , which has the solution, w = 1 1 . Thus, the vectors v and w are eigenvectors of A associated with the eigenvalues λ = 1 and λ = 3, respectively. iii. For any positive integer k , show that the eigenvalues of A k are λ k 1 , λ k 2 , . . . , λ k n , the k th powers of the eigenvalues of matrix A , and that each eigenvector of A is still an eigen- vector of A k . Solution: We will prove by induction. Base case (given): A v = λ v Inductive step: Prove that if A k v = λ k v , then A k +1 v = λ k +1 v . This holds because A k +1 v = A ( A k v ) = A ( λ k v ) = λ k ( A v ) = λ k +1 v . Since both the basis and the inductive step have been performed, by mathematical induction, the statement A k v = λ k v holds for all positive integer k . Q.E.D. (d) Vector and Matrix Calculus [5 pts] Consider the vectors x and a and the symmetric matrix A . i. What is the first derivative of a T x with respect to x ? Solution: Let a i and x i denote the elements of a and x , respectively. Then f ( x ) = a T x = ∑ n i =1 a i x i so ∂f ( x ) ∂x k = ∂ ∂x k ∑ n i =1 a i x i = a k and ∇ x a T x = a . ii. What is the first derivative of x T A x with respect to x ? What is the second derivative? Solution: Let a ij denote the element in row i , column j of A . Then f ( x ) = x T A x = ∑ n i =1 ∑ n j =1 x i a ij x j so ∂f ( x ) ∂x k = ∂ ∂x k n X i =1 n X j =1 a ij x i x j = ∂ ∂x k   X i ̸ = k X j ̸ = k a ij x i x j + X i ̸ = k a ik x i x k + X j ̸ = k a kj x k x j + a kk x 2 k   = 0 + X i ̸ = k a ik x i + X j ̸ = k a kj x j + 2 a kk x k = n X i =1 a ik x i + n X j =1 a kj x j = 2 n X i =1 a ki x i where the last equality follows from a ij = a ji . Note that the k th entry of ∇ x f ( x ) is just the inner product of the k th row of A and x ; therefore, ∇ x x T A x = 2 A x . To find the second derivative, ∂ 2 f ( x ) ∂x k x ℓ = ∂ ∂x k ∂f ( x ) ∂x ℓ = ∂ ∂x k " 2 n X i =1 a ℓi x i # = 2 a ℓk = 2 a kℓ Therefore, ∇ 2 x x T A x = 2 A . 3

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version