Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x*. Let us assume that the variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E, [||Vfi(x)|||xk| ≤ 0² + L ||xk - x* ||². Prove the following statements: 1. If σ > 0 and L = 0, SGD with step size where nk satisfies · E [||x − x. [2] + Σ;=0 130² E[f (zk) - f*]≤ k 2-oj (1) Σj=0jxj Zk= (2) ak In particular, E[f (zk) - f*] converges to 0 if and only if Σ; ni 2. If σ > 0 and L > 0, SGD with a constant step size n satisfies = ∞ and = 0. E ||xk+1 - x* ||² ≤ (1-2μ + m² L)*E ||x − x*||² + (1 − 2nµ + n²L); - ησε 2μ-nL (3) What is the restriction on the stepsize? 3. Let us observe by definition, SGD with step size n satisfies: |xk|1x| = || − x 2 + nổ |Vf(x)|| – 20k (k – x, Vfi(x)). -x -x+ Derive the optimal step size and comment on it. - - (4)