In this project you will implement gradient descent for linear regression on Spark using Scala.

The gradient descent update for linear regression is:

Wi+1 = Wi · αἱ Σ(wx; - yj)x;

- Part 1 (20 points)

First, implement a function that computes the summand (w¹x - y)x,, and test this function on two examples. Use (Vectors) to create dense vectors w and use (Labeled Point)

to create training dataset with 3 features. You can also use (Breeze) to do the dot product.

Part 2 (20 points)

Implement a function that takes in vector w and an observation's Labeled Point and returns a (label, prediction) tuple. Note that we can predict by computing the dot

product between weights and an observation's features. Test this function on a Labeled Point RDD.

Part 3 (20 points)

Implement a function to compute (RMSE) given an RDD of (label, prediction) tuples:

Test this function on an example RDD.

RMSE =

-18

n

Σ(vi - 1/2

i=1 Part 4 (40 points)

Implement a gradient descent function for linear regression:

The function will take trainData (RDD of Labeled Point) as an argument and return a tuple of weights and training errors. Reuse the code that you have written in Part 1 and 2.

Initialize the elements of vector w = 0 and a = 1. Update the value of a in ith iteration using the formula:

Wi+1 = Wi-ai (w/ x; -yj)xj

Bonus (20 points)

Implement the closed form solution:

Test the function on and example RDD. Run it for 5 iterations and print the results.

You can assume X is a (DenseMarix).

α₂ =

α

n√i.

w = (x¹x)¯¹x²