This is a multivariate polynomial regression model written in Python and utilizing NumPy that I wrote after learning about the basics of Machine Learning.
The code for this project is self-documented and basic knowledge of Machine Learning should suffice in order to understand it.
Correction: It turns out that vectorizing everything convoluted the code. The formulas should match those presented in the following section but thorough knowledge of linear algebra and intuition for Machine Learning is required to understand what is actually happening. The code is documented in a way that explains the high-level idea behind the functions but not the details about the computations they carry out. I tried to make things as readable as possible (i.e., avoid outright one-liners) but there's not much else I can do to document a complicated linear algebra formula.
On the upside, after vectorizing everything, the code runs way faster. For example, 100,000 iterations of gradient descent used to take a couple minutes but after vectorization, it typically runs in under 10 seconds.
This section is mainly meant as a way for me to strengthen my understanding of the concepts by explaining them. The explanations given here go into the theory and math behind the model and are far more lengthy than needed.
Of course, the goal of regression is to fit some line to a set of data. The model function represents that line and is what the model uses to make its predictions. Traditionally, a univariate linear regression model uses the function
where
For a multivariate linear regression model, there are multiple input features, which we denote as matrix
which could also be expressed as
More concisely, if we let vector
To accommodate a nonlinear association between input features and target outputs, we require a polynomial regression model, which can be done by introducing polynomial terms into our model, such as
For this model in particular, we choose a degree
Perhaps summation notation is more concise; we can write
(I know this model function is probably terribly confusing; I try to explain it better in the next subsection.) Of course, we could also combine different features together to create a term like
The main downside of defining our model function as polynomial such as the ones above is that it complicates our cost function, making the gradients difficult to compute and limiting the amount of vectorization we can perform. To solve this, map polynomial features to the raw input data and then perform linear regression. For example, if we wanted the feature
For a quick explanation of why this works suppose we are dealing with univariate input data and have the points
From now on, we assume that our features have been mapped to polynomial ones and we continue to denote the features matrix as model.py
but both yield the same result.) Mathematically, we can express this reassignment as
where =
versus ==
in code). If any of this caused more confusion about polynomial feature mapping or the model function for this project, I apologize; hopefully the code makes more sense.
Since we are now only concerned with linear regression, our model function can be rewritten as
Alternatively, we could have our function output a vector of predictions given the entire data set if we define it as
where
The goal of normalization is to get all input features within a similar range, typically one centered around
There are many ways to perform normalization but for this project, I chose z-score normalization. This is done by reassigning each
Here,
With the data preprocessing out of the way, we can move on to the core of the regression algorithm.
Just like with the data normalization function and other aspects of ML, there are a wide variety of cost functions to choose from, each with their own benefits. For the sake of simplicity however, I chose the mean squared error (MSE) cost function. It also has the very useful property that for regression problems, it is always convex, which means there is only one minima for gradient descent to converge to.
For the MSE cost function, we define the loss function as the error between predicted and expected values for a single example. Its formula is given by
where
However, before moving on to
The cost function is defined as the average of the loss function across all examples. It is given by
If we expand using the definition of the loss function and the definition of our model function, the cost function can be expressed as
By using the definition
where vector
While the cost function defined above may work for simpler cases of multivariate regression, it is susceptible to overfitting. This is where the the model function is over-optimized to fit the training data. Especially with a higher degree polynomial, the cost on the training set may be minimized but it will likely not match more general trends in the data, meaning the model will perform poorly on any new examples it is tested on. While overfitting can be solved by using more training examples, this may not always be an option, hence the need for regularization.
In order to mitigate overfitting, the model should punished for having high values of the parameters in vector
where
The purpose of gradient descent is to "descend" down the cost function to a local/global minima in order to find the optimal parameters for the model. Mathematically, this can be expressed as
In order to this, we need the gradient of the cost function, which reveals the "direction" in which the function increases most steeply. By "going" in the opposite direction, we "descend" down the cost function. To compute the gradient, we need the partial derivatives of the cost function with respect to its parameters. The partial derivatives are given by
and
Of course, we can express these in a vectorized manner if we use
Now that we have the partial derivatives, we can move on to the actual gradient descent process itself. This involves tens or hundreds of thousands of iterations with the following updates to the model's parameters at each step
It is important to note that the update steps should be performed simultaneously, that is the partial derivatives should both be computed before updating the respective parameters. More explicitly, the update steps can be expressed as
where
With the inner workings of the model dissected, we are now concerned with evaluating our model.
For this project in particular, I used the popular Boston House Prices Dataset which contains about 500 examples in total. I decided to split the data up such that 80% of it is used for training the model to find the optimal parameters while the other 20% is used for testing the model and seeing how well it might perform on new data. To quantify its performance, I settled on the very simple metric of mean absolute percent error (MAPE), which is given by the formula
While above is the traditional (statistics) formula for MAPE, we can vectorize it in the spirit of this project as
where
After running the program with different parameters (I don't mean the model's parameters
for the polynomial degree, regularization parameter, and learning rate respectively. (Granted, I may later change the values in model.py
without updating this README.md
as I continue to tweak the model.)
Overall, I found that the model performed with about 10% to 16% MAPE for both the training and testing data. I would like to get it below 10% but I suspect the variation in the dataset simply renders this impossible. Alternatively, some more advanced regression techniques could potentially solve this problem.
Through this project, I was able to build a solid foundation in the basics of Machine Learning, namely regression problems. It was really helpful to write the code for the model from scratch and tweaking the settings (learning rate, regularization parameter, etc.) helped build intuition for things like convergence to a local minima and overfitting. Vectorizing everything proved to be a fun exercise and allowed me to apply my knowledge of linear alegbra. In addition, it was incredible to see how the code ran much faster when most of the computational "heavy-lifting" was done by NumPy. I hope to continue my journey through ML and this project served as an excellent launchpad for doing so.
Numpy - Heavily utilized throughout the project and immensely useful for speeding up linear algebra computations.
Coursera Machine Learning Specialization Course 1: Supervised Machine Learning: Regression and Classification - Taught me about regression in Machine Learning and was the main source of inspiration for this project; some of the math and code was adapted from this course.
Linear Regression (Multivariate). Cost Function. Hypothesis. Gradient - Inspired the complete vectorization of computations in this project, some math formulas adapted from this webpage.