[Supervised ML] The problem of overfitting / Regularization - 10

1. Intuition 

---

Regression Example

Underfit / High bias : does not fit the training set well

Overfit / High variance: Fits the training set extremely well 

Generalization: Fits training set pretty well

Even though overfitting example might have a high accuracy on the training set, it will not have a high accuracy for new test dataset.

 

2. Addressing overfitting

---

Option 1: Collecting more examples

 

By collecting more training examples, the function will be more generalized.

 

Option 2: Select features to include/exclude 

 

Overfitting might occur by taking in too many features: wx1 + wx2 + wx3 .........

By selecting only necessary features(feature selection), we can generalize the function. 

 

Option 3: Regularization

 

Reducing the size of parameteres w.

Rather than eliminating the feature, we can gently reduce the impact of the feature. 

 

3. Regularization 

---

3.1 Intuition

By setting w to a really small number, we can reduce the impact of our selected features.

 

How do we implement it in our cost function? 

Lambda is a regularization parameter

In some cases, we penalize the impact of b. We add lambda/2m * b^2. 

 

ex) When regularization term is really big, w will approach to 0.

 

3.2 Regularized Linear Regression

We will implement regularization term to our gradient descent formula in linear regression.

This is the gradient descent formula with regularization included.

 

3.3 Regularized Logistic Regression

Regularized Logistic Regression is the same as linear regression except for f(x) which uses the sigmoid function.