CalTech machine learning, video 12 note(regularization)

10:05 2014-10-08

start CalTech machine learning, video 11

regularization

10:05 2014-10-08

overfitting: we're fitting the data all too well

at the expense of the out-of-sample expense

10:06 2014-10-08

if you think of what the VC analysis told us,

the VC analysis told us that given the data resources &

the complexity of the hypothese set with nothing said 

about the target,given those we can predict the level

of generalization as a bound.

10:08 2014-10-08

data resouce + VC dimension => level of generalization

10:09 2014-10-08

source of the overfitting is to fit the noise

11:15 2014-10-08

stochastic noise/deterministic noise

11:16 2014-10-08

deterministic noise is the function of limitation of your model

11:17 2014-10-08

1st cure for overfitting

11:18 2014-10-08

outline:

* Regularization - informal

* Regularization - formal

* Weight decay

* Choosiing a regulizer

11:19 2014-10-08

unconstrained solution: minimize Ein //in-sample error

11:37 2014-10-08

let's look at the constrained version, what happens

if we constaining the weights

11:40 2014-10-08

so here is the constaint we're going to work with

11:41 2014-10-08

I have a smaller hypothese set, so the VC dimension is going

in the direction of being smaller, so I'm standing at the point 

of better generalization.

11:44 2014-10-08

constraining the weights:

* Hard constaint

* Softer-order constaint

11:45 2014-10-08

Wreg instead of Win

11:46 2014-10-08

you are minimize this subject to the constaint

11:47 2014-10-08

KKT

11:47 2014-10-08

I have 2 thing here, I have the error surface trying

to minimize, and I have the constraint.

11:49 2014-10-08

I'm going to put contours where the in-sample error is constant

11:50 2014-10-08

let's take a point on the surface

11:52 2014-10-08

let's look at the gradient of the objective function

11:53 2014-10-08

gradient of the objective function will give me a good

idea about the direction to move in order to minimize the

objective function

11:54 2014-10-08

move along the circle will change the value of Ein

11:58 2014-10-08

Augmented error: Eaug(w)

12:05 2014-10-08

regularization term

12:06 2014-10-08

I use a subset of the hypothese set and I expect good 

generalization.

12:07 2014-10-08

one step learning including regularization

12:13 2014-10-08

let's apply it and see the results in the real case.

12:14 2014-10-08

so the medicine is working, a small dose of medicine

did the job

12:16 2014-10-08

I think we're overdosing here.

12:16 2014-10-08

if you keep increasing λ => overdose !!!

12:17 2014-10-08

the choice of λis extremely critical

12:18 2014-10-08

the good new is that this will be a heuristic choice

12:18 2014-10-08

the choice of λwill be extremely principled based on validation

12:19 2014-10-08

we went to another extreme: now we're "underfitting"

12:20 2014-10-08

overfitting => underfitting

12:20 2014-10-08

the proper choice of λ is important

12:21 2014-10-08

the most famous regularizer is "weight decay"

12:21 2014-10-08

we know in neural network you don't have a neat

closed-form solution, you use gradient descent

12:22 2014-10-08

batch gradient descent => stochastic gradient descent(SGD)

12:23 2014-10-08

I'm in the weight space & this is my weight, and 

here is the direction that backpropagation suggest to move to.

12:27 2014-10-08

used to be without regularization, I move from here to here

12:27 2014-10-08

shinking & moving

12:28 2014-10-08

the weight decay from this one to the next

12:29 2014-10-08

weight space

12:31 2014-10-08

some weights are more important than others

12:31 2014-10-08

low-order fit

12:33 2014-10-08

Tikhonov regularizer

12:34 2014-10-08

regularization parameter λ

12:36 2014-10-08

you have to use the regularizer, because without 

the regularizer, you're going to get overfitting

12:38 2014-10-08

but there are guidelines to choose the regularizer

12:38 2014-10-08

after you choose the regularizer, there is a check of

the λ

12:39 2014-10-08

practical rule:

stochastic noise is 'high-frequency'

deterministic noise is also non-smooth

12:41 2014-10-08

because of this, here is the guideline for

choosing regularizer:

=> constrain learning towards smoother hypothese

12:42 2014-10-08

regularization is a cure, and the cure has a side-effect

12:42 2014-10-08

it's a cure for fitting the noise

12:43 2014-10-08

punishing the noise more than you punishing the signal

12:43 2014-10-08

in most of the parameterization, small weights correspond 

to smoother hypothese, that's why small weights or 'weight decay'

works well in those cases.

12:45 2014-10-08

general form of augmented error

calling the regularizer Ω = Ω(h)

12:46 2014-10-08

we minimize 

Eaug(h) = Ein(h) + λ/ N * Ω(h) 

// this is what we minimize

12:47 2014-10-08

Eaug is better than Ein as a proxy for Eout

12:50 2014-10-08

augmented error(Eaug) is better than Ein for approximating Eout

12:51 2014-10-08

we found a better proxy for the out-of-sample(Eout)

12:51 2014-10-08

how we choose regularizer?

mainly a heuristic choice

12:52 2014-10-08

perfect hypothese set

12:52 2014-10-08

the perfect regularizer Ω:

constaint in the 'direction' of the target function

12:52 2014-10-08

regularization is an attempt to reduce overfitting

12:55 2014-10-08

harms the overfitting(noise) more than the fitting

12:56 2014-10-08

guidelines:

the direction of smoother

12:56 2014-10-08

we have the error function for the movie rating

12:57 2014-10-08

the notion of simple here is very interesting

12:59 2014-10-08

now you're regularizer to the simpler solution

13:04 2014-10-08

what happened if you choose a bad Ω? // Ω regularizer

we don't worry too much, because we have the saving grace 

of λ, we're going to validation

13:06 2014-10-08

the validation will tell us it's harmful, we'll factor

the regularizer out of the game all together.

13:08 2014-10-08

neural network regularizer

13:09 2014-10-08

weight decay

13:09 2014-10-08

so we have this big network, layer upon layer upon layer...

13:11 2014-10-08

I'm looking at the functionalities that I'm implementing

13:12 2014-10-08

as you increase the weight, you're going to enter the more

interesting nonlinearity here.

13:12 2014-10-08

you're going from the most simple to the most complex

13:13 2014-10-08

weight decay: from linear to logical

13:13 2014-10-08

weight elimination:

fewer weights => smaller VC dimension

13:15 2014-10-08

early stopping as a regularizer

13:17 2014-10-08

regularization through the optimizer

13:18 2014-10-08

the optimal λ:

as you increase the noise, you need more regularization

-----------------------------------------------------

13:38 2014-10-08

there are regularizer stood the test of time

13:38 2014-10-08

machine learning is somewhere between theory & practice

13:39 2014-10-08