一天搞懂机器学习PPT笔记-2

Tips for Training DNN

minimize total loss

more layers do not imply better

- so it is hard to get the power of deep

Learning rate

• popular&simple idea:reduce the learning rate by some factor every few epochs

  – at the beginning,we are far from the destination,so we use larger learning rate

  – after several epochs,we are close to the destination,so we reduce the learning rate.

  – a demo rate function:rate = init rate / sqrt(t+1)

  – learning rate cannot be one-size-fits-all,so we should give different parameters and different learning rates.

hard to find optimal network parameters

- there are many points where the value of judging the parameters is 0.so we has the Momentum

Momentum

– to make sure that we can find the better parameters

Why Overfitting

• training data and testing data can be different
• learning target is trained by the training data
• the parameters achieving the learning target do not necessary have good results on the testing data

panacea for OverFitting

• have more training data
• create more training data,for example:

  

some ways to reduce the time to get the better parameters

• early Stopping

  

• weight decay

  

• drop out

  

  

Variants of Neural Networks

• Convolutional Neural Network(Widely used in image processing)
• Recurrent Neural Network(RNN)