Reading Report Chapter 1 of PRML

main content

• probability theory
  • sum rule and product rule

  • Bayes’ theorem

    p o s t e r i o r ∝ l i k e l i h o o d × p r i o r posterior\propto likelihood\times prior posterior∝likelihood×prior

  • maximum likelihood approah to inferring parameters in a probability distribution using oberved data
  • sum-of-squares error function is a consequence of maximizing likelihood under a Gaussian noise distribution
  • MAP
• decision theory
  • classification
    • minimizing the misclassification rate
    • minimizing the expected loss
  • regression
    • loss function for regression
  • three methods to solve decision problems and some reasons for why posterior probabilities are required in many cases
• information theory
  • entropy
    • the relation between entropy and shortest coding length (noiseless coding theorem)
    • interpretation as a measure of disorder
    • entropy over discrete and continuous variables (differential entropy)
  • relative entropy, K L ( p ∣ ∣ q ) ≥ 0 KL(p||q)\ge 0 KL(p∣∣q)≥0 with equality if and only if p ( x ) = q ( x ) p(x) = q(x) p(x)=q(x)
  • mutual information, KL divergence between the joint distribution and the product of the marginals
• model selection
• curse of dimensionality

some thinking

The book exhibits much strict mathematical inference and emphasizes on the Bayesian viewpoint, which is very different from Hands-On Machine Learning with Scikit-Learn & TensorFlow, a book I have read previously.

Chapter 1 mainly introduces three basic theories, and bring the Bayesian perspective to each one.

In Bayesian view, probabilities provide a quantification of uncertainty which differs from frequentist viewpoints. Posterior is an adjustment of condition probability using prior probability, which is important in some decision problems such as compensating for class priors where the target class is rare in total data set.

Chapter 1 is only the introduction of the whole book and I believe that with further reading of the book I can have a better comprehension of PRML.

some problems

• still don’t have a deep understanding of Bayesian viewpoint
• have much difficulity understanding some mathematical resoning due to lack of some knowledge background such as the calculus of variations
• too slow reading speed and takes a lot of time to read and understand the meaning

future

• go on reading to gain overall viewpoints and better understanding of previous chapters
• enhance some necessary knowledge