A linear model is a statistical learning method that assumes a linear relationship between the target variable and the feature variable, i.e. it can be represented by a linear equation. The general form of a linear model is:
where y is the target variable, x1, x2,..., xn is the feature variable, w0, w1,...,w are the model parameters, and ε is the random error term. The advantage of linear models is that they are simple, easy to understand and interpret, and are suitable for various types of data, such as numeric, categorical, text, etc. The disadvantage of linear models is that they cannot capture nonlinear and complex relationships, and there may be a risk of underfitting.
LR (Logistic Regression) is a commonly used linear model used to solve binary or multiclassification problems. The basic idea of LR is to convert the output of a linear model into a probability value through a Sigmoid function, and then judge the class based on the probability value. The general form of LR is:
where P(y=1∣x) represents the probability that given the characteristic variable x, the target variable y is a positive class, and w0, w1,...,wn are model parameters. The advantage of LR is that it can directly output probability values for probabilistic reasoning and decision-making, and can also prevent overfitting by adding regularization terms. The disadvantage of LR is that it cannot handle nonlinearly separable data, and there may be multicollinearity problems.
Why is LR a linear model? Because the decision boundary of LR is a linear equation, i.e.:
w0+w1x1+w2x2+...+wnxn=0
When the above equation holds, P(y=1∣x)=0.5, i.e. the probability of positive and negative classes is equal. When the above equation is greater than zero, P(y=1∣x)>0.5, that is, the probability of a positive class is greater than that of a negative class. When the above equation is less than zero, P(y=1∣x)<0.5, that is, the probability of a positive class is less than that of a negative class. Therefore, LR can divide the dataset into two parts according to this linear equation, which shows that LR is a linear classifier.