Logistic Regression
In the previous chapter, we looked at how a classification task (predicting high or low happiness) can be carried out by building a decision tree. But decision trees are not the only algorithm you can use for classification! In this chapter, we will look at another algorithm, logistic regression.
But First - Linear Regression
Before we dive into logistic regression, let’s look at its cousin, linear regression. We will be taking a deeper look at linear regression in the next part of this course. But for now, you need to know just enough to help you understand logistic regression.
Take a look at the following scatter plot showing income against happiness for 100 countries.
The mean income per person is measured in thousands of dollars. The happiness is measured by a survey, with each country accumulating a score between 0 and 10, with 0 being deeply unhappy, and 10 being incredibly happy.
We can fit a line to this data:
You will see how this line is fit when we look at regression in the next part of this course. You may be familiar with the general idea of fitting a line to a plot of sample points. You may have done this when analyzing the results of a science experiment at school!
The fitted line gives a feature you can use to predict happiness from income levels. For example, a country with a mean income of $70,000 per person has a predicted happiness rating of 5.9:
We've built a linear regression model!
Binomial Logistic Regression
Instead of a happiness value between 0 and 10, what if we measured countries just with high and low happiness, as in the example in Part 1 of this course?
The plot would look more like this:
The machine learning task now is to use income to predict high or low happiness. But the relationship is not linear, so you can't just fit a line like this:
Consider three areas of the chart: One where the countries with income < $40k are low happiness, another where countries with income > $90k are high happiness, and another, where for countries between are less clear-cut:
A straight line can't model this relationship, but what if you could do this:
This is exactly what logistic regression does!
The curve is called a sigmoid curve, because it is S shaped.
You can use this curve to obtain a probability of a low or high classification. Here we have an 88% probability of high happiness for a country with a mean income of $75,000.
Now we can use a threshold of 50% to make a hard high or low prediction.
Multinomial Logistic Regression
What if there are more than two classes? For example, what if we split the countries into high, medium, and low happiness?
There are a few strategies for dealing with this. The one-vs-rest (OVR) algorithm trains a model for each class against the others. So you would train the following models:
High vs. medium and low
Medium vs. high and low
Low vs. high and medium
Recap
Logistic regression is a regression algorithm. Regression helps predict continuous variables.
With logistic regression, the continuous variable is the probability of a categorial prediction.
Binomial Logistic Regression predicts one of two categories.
Multinomial Logistic Regression makes predictions when there are more than two categories.
In the next chapter, we will build and evaluate a classification model.
