Okay... all that’s left now is to analyze two qualitative variables. Let me to reassure you: you did half the work already in the chapter on contingency tables. If you understood that principle, you’re well ahead of the game.
Questions We Can Ask
The method of analysis will be the same no matter which of the following questions you ask. The only thing that changes is the variable pair:
Do I have the same expense categories on the weekend as during the week? (categ and weekend)
Do I have more money coming in at the beginning of the month or at the end of the month? (debcr and quart_month)
Are my expense amounts larger at the beginning of the month than at the end of the month? (expense_slice and quart_month)
Do my transaction amounts vary from one expense category to the next? (expense_slice and categ)
Are my charge payments always small and my deposits always large? (type and expense_slice)
Are there transaction categories that occur always at the same time of the month, such as rent, for example? (categ and quart_month)
Are there transaction categories for which the form of payment is always the same, for example bank transfer? (type and categ)
Representation
To answer these questions, you can display a contingency table as follows:
X = "quart_month"
Y = "categ"
cont = data[[X,Y]].pivot_table(index=X,columns=Y,aggfunc=len,margins=True,margins_name="Total")
cont
Replace the two qualitative variables on lines 1 and 2 with those you wish to analyze. The contingency table is calculated using the pivot_table
method. Each cell in the contingency table counts a number of individuals. This count is performed using the len
function.
Use Statistics
Unfortunately, we won’t be able to use a model like we did in the last two chapters. But don’t worry! We’ll put things right using a statistical measurement.
Let’s go back to the chapter on contingency tables and look again at what we wrote in the sidebar:
If two events I and J are independent of one another, we expect the number of individuals that satisfy both I and J (let’s call that number nij ) to be equal to fi×nj (this is the calculation you made at the outset: 30% x 10 = 3). On the other hand, the more nij differs from fi×nj , the more reason we have to believe that I and J are not independent.
So, analyzing the correlation between two qualitative variables is the same as comparing the nij s to the fi×njs. The nij s are numbers in the contingency table (except for the TOTAL rows and columns). We could therefore create another table in the same form as the contingency table, but containing the fi×njs instead. Here, then, on the left, is the contingency table that we had before and, on the right, the fi×nj table:
| Coffee | Tea | Other | Total |
Chez Louise | 1 | 9 | 0 | 10 |
Well Ground Coffee | 9 | 6 | 5 | 20 |
Rendezvous Café | 20 | 10 | 10 | 40 |
Sarah's Kitchen | 20 | 5 | 5 | 30 |
TOTAL | 50 | 30 | 20 | 100 |
| Coffee | Tea | Other | Total |
Chez Louise | 5 | 3 | 2 | 10 |
Well Ground Coffee | 10 | 6 | 4 | 20 |
Rendezvous Café | 20 | 12 | 8 | 40 |
Sarah's Kitchen | 15 | 9 | 6 | 30 |
TOTAL | 50 | 30 | 20 | 100 |
Table 2 is what we would expect if the two variables were independent. So we need a statistic that compares the values of these two tables, and that enables us to pinpoint the pairs of cells whose values really differ. These cells will contain values worthy of interest, and will be the source of non-independence between the two variables.
Here we go! If you want to compare two numbers, I suggest you calculate their difference! And to make sure our differences are always positive (to avoid canceling them out by adding them together), let’s square them. This is not the first time we’ve used this little stratagem:
(nij−fi.nj)2
Yes but, if I have nij=2 and fi.nj=4 , the difference when squared will be 4. And if I have nij=1000 and fi.nj=1002, the difference when squared will also be 4. However, an error of 4 when fi.nj equals 4 is a much bigger error than when fi.nj=1002.
That’s true. So we can normalize this difference when squared by dividing it by fi.nj. We then obtain the following formula, knowing that fi=nin :
ξij=(nij−ni.njn)2ni.njn
This measure can be calculated for each cell of the contingency table. It might be interesting to color-code the table according to the results, using dark colors for large results, and light ones for results of close to 0. That way, we can easily tell which cells are the source of non-independence:
Finally, if we add up all of these ξij measures for each cell of the table (from column j=1 to column j=I, and from row i=1 to row i=k), we obtain the statistic ξn (ξ is pronounced “zai” or “sigh”):
ξn=∑ki=1∑lj=1(nij−ni.njn)2ni.njn
Normally, we apply a threshold to this measure, above which we say that the two variables are correlated. It’s a little complicated, but here, we simply need to remember that the larger $ξn$is, the less likely it is that any hypothesis of independence will be valid.
As we have seen, we can associate each cell of the table with a ξij. We can then normalize each ξij by dividing it by ξn. We then obtain for each cell a value of between 0 and 1.
This value can be considered a contribution to non-independence. Optionally, it can be expressed as a percentage, by multiplying it by 100. The closer this contribution gets to 100%, the more the cell in question will be considered a source of non-independence. The sum of all of the contributions is equal to 100%.
Here is the code for generating the color-coded contingency table:
tx = pd.DataFrame(tx)
ty = pd.DataFrame(ty)
tx.columns = ["foo"]
ty.columns = ["foo"]
n = len(data)
indep = tx.dot(ty.T) / n
c = c.fillna(0) # Null values are replaced by 0
measure = (c-indep)**2/indep
xi_n = measure.sum().sum()
sns.heatmap(measure/xi_n,annot=c)
plt.show()
Lines 1 to 6 calculate the indep
table, which is the table representing independence. It calls the matrix product (.dot()
), which you don’t need to know about at this level.
On line 9, measure
contains all of the ξij for each cell of the table. We can then calculate the contributions (defined above), dividing each ξij by ξn (placed within the variable xi_n
). We do this on line 11 with measure/xi_n
. We then obtain a value of between 0 and 1 for each cell.