{"id":166,"date":"2014-07-22T18:31:52","date_gmt":"2014-07-22T18:31:52","guid":{"rendered":"http:\/\/sonny..ogi.edu\/~kgorman\/blog\/?p=166"},"modified":"2014-07-22T18:31:52","modified_gmt":"2014-07-22T18:31:52","slug":"a-tutorial-on-contingency-tables","status":"publish","type":"post","link":"https:\/\/www.wellformedness.com\/blog\/a-tutorial-on-contingency-tables\/","title":{"rendered":"A tutorial on contingency tables"},"content":{"rendered":"

Many results in science and medicine can be compactly represented as a table containing\u00a0the co-occurrence frequencies of two or more discrete random variables. This data structure is\u00a0called the\u00a0contingency table<\/em> (a name suggested by Karl Pearson<\/a>\u00a0in 1904). This tutorial will\u00a0cover\u00a0descriptive and inferential statistics that can be used on the simplest form of contingency table, in which both the outcomes are binomial (and thus the table is 2×2).<\/p>\n

Let’s begin by taking a look at a real-world example: graduate school admissions a single department at UC\u00a0Berkeley in 1973. (This is part of a famous\u00a0real-world example<\/a> which may be of interest to the\u00a0reader.)\u00a0Our dependent variable is “admitted” or “rejected”, and we’ll use applicant gender\u00a0as an independent variable.<\/p>\n\n\n\n\n\n
<\/td>\nAdmitted<\/td>\nRejected<\/td>\n<\/tr>\n
Male<\/td>\n120<\/td>\n205<\/td>\n<\/tr>\n
Female<\/td>\n202<\/td>\n391<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

I can scarcely look at this table without seeing the inevitable question: are admissions in this department gender biased?<\/p>\n

Odds ratios<\/strong><\/p>\n

37% (= 120 \/ 325) of the male applicants who applied were admitted, and\u00a034% (= 202 \/ 593) of the female applicants were. Is that 3% difference a meaningful one? It is tempting to focus on “3%”, but the researcher should absolutely avoid this temptation. The magnitude of the difference between admission rates in the two groups (defined by the independent variable) is very sensitive to the base rate<\/em>, in this case the\u00a0overall admission rate. Intuitively, if 2% of males were admitted and only 1% of females, we would definitely consider the possibility that admissions are gender-biased: we would estimate that males are\u00a0twice<\/em> as likely to be admitted as females.\u00a0<\/em>\u00a0But we would be much less likely to say there is an admissions bias if those percentages were 98% and 99%. Yet,\u00a0in both scenarios the admission rates differ by exactly 1%.<\/p>\n

A better way to quantify the effect of gender on admissions\u2014a method that is insensitive to the overall admission rate\u2014is the\u00a0odds ratio<\/em>. This name is practically the definition if you are familiar with the notion\u00a0odds<\/em>. The odds of some event occurring with probability P<\/em>\u00a0is simply\u00a0P <\/em>\/ (1 \u2013\u00a0P<\/em>). In our example\u00a0above, the odds of\u00a0admission for a male applicant is\u00a00.5854, and is 0.5166 for a female applicant. The ratio of these two is 1.1334 (= 0.5854 \/ 0.5166). As\u00a0this ratio is greater than one, we say that maleness was\u00a0associated<\/em> with admission. This is not\u00a0enough to establish bias: it simply means that males were somewhat more likely to be admitted than females.<\/p>\n

Tests for association<\/strong><\/p>\n

We can now return to the original question: is this table likely to have arisen if there is in fact no gender bias in admissions? Pearson’s chi-squared test<\/a>\u00a0estimates the probability of the\u00a0observed contingency table under the null hypothesis that there is no association between\u00a0x\u00a0<\/em>and\u00a0y;<\/em>\u00a0see here<\/a> for a\u00a0worked example. We reject the null hypothesis that there is no association when this probability is sufficiently small (often at P <<\/em> .05). For this table,\u00a0\u03c7<\/em>2 <\/sup>= 0.6332, and\u00a0the probability of the data under the null hypothesis (no association between gender and admission rate) is\u00a0P<\/em>(\u03c7<\/em>2<\/sup>) = 0.4262. So, we’d probably say the observed difference in admission rates was not sufficient to\u00a0establish that females were\u00a0less likely to be admitted than males in this department.<\/p>\n

The chi-squared test for contingency tables depends on an approximation\u00a0which is asymptotically valid,\u00a0but inadequate for small samples; a popular (albeit arbitrary) rule of thumb is that a sample is “small” if any of the four cells has less than 5 observations. The best\u00a0solution is to use an alternative,\u00a0Fisher’s exact test<\/a>; as\u00a0the name suggests, it provides an exact p<\/em>-value. Rather than working with the \u03c7<\/em>2<\/sup> statistic, the null hypothesis for the Fisher test is that the true (population) odds ratio is equal to 1.<\/p>\n

Accuracy, precision, and recall<\/h1>\n

In the above example, we attempted to measure the association of two random variables which represented different constructs (i.e., admission status and gender). Contingency tables can also be used to look at random variables which are in some sense imperfect\u00a0measures of the same underlying\u00a0construct. In a machine learning context, one variable might represent\u00a0the predictions of a binary classifier, and\u00a0the other represents the labels taken from the “oracle”, the trusted data source the classifier is intended\u00a0to approximate. Such tables are sometimes known as\u00a0confusion matrices<\/em>. Convention holds that one of the two outcomes should be labeled (arbitrarily if necessary) as “hit” and the other as “miss”\u2014often the “hit” is the one which requires further attention whereas the “miss” can be ignored\u2014and the the following labels be assigned to the four cells of the confusion matrix:<\/p>\n\n\n\n\n\n
Prediction \/ Oracle<\/td>\nHit<\/td>\nMiss<\/td>\n<\/tr>\n
Hit<\/td>\nTrue positive (TP)<\/td>\nFalse positive (FP)<\/td>\n<\/tr>\n
Miss<\/td>\nFalse negative (FN)<\/td>\nTrue negative (TN)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The oracle labels are on the row, corresponding to the dependent variable\u2014admission status\u2014in the Berkeley example, and the prediction labels are on the column, corresponding to gender.<\/p>\n

When both variables of the 2×2 table measure the same construct, we start with the assumption that the two random variables are associated, and instead measure agreement.<\/em>\u00a0The simplest measure of agreement is\u00a0accuracy<\/em>, which is the probability that an observation will be correctly classified.<\/p>\n

accuracy = (TP + TN<\/em>) \/ (TP<\/em> + FP<\/em> + FN<\/em> + TN<\/em>)<\/p>\n

Accuracy is not always the most informative measure, for the same reason that differences in probabilities were not informative above: accuracy neglects the\u00a0base rate <\/em>(or\u00a0prevalence<\/em>). Consider, for example, the plight of the much-maligned Transportation Safety Administration<\/a>\u00a0(TSA). Very, very\u00a0few airline passengers attempt\u00a0to commit a terrorist attack during their flight. Since 9\/11, there have only been\u00a0two documented attempted terrorist attacks by passengers on commercial airlines, one by\u00a0Richard Reid<\/a> (the “shoe bomber”) and one by\u00a0Umar Farouk Abdulmutallab<\/a> (the “underwear bomber”), and in both cases, these attempts\u00a0were thwarted by some combination of the attentive citizenry and incompetent attacker,\u00a0not by the TSA’s security theater<\/a>.\u00a0There are approximately 650,000,000\u00a0passenger-flights per year on US flights, so according to my back-of-envelope calculation, there have been around 7 billion passenger-flights\u00a0since 9\/11. In the meantime, the TSA could have\u00a0achieved sky-high accuracy simply by\u00a0making no arrests at all. (I have, of course, ignored the possibility that security theater serves as a deterrent to terrorist activity.) A corollary is that the TSA faces what is called the false positive paradox<\/a>: a false positive (say, detaining a law-abiding citizen) is much more likely than a true positive (catching a terrorist). The TSA isn’t alone: a\u00a0famous paper<\/a> found that few physicians used the base rate (“prevalence”)\u00a0when estimating the likelihood that a patient has a particular disease, given that they had a positive result on a test.<\/p>\n

To better account for the role of base rate, we can break accuracy down into its constituent parts. The best known of these measures is\u00a0precision <\/em>(or\u00a0positive predictive value<\/em>),\u00a0which is defined as the probability that a predicted\u00a0hit is correct.<\/p>\n

precision = TP<\/em> \/ (TP<\/em> + FP<\/em>)<\/p>\n

Precision isn’t completely free of the base rate problem, however; it fails to penalize false negatives. For this, we turn to\u00a0recall<\/em> (or\u00a0sensitivity<\/em>, or\u00a0true positive rate<\/em>), which is the probability that a true hit is correctly discovered.<\/p>\n

recall =\u00a0TP <\/em>\/ (TP<\/em> + FN<\/em>)<\/p>\n

It is difficult to improve precision without sacrificing recall, or vis versa. Consider, for example, an\u00a0information retrieval<\/a>\u00a0(IR) application, which takes natural language queries as input and attempts to return\u00a0all documents relevant for the query. Internally, the IR system ranks all documents for relevance to the query, then returns the top\u00a0n.\u00a0<\/em>A document which is relevant and returned by the system is a true positive, a document which is irrelevant but returned by the system is a false positive, and a document which is relevant but not returned by the system is a false negative (we’ll ignore true negatives for the time being). With this system, we can achieve\u00a0perfect<\/em>\u00a0recall by returning all documents, no matter what the query is, though the precision will be very poor. It is often helpful for\u00a0n<\/em>,\u00a0<\/em>the number of documents retrieved,\u00a0<\/em>to vary as a function of query; in a sports news database, for instance, there are simply more documents about the New York Yankees than about the congenitally mediocre\u00a0St. Louis Blues<\/a>. We can maximize precision by reducing the average\u00a0n\u00a0<\/em>for queries, but this will also reduce\u00a0recall, since there will be more false negatives.<\/p>\n

To quantify the tradeoff between precision and recall, it is conventional to use the harmonic mean of precision and recall.<\/p>\n

F<\/em>1 = <\/em>(2 \u00b7 <\/em>precision\u00a0\u00b7 <\/em>recall) \/ (precision +\u00a0recall)<\/p>\n

This measure is known also known as\u00a0the F<\/em>-score (or\u00a0F<\/em>-measure), though it is properly called\u00a0F<\/em>1, since an\u00a0F<\/em>-score need not weigh precision and recall equally. In many applications, however the real-world costs of false positives and false negatives are not equivalent. In the context of screening for serious illness, a\u00a0false positive\u00a0would simply lead to further testing, whereas a false negative\u00a0could be fatal; consequently, recall is more important then precision. On the other hand, when the resources necessary\u00a0to derive value from true positives are limited\u00a0(such as in\u00a0fraud detection), false negatives are considered more acceptable than false positives, and so precision is ranked above recall.<\/p>\n

Another thing to note about F<\/em>1: the harmonic mean of two positive numbers is always closer to the smaller of the two. So, if you want to maximize F<\/em>1, the best place to start is to increase\u00a0whichever of the two terms (precision and recall) is smaller.<\/p>\n

To make this all a bit more concrete,\u00a0consider the following 2×2 table.<\/p>\n\n\n\n\n\n
Prediction \/ Oracle<\/td>\nHit<\/td>\nMiss<\/td>\n<\/tr>\n
Hit<\/td>\n10<\/td>\n2<\/td>\n<\/tr>\n
Miss<\/td>\n5<\/td>\n20<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

We can see that false\u00a0negatives are somewhat more common than false\u00a0positives, so we could have predicted that precision (0.8333) would be somewhat greater than recall (0.6667), and that F<\/em>1 would be somewhat closer to the latter (0.7407).<\/p>\n

This of course does not exhaust the space of possible summary statistics of a 2×2 confusion matrix: see Wikipedia<\/a> for more.<\/p>\n

Response bias<\/strong><\/p>\n

It is sometimes useful to directly quantify\u00a0predictor\u00a0bias<\/em>, which can be thought of as a signed measure representing the degree to which\u00a0prediction system’s base rate differs from the true base rate. A positive bias indicates that the system predicts\u00a0“hit” more often than it should were it hewing to the true base rate, and a negative bias indicates that “hit” is guessed less often than the true base rate would suggest. One conventional measure of bias is\u00a0Bd<\/sub><\/em>”, which is a function of recall and\u00a0false positive rate\u00a0<\/em>(FAR), defined as follows.<\/p>\n

FAR = FP<\/i>\u00a0\/\u00a0(TN<\/em> + FP<\/em>)<\/p>\n

Bd<\/sub>”<\/em> has a rather unwieldy formula; it is<\/p>\n

Bd<\/sub>”<\/em> =\u00a0[(recall\u00a0\u00b7 (1 \u2013\u00a0recall)) \u2013\u00a0(FAR\u00a0\u00b7 (1 \u2013\u00a0FAR))] \/ [(recall\u00a0\u00b7 (1 \u2013\u00a0recall)) + (FAR\u00a0\u00b7 (1 \u2013\u00a0FAR))]<\/p>\n

when HR \u2265 FAR and<\/p>\n

Bd<\/sub>”<\/em>\u00a0=\u00a0[(FAR \u00b7 (1 \u2013 FAR)) \u2013\u00a0(recall \u00b7 (1 \u2013 recall))] \/ [(recall\u00a0\u00b7 (1 \u2013\u00a0recall)) + (FAR\u00a0\u00b7 (1 \u2013\u00a0FAR))]<\/p>\n

otherwise (i.e., when FAR > HR).<\/p>\n

You may also be familiar with\u00a0\u03b2<\/em>, a parametric measure of bias, but there does not seem to be anything to\u00a0recommend it over\u00a0Bd<\/sub>”, <\/em>which makes fewer assumptions\u00a0<\/em>(see\u00a0Donaldson 1992 and citations therein).<\/p>\n

Cohen’s\u00a0\u039a<\/em><\/strong><\/p>\n

Cohen’s\u00a0\u039a <\/em>(“kappa”)\u00a0is a statistical measure of\u00a0interannotator agreement<\/em>\u00a0which works on 2×2 tables. Unlike other measures we have reviewed so far, it is adjusted for\u00a0the percentage of agreement that would\u00a0occur by chance.\u00a0\u039a<\/em>\u00a0is computed from two terms. The first,\u00a0P<\/em>(a<\/em>), is the observed probability of agreement, which is the same formula as\u00a0accuracy. The second,\u00a0P<\/em>(e<\/em>), is the probability of agreement due to chance. Let\u00a0Px\u00a0<\/sub><\/em>and Py<\/sub><\/em>\u00a0be the probability of a “yes” or “hit” answer from annotator\u00a0x<\/em> and\u00a0y<\/em>, respectively. Then, P<\/em>(e<\/em>) is<\/p>\n

Px<\/sub><\/em> Py<\/sub><\/em> + (1 \u2013\u00a0Px<\/sub><\/em>) (1 \u2013\u00a0Py<\/sub><\/em>)<\/p>\n

and \u039a\u00a0<\/em>is then given by<\/p>\n

[P<\/em>(a<\/em>) \u2013\u00a0P<\/em>(e<\/em>)] \/ [1 \u2013\u00a0P<\/em>(e<\/em>)] .<\/p>\n

For the previous 2×2 table,\u00a0\u039a = <\/em>.5947; but, what does this mean?\u00a0K <\/em>is usually\u00a0interpreted with reference to conventional\u2014but entirely\u00a0arbitrary\u2014guidelines. One of the best known of these is due to Landis and Koch (1977), who propose 0\u20130.20 as “slight”, 0.21\u20130.40 as “fair agreement”, 0.41\u20130.60 as “moderate”, 0.61\u20130.80 as “substantial”, and 0.81\u20131 as “almost perfect” agreement.\u00a0\u039a<\/em> has a known statistical distribution, so it is also possible to test the null hypothesis that the observed agreement is entirely due to chance. This\u00a0test is rarely performed or reported, however, as the null\u00a0hypothesis is exceptionally unlikely to be true in real-world annotation scenarios.<\/p>\n

(h\/t: Steven Bedrick<\/a>.)<\/p>\n

References<\/h1>\n

A. Agresti. 2002.\u00a0Categorical data analysis.\u00a0<\/em>Hoboken, NJ: Wiley.
\nW. Donaldson. 1992. Measuring recognition memory.\u00a0Journal of Experimental Psychology: General<\/em>\u00a0121(3): 275-277.
\nJ.R. Landis & G.G. Koch. 1977.The measurement of observer agreement for categorical data.\u00a0Biometrics <\/em>33(1): 159-174.<\/p>\n","protected":false},"excerpt":{"rendered":"

Many results in science and medicine can be compactly represented as a table containing\u00a0the co-occurrence frequencies of two or more discrete random variables. This data structure is\u00a0called the\u00a0contingency table (a name suggested by Karl Pearson\u00a0in 1904). This tutorial will\u00a0cover\u00a0descriptive and inferential statistics that can be used on the simplest form of contingency table, in which … Continue reading “A tutorial on contingency tables”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[10],"tags":[],"class_list":["post-166","post","type-post","status-publish","format-standard","hentry","category-stats"],"_links":{"self":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/comments?post=166"}],"version-history":[{"count":0,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/166\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/media?parent=166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/categories?post=166"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/tags?post=166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}