The New York Times has reported that the Internal Revenue Service gave one of its most rigorous audit types to James B. Comey, the former FBI director, and Andrew G. McCabe, his former deputy.
This has raised many perfectly reasonable questions, most of them variants of: What are the probabilities? As the article points out, the chances of two high-ranking political enemies of President Donald J. Trump being audited by pure coincidence are slim.
But lowercase is not zero.
If we wanted to believe it was a coincidence, how unlikely would we say it is? Here, we try to estimate this probability as seriously as we can.
First, the facts: both men were selected for audits under the National Research Program (NRP), a small subset of all the audits the IRS conducts each year. These audits examine a sample of returns to collect data on tax compliance.
According to the IRS, there were about 5,000 such audits in 2017, 4,000 in 2018 and 8,000 in 2019, chosen from about 154 million individual tax returns each year. The audit of Mr. Comey was up for his 2017 tax return; Mr. McCabe was on his 2019 return.
Many aspects of the NRP complicate our calculations, including the sampling methodology of IRS auditors and the different years of audits. We will return to these topics later. For now, we will assume that all taxpayers have the same chances of being audited and that both men were audited in 2017.
If this problem appeared in a textbook on probability, it could read:
If there are 154 million marbles (the approximate number of tax returns filed each year) in a giant ballot box, and a small number of them are red (including those representing Mr. Comey and Mr. McCabe) , what are the chances that you will get two or more red balls if you randomly remove a few thousand from the ballot box (the number of audits this year)?
It may seem complicated, but it is a relatively well-studied problem, something many math or statistics students would find in their college studies. People have already derived equations to estimate these probabilities, with names like hypergeometric distribution, which has applications such as electoral auditing and card counting.
We can simply enter our estimates for the total number of balls, the number of red balls, and the number of draws, and we will have a probability. If we believe that there are only two red balls, that is, if we limit the exercise only to Mr. McCabe and Mr. Comey, this equation gives a probability of about one in 950 million.
These are considerably higher odds than your chances of winning the Powerball. It is also an almost meaningless result. At best, it is the right answer to the wrong question.
Understanding why it requires recognizing an absurdity inherent in our exercise: to better estimate the probability of an unlikely event, we must set aside the knowledge that it has already happened. (The probability of it happening is 100 percent).
Jordan Ellenberg, a professor at the University of Wisconsin who has written books on math and reasoning, described it this way: “In some counterfactual universe, what is the probability that this will happen, that it has already happened in our universe?”
It may seem strange, but the same problems appear even in probabilistic exercises as basic as throwing a coin.
If you tossed a coin 20 times in a row, your specific sequence of heads and tails is extraordinarily rare, about one in a million, but it happened. And some sequence of laps will always happen. It’s an amazing coincidence only if this is the sequence you set out to achieve before spinning.
Similarly, it is incorrect to limit our search to only Mr. Comey and Mr. McCabe, because it is likely that we would be examining these probabilities if we knew that two other notable political enemies of an administration were audited instead of these two men. .
A better question is: What is the probability that two or more people like Mr. Comey and Mr. Will McCabe be audited during this period?
Should this group of people include two senior FBI officials? Are there two senior Justice Department officials? It is this framing, a subjective rather than a factual decision, that drives more any probability estimate, rather than any statistical distribution option or sampling weights.
Here is a graph of the probability that our equation produces in different options for the number of red marbles, ranging from two (Mr. Comey and Mr. McCabe and no one else) to 400 (a conservative estimate of the number of Americans, Mr. Trump). insulted by his name on Twitter since he began his presidential candidacy).
The likelihood increases dramatically with the choice of who should be considered a red marble next to Mr. Comey and Mr. McCabe.
The question is not to decide a number but to recognize that our choice of group size is what drives our response. While some conjectures are certainly better than others, many options are defensible.
Addressing the details
Now let’s try to cut something a little more realistic and go back to some of the things we ignored in our simple interpretation of this problem.
First, the two men were not audited during the same year. By expanding our scope to cover the three-year period from 2017 to 2019, our resulting probabilities increase significantly. This is simple: if a person has a certain chance of being audited in a given year, more years means more opportunities to be audited.
Second, we are only interested in the probability that at least two people are chosen. We will not consider the probability that the same person is chosen twice; it seems unlikely given that audits can last for a year, according to Mr. Comey. Note that we are analyzing the probability that at least two people will be selected, not exactly two, as it would also be significant if three or more individuals from a group were chosen.
Finally, the IRS does not select people in a truly random manner. In contrast, the agency tends to select certain types of taxpayers, including those who earn high, more often than others. For fiscal year 2001, the NRP sample included returns of people around the 90th percentile of income at about 1.7 times the rate that would be expected if they were income-chosen income. This rate rose to the higher income ranges, so people with incomes up 0.5 percent were more than 10 times more likely to be in the sample than someone closer to average income.
We can probably assume that any group of enemies of Mr. Trump would win more than a random sample of Americans. But we cannot realistically estimate the full income of all members of our group each year. We also know that the IRS has taken other factors into account in its sampling, such as the type of taxpayer filing returns and that sampling methods may change year after year. This leaves us with little guidance on how to match IRS methods. Therefore, we will leave our estimates unweighted by revenue. As a fundraising exercise, if you care about how income affects those results, you can double the resulting probability if you think members of a group have very high incomes, and multiply it by 10 if you think you’re extraordinarily rich.
Putting them all together
By incorporating these options, the following table provides some estimated probabilities based on the size of the group under consideration.
Alternatively, if our choices are unsatisfactory, we’ve created a simple calculator for you to make your own odds:
So which estimate is “correct”?
Most realistic exits from this equation could accurately be described as “very rare” or even “extraordinarily rare,” but none is evidence of misconduct.
“It’s a bit like irresistible force and the immovable object,” said Andrew Gelman, a professor of statistics and political science at Columbia University, when he explained in a summary about this exercise. “On the one hand, you’re saying it’s completely random. On the other hand, you suspect not. “
Mr. Gelman, like all other statisticians who spoke to The Times about this issue, said the biggest hurdle was not any of the details but defining the question itself.
When we try to calculate the probability of a given event because we suspect it may not be random, we end up in the tricky position of trying to imagine how we would have predicted the probability of the event before it happened, said David Spiegelhalter. He directs the Winton Center for Risk and Evidence Communication at the University of Cambridge, an organization dedicated to improving the way quantitative evidence is used in society.
The math is easy, he said, but the wording of the question is complicated, bordering on “nonsense,” in large part because of how difficult it is to determine the group we care about.
“‘What is the possibility of this happening?’ it’s an easy statement to make, ”he said. “It’s a familiar statement to make. But it’s actually a very difficult question to answer.”
Mathematics has its limits. The point of trying to estimate a probability like this, said Mr. Gelman, it’s not about putting too much value on numbers, but letting the result drive you to find out more.
In this case, the best question is not one with an answer that can be found in a statistics textbook.
Instead, said Mr. Gelman, the question to ask is, “What’s going on?”
Matthew Cullen contributed to the report.