Be sure to follow me on Twitter @leftiblog

Wednesday, June 14, 2006


Innumerate headlines

Fraudulent Katrina and Rita Claims Top $1 Billion
You'll notice, I'm sure, there is no "may" in that headline, or an "are projected to." Just a simple declarative verb: "top." And what is the actual number which should be substituted for the "$1 Billion" in that headline, so that it would be, you know, true? $16.8 million. 1.7% of the amount claimed.

Now statistics do have a certain validity, and extrapolations have a mathematical basis to them. But even had the headline been written in a truthful manner, the ability to extrapolate based on such a small number is limited, to put it mildly. This projection is based on 1,500 actual cases of fraud (or, to be more precise, suspected fraud; I don't know that any of it is actually proven yet). The obvious question to ask is, how many cases were examined to find those 1,500? Obvious perhaps, but the reporter covering the story (and the hordes of TV reporters like Anderson Cooper who have been hyping the story) neglected to ask it.

But let's assume that a statistically valid sample was taken (and, I repeat, based on this article and all the TV coverage I've seen of the story so far, there's no evidence that's true). What do the statisticians at GAO project as the final total? "Between $600 million and $1.4 billion." And that's assertedly with a "95% confidence level," meaning it could actually be lower (or higher, of course). Hardly justifying a statement that the "fraudulent claims top $1 billion."

Think about this a little more. "Between $600 million and $1.4 billion" means that some statistician gave the result as $1 billion +/- $400 million. That's one heck of an error bar! Typical polls (e.g., for elections) quote error bars of 3 points, e.g., 48 percent +/- 3 for one candidate, and 52 +/- 3 for the other. Imagine if you saw a poll reading 48 percent +/- 20 percent for one candidate, and 52 +/- 20 percent for the other. You wouldn't be too impressed with that poll, which would have obviously been based on far too small a sample to be significant. Yet that's the kind of precision we're being asked to accept here.

Looked at another way, imagine trying to project the results of an election after 1.7% of the votes had been counted (and in the absence of any exit polls). Obviously such a result would be useless at projecting a winner. And that would be even truer if you knew that that 1.7% wasn't a random selection, but came, say, exclusively from gated communities with homes valued at more than $2 million. And what do we know about how these 1,500 cases of suspected fraud were found? Precisely nothing. Perhaps someone did a preliminary scan of all the claims, and deliberately started by analyzing the ones which sounded the most suspicious. That's certainly the way I would start if I were tasked to find fraud in such a situation. As to what the GAO actually did, however, we have no idea. Did they just take the cases in alphabetical order and start going through them? I doubt it.

But for the Washington Post, CNN, and everyone else hyping the story, none of these issues matter. Only the hype. And, in my opinion, the ability to strike a blow at the very idea of government, which never does anything right, so we might as well do away with it and let people solve their own problems with their own resources, rather than run the risk that some small number of people will take advantage of the system.

Update: Here's the Los Angeles Times: "Non-Victims Bilk $1.4 Billion From FEMA Katrina Funds." Not "may have bilked." Not even "more than $600 million." Just "$1.4 billion," flat out. Flat out alright. A flat out lie.

But I also just found the answer to a question I raised above, in a a more recent AP story. How many cases were studied to find these 1,500 cases of suspected fraud, and on which to project a $1 +/- 0.4 billion total? "The GAO looked only at .01 percent of the 2.5 million applications for assistance." Now that, it turns out, is clearly nonsense, more innumeracy. Because .01 percent of 2.5 million is 250! And it's kind of hard to find 1,500 cases of fraud by examining 250 cases! My best guess is that this is a misprint, and should simply read .01, which is one percent. Which would account for the "projection" (multiply the actual total, $16.8 million, by 100, to get $1.68 billion). But that projection clearly rests on shaky grounds. Although not as shaky as the bogus headlines which accompany it.

This page is powered by Blogger. Isn't yours? Weblog Commenting by HaloScan.com High Class Blogs: News and Media