The Prosecutor’s Fallacy

Good.

Now: If you, defendant, got a positive lie detector test, what is the chance you were actually lying?What the polygraph examiner really wants to know is not P(+|L), which is the accuracy of the test; but rather P(L|+), or the probability you were lying given that the test was positive.

We know how P(+|L) relates to P(L|+).

P(L|+) = P(+|L)P(L) / P(+)To figure out what P(+) is independent of our prior knowledge of whether or not someone was lying, we need to compute the total sample space of the event of testing positive using the Law of Total Probability:P(L|+) = P(+|L)P(L) / P(+|L)P(L) + P(+|L^c)P(L^c)That is to say, we need to know not only the probability of testing positive given that you are lying, but also the probability of testing positive given that you’re not lying (our false positive rate).

The sum of those two terms gives us the total probability of testing positive.

That allows us to finally determine the conditional probability that you are lying:The probability that you are actually lying, given that you tested positive on the polygraph, is 45.

68%.

The probability of a false positive is 54.

32%.

The probability that you’re actually lying, given a positive test result, is only 45.

68%.

That’s worse than chance.

Note how it differs from the test’s accuracy levels (81% true positives and 71% true negatives).

Meanwhile, your risk of being falsely accused of lying, even if you’re telling the truth, is also close to-indeed, slightly higher than-chance, at 54.

32%.

Not reassuring.

Marston was, in fact, a notorious fraud.

The Frye court ruled that the polygraph test could not be trusted as evidence.

To this day, lie detector tests are inadmissible in court because of their unreliability.

But that does not stop the prosecutor’s fallacy from creeping in to court in other, more insidious ways.

This statistical reasoning error runs rampant in the criminal justice system and corrupts criminal cases that rely on everything from fingerprints to DNA evidence to cell tower data.

What’s worse, courts often reject the expert testimony of statisticians because “it’s not rocket science”-it’s “common sense”:In the Netherlands, a nurse named Lucia de Berk went to prison for life because she had been proximate to “suspicious” deaths that a statistical expert calculated had less than a 1 in 342 million chance of being random.

The calculation, tainted by the prosecutor’s fallacy, was incorrect.

The true figure was more like 1 in 50 (or even 1 in 5).

What’s more, many of the “incidents” were only marked suspicious after investigators knew that she had been close by.

A British nurse, Ben Geen, was accused of inducing respiratory arrest for the “thrill” of reviving his patients, on the claim that respiratory arrest was too rare a phenomenon to occur by chance given that Green was near.

Mothers in the U.

K.

have been prosecuted for murdering their children, when really they died of SIDS, after experts erroneously quoted the odds of two children in the same family dying of SIDS as 1 in 73 millionBen GeenThe data in Ben Geen’s case are available thanks to Freedom of Information requests — so I have briefly analyzed them.

# Hospital data file from the expert in Ben Geen's exoneration case# Data acquired through FOI requests# Admissions: no.

patients admitted to ED by month# CardioED: no.

patients admitted to CC from ED by month with cardio-respiratory arrest# RespED: no.

patients admitted to CC from ED by month with respiratory arrestThe most comparable hospitals to the one in which Geen worked are large hospitals that saw at least one case of respiratory arrest (although “0” in the data most likely means “missing data” and not that zero incidents occurred).

ax = sns.

boxplot(x='Year', y='CardioED', data=df)ax = sns.

respiratory eventsThe four hospitals that are comparable to the one where Geen worked are Hexham, Solihull, Wansbeck, and Wycombe.

The data for Solihull (for both CardioED and RespED) are extremely anomalous:After accounting for the discrepancies in the data, we can calculate that respiratory events without accompanying cardiac events happen, on average, roughly a little under 5 times as often as cardiac events (4.

The average number of respiratory arrests per month unaccompanied by cardiac failure is approximately 1–2, with large fluctuations.

That’s not particularly rare, and certainly not rare enough to send a nurse to prison for life.

(You can read more about the case and this data here and see my jupyter notebook here.

)Common sense, it would seem, is hardly common — a problem which the judicial system should take much more seriously than it does.

Originally published at https://www.

espritdecorpus.

com/posts/the-prosecutors-fallacy/.

.