Saturday, 18 July 2009

Ben Goldacre's Bad Science on the quality of Home Office Research

"We'd all like to help the police do their job well. They, in turn, would like to have a massive database with DNA profiles from everyone who has been arrested, but not convicted of a crime.

We worry that this is intrusive, but some of us are willing to make concessions on our principles and the invasion into our privacy in the name of preventing crimes. To do this, we'd like to know the evidence on whether this database is helpful, to help us make an informed decision.

Luckily, the Home Office has now published a consultation paper on the subject. They defend their database by arguing that innocent people who have been arrested are as likely to commit crimes in the future as guilty people. "This," they say, "is obviously a controversial assertion." That's not true: it's a simple matter of fact, and you could easily assemble some good quality evidence to see if it's true or not.

The Home Office has assembled some evidence. It is not good quality. In fact, this study from the Jill Dando Institute, attached to their consultation paper as an appendix, is possibly the most unclear and poorly presented piece of research I have ever seen in a professional environment. Or am I having a bad day? Join me in my struggle to understand their work.

They want to show that the level of criminal activity in a group of people who have been arrested, but on whom no further action has been taken, is the same as the level of criminal activity in people who have been arrested and convicted of a crime, or who accept a caution. On page 30 they explain their methods haphazardly. They describe some people sampled on 1 June 2004, 1 June 2005 and 1 June 2006. These dates are never mentioned again. They then leap to talking about Table 2. This contains data on people each from a sample in 1996, 1995, and 1994, followed up for 30 months, 42 months, and 54 months respectively. Are these anything to do with the people from 2004, 2005, and 2006? I have no idea.

In fact, I have no idea what "sample" means. Crucially, I also don't know what the numbers in the table mean, because they don't explain this properly. I think it is the number of people, from the original group, who have subsequently been arrested again.

Anyway. Then they start to discuss the results from this table. They say that these figures show that arrested non-convicted people are the same as convicted people. There are no statistics conducted on these figures, so there is absolutely no indication of how wide the error margins are, and whether these are chance findings.

At a few hundred people, this study seems pretty small for one that is supposed to give compelling evidence that there is no difference between two groups – to prove a negative like this, you'd generally want a large sample.

This research was incomprehensible and unreadable. Anybody who claims to have been persuaded by the data quoted here is telling you, loudly and clearly in the subtitles, that they don't need to understand a piece of research in order to find it compelling. If research of this calibre is what guides our policy on huge intrusions into the personal privacy of millions of innocent people, then they might as well be channelling spirits."

The Home Office Document can be accessed here

No comments:

Post a Comment