Statistics - Do they just make them up? Part Two

Searching for material for my teaching I came across a BBC Radio Four programme A Baby ASBO first broadcast in 2009. About five minutes into the programme a new initiative aimed at Primary School Children called Growing Against Gangs is introduced. The programme ran by an organisation called Gang Resistance Education and Training and is adapted from a similar scheme in the United States. The programme involves police officers delivering lessons to primary school children. In this programme we hear PC Emma Hart "teaching" some young children. She tells the children she is "going to through some facts" with them. She checks that they understand what facts are before telling them:

Most gang members make very little money ... 95% of drug dealers are working for less than the minimum wage you could earn at McDonalds ...

What PC Hart fails to tell the children that her "fact" is based on a paper published in the Quarterly Journal of Economics in 2000 by Steven Levitt and Sudhir Venkatesh An Economic Analysis of a Drug Selling Gang's Finances. The paper is based on data collected by Venkatesh on a Chicago gang in the mid 1980s whose activities included dealing crack.  In fact the paper although showing that the rewards for drug dealing were very much concentrated at the higher levels had concluded that average rewards were better than the participants would have gained in legitimate occupations. However these rewards were not sufficiently high as to justify the risks involved in gang activity. The activity could be better explained they argued by gang members hopes of high future earnings (when and if they made it higher up the drug dealing ladder) rather than current earnings.

This research has featured in Levitt's best selling Freakonomics and Venkatesh's Gangleader for a Day and have become generalised both in terms of all drug dealers and all places.  Although I am not a great fan of either book at least reading them meant I could spot the dodgy source being used by PC Hart as having established that the children knew what a fact was then provided them with at best spin, at worse a lie.  The sad news is I only heard it on the radio and was not present to point out to the kids the dangers of trusting what a police officer tells them.  A valuable lesson in Lambeth!

Statistics - Do they just make them up? Part One

One of the advantages of spending an unhealthy proportion of your life absorbing information about the criminal justice system is you can often spot the origins of statistics that appear out of no way.  Two examples:

Yesterday the Government issued a press release announcing "Government is tackling homelessness to stop the revolving door of re-offending".  Reading through this rather poor attempt to spin (failing) policies inherited from the previous Government I came across this claim:

Getting ex-prisoners into stable homes could reduce re-offending rates by as much as a fifth.

This claim normally phrased as "almost 20 per cent" can be found in almost any document about the accommodation of homeless ex-prisoners. But where does it come from? 

Whilst in the last 5 or 6 years I have never seen this claim referenced it was footnoted in the 2004 Home Office publication "Reducing Re-offending: National Action Plan" which identified the source as

HO OAsys pilot study, 2001 (unpublished). Data only covered 1 year after release and studied those with severe accommodation problems.

So a small unrepresentative survey, not worth publishing, is nearly ten years later the evidence base that is being relied upon.

Even more interesting is that whilst there is considerable evidence that housing is a key factor in allowing ex-prisoners to escape the criminal justice system the Home Office and the Ministry of Justice have not commissioned the research to provide an adequate evidence base on which to base policy.

OAP hoodies rampage through Lancashire

I saw this story last month in the Lancashire Evening Post - Hundreds of 'OAP hoodies' commit crimes in Lancashire.

It presented the image of a crime plague by elderly yobs. They had discovered that 550 pensioners have been arrested in the past two years with 320 charged. Nigel Evans, a local right wing Tory MP claimed  "The figures will help to explode the myth that anyone over the age of 65 with a bus pass is fine" and John O'Reilly, chairman of Lancashire Police Federation declared "people will be surprised at the number of plus-65 hoodies we have in the county."

The article ends by telling us that Lancashire's Jails have 28 prisoners over 70. To put this in some context Lancashires population has 125,000 people over 70.

Although this journalism is designed to be sensational and distort the reality the rising number of old people being caught up in criminal justice system and ending up in prison is a real concern. The over 60s are the fastest growing group of prisoners. The Prison Reform Website identifies these key facts about Senior prisoners

• On 30 June 2007 there were 2,221 prisoners aged over 60 in England and Wales, including 405 over 70. The number of sentenced prisoners aged 60 and over rose by 169% between 1995 and 2005.
• More than one in ten older prisoners belong to a minority ethnic group, far higher than the proportion of the general population.
• The majority of men in prison aged 60 and over (56%) have committed sex offences. The next highest offence is violence against the person (20%) followed by drug offences (11%).
• In March 2007, the number of prisoners over 60 serving sentences of 1-5 years was 541. 551 were serving sentences of 6-10 years. 749 were serving sentences of over 10 years.
• The number and proportion of men aged over 60 sentenced to prison by the courts has increased significantly. Between 1995 and 2000 the number of elderly males given custodial sentences increased by 55%. In 1995 fines accounted for the majority of sentences (31%). By 2000 imprisonment accounted for the majority of sentences (31%) and fines accounted for 24%.
• The significant rise in the number of male prisoners aged over 60 is not matched by a corresponding rise in the number of men convicted by the courts for indictable offences. Between 1995 and 2000 the number of convictions for this age group increased by only 8%.
• The increase in the elderly prison population is not explained by demographic changes, nor can it be explained by a so-called ‘elderly crime wave’. The increases are due to harsher sentencing policies which have resulted in the courts sending a larger proportion of criminals aged over 60 to prison to serve longer sentences. This has particularly been the case in relation to sex offenders and drug traffickers. The courts are also tending to imprison those older offenders whose crimes most challenge society’s age-related stereotypes.
• A Department of Health study conducted in 1999/2000 of 203 sentenced male prisoners aged 60 and over in 15 establishments in England and Wales (about one-fifth of that total population) reported that 85% had one or more major illnesses reported in their medical records and 83% reported at least one chronic illness or disability when interviewed. The most common illnesses were psychiatric, cardiovascular, musculoskeletal and respiratory.
• More than half of all elderly prisoners suffer from a mental disorder. The most common disorder is depression which often emerges as a result of imprisonment.
• In 2003, 21 people aged over 65 died of natural causes whilst in prison.
• Most older prisoners are held more than 50 miles from home, causing particular problems for visitors, many of whom are themselves older people.
• A thematic review of older prisoners by HM Inspectorate of Prisons published in December 2004 found little evidence that their individual needs were being assessed or provision made for them. It concluded ‘Prisons are primarily designed for, and inhabited by, young and able-bodied people; and in general the needs of the old and infirm are not met. A number of academic studies and a report by the Prison Reform Trust and the Centre for Policy on Ageing have also concluded that the health, social care, rehabilitation and resettlement needs of older prisoners are not being satisfactorily met.
• Despite the dramatic rise in the number of elderly prisoners the Home Office has no plans to put in place a separate national strategy for elderly prisoners. The Department of Health is developing a health policy for older prisoners and the Disability Discrimination act (2005) now applies to prison.

Fox News doesn't add up

Ben Goldacre's Bad Science on the quality of Home Office Research

Home Office research so feeble someone ought to be locked up

"We'd all like to help the police do their job well. They, in turn, would like to have a massive database with DNA profiles from everyone who has been arrested, but not convicted of a crime.

We worry that this is intrusive, but some of us are willing to make concessions on our principles and the invasion into our privacy in the name of preventing crimes. To do this, we'd like to know the evidence on whether this database is helpful, to help us make an informed decision.

Luckily, the Home Office has now published a consultation paper on the subject. They defend their database by arguing that innocent people who have been arrested are as likely to commit crimes in the future as guilty people. "This," they say, "is obviously a controversial assertion." That's not true: it's a simple matter of fact, and you could easily assemble some good quality evidence to see if it's true or not.

The Home Office has assembled some evidence. It is not good quality. In fact, this study from the Jill Dando Institute, attached to their consultation paper as an appendix, is possibly the most unclear and poorly presented piece of research I have ever seen in a professional environment. Or am I having a bad day? Join me in my struggle to understand their work.

They want to show that the level of criminal activity in a group of people who have been arrested, but on whom no further action has been taken, is the same as the level of criminal activity in people who have been arrested and convicted of a crime, or who accept a caution. On page 30 they explain their methods haphazardly. They describe some people sampled on 1 June 2004, 1 June 2005 and 1 June 2006. These dates are never mentioned again. They then leap to talking about Table 2. This contains data on people each from a sample in 1996, 1995, and 1994, followed up for 30 months, 42 months, and 54 months respectively. Are these anything to do with the people from 2004, 2005, and 2006? I have no idea.

In fact, I have no idea what "sample" means. Crucially, I also don't know what the numbers in the table mean, because they don't explain this properly. I think it is the number of people, from the original group, who have subsequently been arrested again.

Anyway. Then they start to discuss the results from this table. They say that these figures show that arrested non-convicted people are the same as convicted people. There are no statistics conducted on these figures, so there is absolutely no indication of how wide the error margins are, and whether these are chance findings.

At a few hundred people, this study seems pretty small for one that is supposed to give compelling evidence that there is no difference between two groups – to prove a negative like this, you'd generally want a large sample.

This research was incomprehensible and unreadable. Anybody who claims to have been persuaded by the data quoted here is telling you, loudly and clearly in the subtitles, that they don't need to understand a piece of research in order to find it compelling. If research of this calibre is what guides our policy on huge intrusions into the personal privacy of millions of innocent people, then they might as well be channelling spirits."

The Home Office Document can be accessed here