« Notable Atkins capital habeas ruling from the Eighth Circuit | Main | Another significant federal child porn restitution decision from NDNY district court »

August 11, 2010

The data and quantification challenges for criminal justice work and reform

I have been noting in a number of recent posts (examples here and here) the challenges of assessing through data the on-going medical marijuana and pot prohibition reform movements.  This interesting new op-ed by Amy Bach, which is headlined "Justice by the Numbers," effectively spotlights that good data and ready means of quantification poses an enduring challenge for lots of criminal justice work and reform.  Here are excerpts from her op-ed:

In communities across the country, people use statistics on hospitals, schools and other public services to decide where to live or how to vote. But while millions of Americans deal with their local criminal courts as defendants and victims each year, there is no comparable way to assess a judicial system and determine how well it provides basic legal services.

This lack of data has a corrosive effect: without public awareness of a court system’s strengths and weaknesses, inefficiencies and civil liberties violations are never remedied.

That’s why America needs a “justice index” to show how the essential aspects of our local courts are working. The index, compiled according to national standards, would function roughly like college rankings, evaluating county courts on factors like cost, recidivism, crime reduction and collateral consequences, including whether people lose their jobs or homes after contact with the criminal justice system.

True, hospitals and schools serve everyone, while most Americans will never directly interact with a criminal court. But many will — an estimated 47 million Americans have criminal records, and though exact statistics don’t exist, it’s a good bet that similar numbers have passed through the courts as victims.

Of course, those numbers count only direct contact. We all benefit from better courts, which deter crime and remove public threats from the streets.

A justice index would be relatively straightforward to create. It would start by amassing data from the country’s 25 biggest counties, where the courts are most likely to collect large amounts of information.

Next, a panel of lawyers, community representatives, statisticians and law professors would establish standards for the measurements — for example, the percentage of people who plead guilty without an attorney or average bail amounts, because a high bail figure often compels defendants to plead guilty.

Another critical measurement would be the percentage of certain types of cases that get thrown out after a defined period of time, a possible indicator of inefficiency as well as disregard for traditionally under-prosecuted crimes. The index would also assess whether a county court has certain legal protections in place, like requiring that interrogations and confessions be taped....

Rankings for hospitals and public schools create healthy competition. To get the justice we deserve, we would do well to bring a similar approach to bear on our criminal courts.

I love the idea of some kind of “justice index,” but I strongly disagree with Amy Bach's assertion that such an index "would be relatively straightforward to create." Indeed, I think the main reason such an index does not already exist is because there are such fundamental disagreements concerning what constitutes justice, and there are particularly strong disagreements as to what are the most problematic forms of "injustice" that a jurisdiction should be trying to remedy. I would love to hear reader opinions on (1) what factors would be most important in some form of national “justice index” and (2) whether they would generally trust a "panel of lawyers, community representatives, statisticians and law professors" to be principally tasked with creating and applying such an index.

August 11, 2010 at 11:40 AM | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference The data and quantification challenges for criminal justice work and reform:


I have been working with Iowa criminal justice system data for about ten years and in Iowa the quality of the data was rather poor ten years ago and somewhat improved today. I would not be surprised if data quality of the court data will be an issue for the 25 largest counties.

To get started with the review process peer review might be a good option using retired judges, prosecutors and defense attorneys leavened with citizen representatives.

I agree with Amy that independent oversight of the courts is needed.

Posted by: John Neff | Aug 11, 2010 4:12:23 PM

Accurate statistical information is needed so that a local community can assess whether their criminal justice system is meeting expectations. I too agree that independent oversight of the courts is needed but it is "pie in the sky" thinking to believe that national standards can ever be applied to what is first and foremost a local matter.

Posted by: mjs | Aug 11, 2010 7:46:25 PM

Good thought provoking question. What is justice, what do we want from it?

Subjective rankings are not reliable, and reflect public relations, not outcomes. I would not send my dog, nor even any lawyer to the top hospitals cited. The faculty publish a lot, so name recognition is a factor. However, these hospitals are all in slums. The low level employees have really bad attitudes, poor work ethics, and bad training. And, you will never see the "Names." They are scholars. Patients are a nuisance to them. In the case of hospitals, outcomes for the specific condition are the best rating. These outcome data are impossible to get.

Justice serves to keep us safe. It serves to prevent the punishment and incapacitation of innocent people. What is the rate of crime victimization in population surveys? What is the rate of false incarceration? Poverty is absolutely not a cause of crime, so that there will not be a bias in these simple statistics. They may be combined to come up with an overall Justice Rating (1/(FCR+CVR)*100%).

Posted by: Supremacy Claus | Aug 13, 2010 6:57:44 AM

It goes without saying that this wouldn't be an easy undertaking, but I believe, in the long run, that it would pay off in spades.

Posted by: JDU | Aug 13, 2010 3:00:13 PM

A true "justice index" is indeed probably anything but simple to devise. One basic problem is that it is hard to tell what statistics reflect differences in criminal activity, and which reflect differences in the criminal justice system itself. Also, many components of the criminal justice system are only loosely coupled. Problems with jury selection and the conduct of jury trials, for example, may have little relationship to problems with the way that pre-trial release is handled.

A less ambitious, but perhaps more useful program would be to collect more basic information about a wide variety of subjects and to leave things like justice indexes to political advocates.

In general, my intuition is that data sets about unexplored questions (e.g. data sets on total institutionalized populations rather than merely criminal incarceration, data sets distinguishing technical violations from other parole violations, etc.) tend to be more valuable than better data about matters that have already received some significant study. It is less helpful to know which court system is best, than to discover previously unrecognized problems.

For example, while there is copious information available about felony sentencing and of federal sentencing of all kinds, there is very little information available about misdemeanor and ordinance violation sentencing. Similarly, there is very little good data on recidivism by people convicted of misdemeanors under state law, the demographics of people who have large numbers of misdemeanor convictions, the predictors that determine when people convicted of minor offenses will later commit serious offenses, or differences in sentencing for minor violations between jurisdictions or across demographic lines. Yet, much of the information is out there as a matter of public record, and has significant budgetary and criminal justice implications.

It would be interesting to see which factors are most empirically important to judges in sentencing for misdemeanors and property crimes, and perhaps to look for opportunities to recognize those factors by lowering statutory sentences for unaggravated offenses that usually produce low sentences in practice, while singling out offenses with factors that are important to judges in sentencing, in order to reduce unnecessary disparities across jurisdictions.

Another piece of data which would be interesting, both to those concerned about civil rights and those concerned about laxity in law enforcement, would be to identify the factors distinguish the population of people who have lots of arrests but few convictions. When and why are people arrested and not convicted? If the reasons are not good ones, what practices should be changed?

Posted by: ohwilleke | Aug 13, 2010 6:59:25 PM

Post a comment

In the body of your email, please indicate if you are a professor, student, prosecutor, defense attorney, etc. so I can gain a sense of who is reading my blog. Thank you, DAB