Risk Assessment Algorithms Remove the Individual from the Criminal Process, Says Villanova Law Professor
Villanova's Itay Ravid recently authored two papers examining the use and effects of risk assessment algorithms in the criminal legal system.
Risk assessment algorithms predicting your likely future behaviors are utilized daily across a wide range of sectors. They inform banks of potential credit risk. Insurance companies use them for underwriting. Healthcare systems apply them to predict disease progression or potential complications.
These predictive tools are even deployed in the criminal legal system, permeating every step from policing all the way to release and rehabilitation, and informing decisions along the way.
Itay Ravid, JSD, is an associate professor at Villanova University Charles Widger School of Law. He studies criminal law and procedure and recently published two papers looking at the uses and effects of risk assessment algorithms in criminal justice—one in the Boston College Law Review and the other in the Vanderbilt Law Review.
While previous studies have largely focused on their evolution and how they are used in specific stages of the criminal process, Professor Ravid is interested in how these algorithms transform criminal law conceptually—particularly how the underlying logic that drives algorithmic prediction is antithetical to the concept of subjective culpability—the backbone of criminal justice in the United States (and other Western liberal democracies).
“Technology can evolve but there is an inherent logic that drives these algorithms in criminal procedure,” he said. “That doesn't change, even if the algorithms themselves are refined and perform better.”
Assessing Risk in The Criminal Legal System
The general goal of these algorithms used in criminal justice is to predict potential recidivism and other similar risks from an individual. They do this through a complex statistical analysis that combines certain factors, creating a score that is supposed to reflect the level of risk. The algorithms—some of which are created by AI— are often developed by for-profit actors and companies.
These risk assessment algorithms and related predictive tools are used in the decision-making process of nearly all aspects of criminal justice in the United States: policing, bail, sentencing, parole, different rehabilitation paths, and even compassionate release.
A simple example would be to look at two individuals convicted of the same burglary crime. One individual is unemployed and has been arrested and convicted of burglary before. The other holds a stable job and has no prior arrests. The algorithm may deem the first individual as “higher risk” based off those factors, potentially leading to a denial of probation, higher bail, longer prison sentence or stricter parole conditions.
There are many, many factors that are considered, and they have changed over time. For instance, in early algorithms, race was utilized as a factor, but this is less explicit now with advancement in science, through public discourse and with the recognition of certain biases. The algorithms are constantly updated to account for biases, but many argue that even factors utilized today are still inherently prejudiced.
“Things like socioeconomic status and level of education can have direct correlation to biased social demographic factors,” Professor Ravid said.
Consider the burglary example. What if the individual with prior arrests lived in an area with far heavier policing due to algorithmic prediction? What if the individual with no prior arrests had actually committed more burglary offenses, but was never arrested due to living in a less policed area? And why are some areas more policed compared to others?
The concerns begin to become more evident.
First, the use of algorithms—especially ones created by for-profit actors—in the criminal legal process at all.
“Criminal justice is a government operated system, but the government kind of strips itself from the accountability on the decision-making process by using those algorithms that were created by private actors that they hired to do it,” Professor Ravid said. “This setting creates difficult questions of accountability, transparency, and due process.”
Then, there are the biases that may be factored into the algorithm, creating a biased output. Most common are racial and other similar demographic biases, affecting mostly marginalized communities. Then, the reliance on the result of the assessment itself represents a level of automation bias—the human tendency to over-rely on automated systems rather than human judgement, especially in an overworked industry such as criminal justice. Even anchoring bias, which occurs simply from having visible information presented to you.
“If you're tasked with making a decision, there's some bias that comes from just having something written on a paper that might suggest one thing or another, whether you intend to put stock in it or not,” Professor Ravid said. “Many people say, ‘Okay, as long as there is a human in the loop, these are just recommendations.’ The reality is a bit more complicated, and studies have raised concerns about tendency to rely on automated decision-making tools that inhibit meaningful human considerations.”
Professor Ravid’s recent papers look at these oft-studied issues from a different, broader lens. In all facets of the criminal legal system—which is supposed to focus on the individual—the individual is substantially being removed from the equation.
“The system de facto transforms itself into a system where people are being stripped away from their individuality and become a subject of investigation through an algorithmic assessment,” he said.
“And as long as this is expanded and being used more, I think our reliance on this will likely increase in environments like the criminal legal system, where everyone's so overworked and the system has always struggled with treating individuals within it.”
While risk assessment algorithms aren't used to directly determine guilt, Professor Ravid says they affect that decision "quite a bit."
Individual Culpability and How We Determine Guilt
Modern criminal law, Professor Ravid says, is driven by a concept called subjective culpability, which takes into account person’s mental state, encompassing things like intent, knowledge, foresight and many other highly individualistic factors.
“There were times in our history where we more focused on the act itself: if you killed a person, you should be punished,” Professor Ravid said. “It’s not like criminal law didn’t care about the mental state, but we almost assumed that the act itself suggests one had the required mental state to be charged. We didn't go as deep in figuring out what the person’s mental state was when committing the crime, or why. For example, was it self-defense? Negligence? Recklessness?
“But over time, with the evolution of science, psychology and psychiatry, we realized we have to do more on those fronts. Therefore, the idea of subjective culpability became a principle that's driving the modern criminal legal system, and it is attached to our dignity and autonomy as individuals.”
Risk assessment tools, he says, “are just clearly doing the opposite.”
“They do not care about you, the individual. They care how well you are reflected through the characteristics of others. It’s turning you from that free will individual to a predicted object, where your behavior is predicted even if you're not going to recidivate or commit a crime in the first place, so long as your characteristics are correlated with others who might.”
If done well, risk assessment algorithms could be very close to a predictor, the professor cedes.
“But there is always an outlier. I, the individual, want to be that outlier, and I want to have the power to be able to be that outlier and tell the system, ‘Hey, I'm not that person.’
And, because the criminal legal system is so intertwined, he says, the results of an algorithm can have cascading effects. A high-risk finding at the investigative-policing level could make prosecutors more attuned to if and how they charge an individual. If they decide to charge, it can affect the bail hearing and carry weight in the plea bargain offered.
“Because each step follows another, we end up with the reality that these risk assessment determinations have the potential to affect outcomes in the criminal process, even if we are not technically determining guilt through them,” Professor Ravid said. “We do not use risk assessment algorithms to directly determine guilt, but it ends up affecting that decision quite a bit.”
He points to plea bargains, mentioning how 90 percent of cases end in a plea, and how studies suggest people take plea bargains for crimes they did not commit rather than face the prospect of worse outcomes in trial.
He then references a case in which a woman was caught crossing the border with drugs in her car, claiming she did not know they were there. It is admittedly not the most convincing argument. But in order to charge someone with a drug crime, knowledge beyond reasonable doubt has to be proven by the prosecution.
“They brought in an expert who said something like, “Well, usually people in those situations tend to know that they have drugs in their car.” So the jury said, ‘Okay, other people usually know, therefore this suggests to us that she also knew.’ It’s probabilistic evidence. But because the system is becoming so used to thinking about individuals outside of their individuality through this algorithmic lens, I think it's also affecting the way we envision guilt more broadly, and this case is illustrative of that phenomenon.”
In his paper in the Vanderbilt Law Review, he discusses how risk assessment tools used throughout the criminal process can be counter-rehabilitative, directly contradicting the function, and certainly the aspiration, of many holistic rehabilitation models used today.
For instance, high risk scores could lead to decreased family visit time, or increased time in isolation while incarcerated. Individuals classified as “high-risk” can begin to “foster deeply entrenched beliefs about the inescapability of criminal identities” and “frequently undergo a process of stigma internalization, wherein biased racial or gender-based risk profiles potentially function as self-fulfilling prophecies.”
This can cause individuals to feel as though they may never be able to move on from their past, even if a component of their past is actually not entirely their own due to the predictive nature of the algorithms. They then begin to deviate from their authentic self to try to fit into the system.
“Holistic rehabilitation generally aspires to look at the individual as a whole and see what they need and what we can give them to make sure that they will re-enter back in society as functional, equal, supported members,” Professor Ravid said. “Thinking about themselves through the algorithmic lens, however, is counter rehabilitative.”
In situations where algorithms are used in predictive policing, an individual's ability to challenge the algorithm becomes far more difficult than in a courtroom setting, where a lawyer would be present.
Re-individualizing Criminal Justice
To regain autonomy and dignity lost from the use of algorithms, Professor Ravid says we must re-individualize the system. It is a very challenging task. The most meaningful way, he proposes, is through a process that will allow a right to contest algorithmic decision in the criminal process, forcing transparency and accountability and increasing the visibility of individual narratives.
“The idea is that by giving the right to contest the decision, you will be exposed to how it was decided, the factors that were considered and why you should be recognized as this outlier from the decision,” he said.
There are roadblocks. In the United States, even in a non-criminal legal system, rights to contest AI are just starting to be recognized, and if they are, they remain at the state level. It is something he says is moving forward in the civil arena, particularly with challenging automated decisions related to credit and housing through an administrative process. But he admits that while a similar approach could work in the criminal space, there are further issues.
One, circling back to the private actors creating the algorithms for profit, is that defendants may be unable to challenge scores produced by proprietary methodologies or code, because developers refuse to disclose them.
Even if they did, would the defendant or defendant’s lawyer be able to make sense of it enough to put forth the argument?
“I actually suggest that it’s the prosecutor’s office who should have experts from within, who can comfortably handle and explain what is being done,” Professor Ravid said. “This would make the system more accountable to what is decided and how.”
But not all algorithms used in criminal justice occur in a setting for which a lawyer would even be present.
“Sure, in a bail hearing you’d potentially have a lawyer, even a court-appointed one, and could maybe contest the decision. But not in the case of something like predictive policing, which is happening all the time and is kind of a free-for-all. Many people don’t have the capacity to proactively have their own representation to contest a decision in such a setting.
“As we consider all the challenges related to transparency and understanding of the data, I’m attempting to show through this work a conceptualization of how we as individuals should be understood within the system and how we should be evaluated. Because from the judiciary to prosecutors, defense lawyers—everyone involved in the system, this issue is penetrating the discourse and affecting how all of us understand, navigate, and apply key elements within our criminal legal system.”