Cathy: When I say models discriminate, in fact that is the goal of models. Models are setup to discriminate. The question is whether it’s doing it illegally or unethically. I would consider the legally protected groups. I don’t think it’s a full list but it is at least a partial list. For example, let’s look at recidivism risk models for sentencing, like the Arnold Foundation studies. If there was a white guy and a black guy who have the same history of violence, do they get the same risk model? Do they get the same risk score?

The most direct way of figuring it out is if you have race on hand but you probably don’t want to use race as one of the characteristics of an individual if you are assessing an individual for creditworthiness or various other kinds of risk. Generally speaking people don’t use race, but they do use proxies for race all the time. Without even thinking about it, they will use zip code for example. People who are a little smarter than that say oh, zip code is obviously a proxy for race because our country is so segregated, so they don’t use zip code. But, without really processing it, they use other things as subtle proxies for race.