Being Aware of Bias in Data Science
My first exposure to the dangers of poorly implemented data science was from an episode of the podcast, “99 Percent Invisible” called “The Age of the Algorithm.” The episode is an interview with author and former data scientist Cathy O’Neil. O’Neil warns against badly designed algorithms which she calls Weapons of Math Destruction, the title of her book.
O’Neil argues that these dangerous algorithms are marked by three things. Firstly, they are widely used and important. Next, the intricacies of the algorithms are kept secret. Lastly, these algorithms are destructive.
Several of these ‘WMDs’ are misused in the criminal justice system. O’Neil explains that predictive policing algorithms are often very dangerous. These algorithms are supposed to take crime statistics and tell police where they should station their officers to reduce crime. However, no one has data on every crime committed, so these algorithms rely on arrest data. Because the police are taking in arrest data, they end up over-policing areas where they tend to make arrests.
Another dangerous use of algorithms in the criminal justice system is in sentencing. States started giving “recidivism scores” to judges to help guide them when sentencing prisoners. The scores were intended to limit racism by judges, who have historically impose 20% longer sentences for black men than for white men who have committed the same crimes, according to the ACLU. However, these scores factored in data such as whether the defendant grew up in a high crime neighborhood or whether they have a family member in prison. These data points can often serve as a proxy for race, and make the algorithm just as problematic as the judges it was designed to replace.
These algorithms are often built by companies and then sold to states, with the intricacies kept secret from the public. This system leaves the companies and the justice system immune from accountability.
More dangerous uses of data come in hiring. Amazon built an AI designed to look through many applicants to a new job and find the most qualified person. After the AI had already been in use they realized that it intentionally would discriminate against women. The AI would look for the word “women’s”, as in “captain of the women’s rowing team,” and eliminate those candidates. It also would weed out applicants who attended two all-women’s colleges. Obviously, in an already male-dominated industry, this would even further exacerbate the problem.
The important thing to keep in mind is that every algorithm carries with it the biases of the person who created. We, as aspiring data scientists, should interrogate our own biases and think about how they affect our work.
Sources: https://99percentinvisible.org/episode/the-age-of-the-algorithm/
