Community of AI researchers and practitioners calls for rigor in the Toronto Police Services Board’s policy on the use of AI technologies

It is the first of its kind in Ontario and among police boards and commissions in Canada. In September 2021, the Toronto Police Services Board (TPSB) issued an open call for consultation to draft a policy that will govern the use of AI technology.

While recently there have been submissions in favor of PIPEDA reform for the development of a AI regulatory framework in Canada, there is no current legislation that specifically regulates AI in the province of Ontario. The TPSB also recognizes that without comprehensive guidelines on the use of artificial intelligence in policing, substantial risks to “the privacy, rights and dignity of individuals and communities” may materialize.

This follows a backdrop in recent years where governments and private sectors have rushed, globally, to leverage AI technology to efficiently manage large amounts of information for to make more effective decisions. technologies such as Account software, used by the New York City Police Department to predict crimes, or facial recognition technology in Video surveillance in the UK, are examples of a few that are currently in place. What is evident are the human rights implications and accountability for which the institutions deploying them have given little consideration.

A group of AI researchers and industry practitioners within Aggregate intelligence (AISC), which brings together the community of AI practitioners to accelerate innovation, responded to the call for projects (Overall Intelligence Submission (AISC)). Its members, who represent various industry sectors and academic disciplines, in various capacities including AI research, engineering, governance and ethics, came together and Amir Feizpour, CEO of Aggregate Intellect, launched the discussion between the owners of AI Ethics Stream, Willie Costello and Somaieh Nikpoor.

According to Nikpoor, the timing of this call for comments was fortuitous:

“I have found the TPSB’s call for public comment to be a great opportunity to influence the development of important public policy and have a positive impact. AI technologies impact our daily lives. I am concerned about the ethical risks of AI, but we should think beyond fairness and algorithmic bias to identify the full range of effects of AI technologies on security, privacy, and security. society in general. Our community has recognized the need to step up and engage meaningfully in this initiative.

For Feizpour, he points out that the responsible use of AI is an important component of the Aggregate Intellect platform.

“In many cases, we act as a think tank on the fair and ethical use of AI as well as the technical, engineering and product aspects. So when we saw this, it was really a call for us, to put all our talk and thought into action on a very important matter of public safety and privacy.

An important objective of the draft policy is to ensure that new technologies”do not introduce or perpetuate biases, including those against vulnerable populations, into police decisions.” Notable technologies like PredPol, which have been used in the United States for the past 2 years to predict crime revealed themillions of crime predictions”… show Predpol “mostly avoided white neighborhoods and targeted black and Latino neighborhoods.”

In the to study initiated by The Markup and Gizmodo, more than five million predictions made by PredPol have been analyzed. The “persistent patternsrevealed that neighborhoods that received recommendations of “few patrols” tended to be “Whiter and more middle to upper income“, while those targeted for increased patrol were more likely to be”home to blacks, Latinos and families who would qualify for the federal free and reduced meals program”.

Trust in the software’s recommendations resulted in communities that were “targeted “relentlessly”: crimes were predicted every day, sometimes several times a day, sometimes in several places in the same neighborhood: thousands and thousands of crime predictions over the years.“PredPol has been judged”biased by proxywithin the law enforcement community. Law enforcement agencies that rely on technology to get it right are seeing the downstream effects that lead to misguided over-surveillance in areas of concern, increased arrest rates and outright privacy violations. and civil rights – harms that will continue over time.

This is a clear example of the unintended bias that can creep into machine learning models. When parameters are included that are proxies (for example, zip code is a proxy for race or education) that favor certain locations; or when the input data is not correctly represented by the target population and instead favors criminal populations; or where the data comes from periods and/or geographical areas of higher identified discriminatory practices; or when the data is incomplete or incorrect – these implicit biases will lead to inaccurate model results. More importantly, the implications for the fundamental rights of individuals can be far-reaching.

Aggregate Intellect’s response clarified that bias should not be defined as “consistently wrong output” as it represents “an overly narrow mathematical definition of bias, but should be broadened to include, among others: historical biases, measurement, representation and population”.

In their summary, the Aggregate Intellect community was explicit in their response:

“Any AI technology” where the training or transactional data is known to be of poor quality, carrier of bias, or where the quality of such data is unknown” should ever be considered, and should therefore be considered a risk extreme, not high risk Any AI technology based on poor quality or biased data is inherently compromised.

“No AI technology that helps to ‘identify, categorize, prioritize or otherwise make decisions about members of the public’ should be considered low risk. Automating such actions through technology, even with the including a human in the loop, is an inherently risky activity and should be classified as such by policy.

Further, they point out that technology that has created a significant cause for concern or has been explicitly rejected by the citizens of Toronto should automatically satisfy and therefore be classified as an extreme risk.

Costello included the importance of data quality:

“The draft policy did not include any checks or requirements to ensure data supply and quality. Our response, on the other hand, emphasizes the importance of data quality in the development of reliable AI systems and recommends an assessment of data quality by independent and disinterested specialists. We further recommend that any data collected through social media (which is known to be of low quality and amenable to bias) should not be considered a viable source.

It has been argued that the AI ​​algorithms are protected by intellectual property laws, which has made it difficult for accusers to challenge the results. the Black box paradox has been questioned as more government, privacy and human rights activists demand greater transparency that justifies model results and moves towards fairness. Especially in a sector that has the potential to undermine human rights, criminal justice systems, many will say, need to be subjected to greater scrutiny. The TPSB considered high-risk technologies as those “systems that cannot be fully explained in their behavior”.

Explainability is particularly key to building trust between the community and the police, however, Aggregate Intellect noted that “explainability” was not defined in the policy and warned that it would be “potentially setting a bar that most AI technologies wouldn’t cross… or setting the wrong bar, which wouldn’t actually ensure that the behavior of the technology is meaningfully justified”. They called for related concepts of “interpretability and justificationalso to be defined to include representative players from industry, civil society, social sciences and IT.

Accountability in AI systems means continuous auditing and monitoring as more data is ingested and the model continues to evolve. Aggregate Intellect was not satisfied with the proposed monitoring and reporting schedule. Instead, they opted for continuous monitoring throughout the deployment mandate, with reports submitted within 3 months of initial deployment.

According to Costello:

“Our fundamental problem with the draft policy was that its monitoring schedule was not frequent enough. Given the scale and impact that AI technologies can have, as well as the long-term risks of model drift and deterioration, we have not seen fit to limit monitoring to just 12 months. after deployment and wait up to 15 months for reports. months after deployment.

This is a plausible first step. Although the Toronto Police Services Board’s draft policy emphasizes the role of humans in evaluating a recommendation made by extreme/high risk technologies before follow-up action is taken, it does not address the complexity of this task. Nikpoor pointed out that an AI system with a human in the loop requires a wide variety of tasks at different complexities, which requires knowledge and understanding of various disciplines.

Although Costello noted that no policy will be foolproof, he stressed the need to actively involve the public throughout the decision-making process, as their recommendations aim to do. According to Nikpoor, this is a work in progress:

“There are always risks associated with AI technologies that we cannot predict and sometimes those risks can be significant. Public engagement at all stages of AI development and deployment and gathering feedback from a wide range of audiences not only ensures that all voices and perspectives are heard, but also helps to identify emerging risks.

Read done Presentation of Global Intelligence (AISC) at TPSB.

Note: The views presented in this answer are the opinions of the individuals and do not reflect the views of their employers listed below.

Willie Costello, Data Scientist, Shopify; AI Ethics Stream Owner, Aggregate Intellect

Somaieh Nikpoor, Owner of AI Ethics Stream, Aggregate Intellect

Daria Aza, Data Analyst, Manulife

Anh Dao, co-founder, Healthtek

Aaron Maxwell, Policy Analyst

Sai Kumar Nerella, Machine Learning Engineer, Talem Health Analytics

Jay Sheldon, Principal Consultant,, President @Dialography

Amir Feizpour, CEO, Aggregate Intellect

James G. Williams