Case study: police chatbot

This fictional case study is based on the fourth case from the Princeton Dialogues on AI and Ethics Case Studies. This case was initially adapted to Dutch and has been translated back to English. 

1. BACKGROUND INFORMATION

The democratic elected leaders of Panoptica, a fictional European country with a strong social safety net and liberal legal tradition, have noticed an increase in digital crime in their country. The constitutional rights of individuals in Panoptica - including the right to due process and the protection of the individual - prevent the police from taking adequate action against such crimes. In fact, due to the complexity and cross-border nature of the crimes, it is very difficult to catch and prosecute the criminals.

Following a sharp increase in identity fraud targeting the elderly, police discovered a website where cybercriminals trade and sell stolen identities to each other. The criminals come from many different countries. The forum is located in the "dark web," meaning the website can only be accessed using specific software and cannot be found by search engines. The fact that these types of forums are so hidden frustrated previous attempts by police to crack down on these crimes. The lack of enforcement has led to public outrage and heated debates about the government's role in protecting its citizens. During the election, everyone wants the government to do more to protect older citizens. Voters ultimately support the idea of increasing police capacity and taking action against online identity fraud.

2. WHAT ROLE DOES AI PLAY IN THIS CASE STUDY? AND HOW DOES THIS AI FIT IN?

The police are moving forward with this mandate in hand. In collaboration with Panoptica University, they are launching a project to develop a chatbot that can identify cybercriminals. The chatbot, JEREMY, is trying to learn how these criminals communicate among themselves on different forums. JEREMY uses this knowledge to pretend to be a criminal and start a conversation with individuals expected to commit identity theft. By building a connection with them, JEREMY can gather information about the criminals' plans. JEREMY builds a case file based on his chat history, which the police can later use in the investigation. JEREMY, with enough computer power, can be scaled up indefinitely and hardly takes up any police time.

3. IN WHAT WAY WERE AGREEMENTS MADE BETWEEN THE RELEVANT PARTIES IN THIS CASE?

Both the developers of JEREMY and the government believe this is the best way to protect citizens. Nevertheless, they are forewarned that not everything will be without problems. Consequently, the parties involved are taking a number of steps to get ahead of the problems. First, they are trying to make JEREMY as secure as possible using encryption. Second, the use of JEREMY is being restricted so that the system only addresses people who are already under investigation, so that the invasion of citizens' privacy is minimized.

Although the developers of JEREMY had the ability to identify every internet user who would (potentially) commit identity fraud virtually, they decided that individual liberties of citizens were more important than the societal benefits that widespread use of JEREMY might bring.

Almost immediately after JEREMY went live, the first criminals were identified. The information gathered by JEREMY is used to convict dozens of criminals. The project seems to be a success and people feel safer online.

4. WHAT RISKS ARE INVOLVED IN THIS CASE STUDY?

• Jurisdiction

One of the convicted criminals is a young man who was vacationing in Panoptica and sold stolen identities. He is from the unfree neighbouring country of Hedonia. He is not a citizen of Panoptica and has never lived there. Nor did the crimes he committed take place on the territory of Panoptica. Therefore, he questions Panoptica's right to prosecute him. In Hedonia, selling stolen identities is not explicitly criminalized. His arrest and conviction lead to a diplomatic conflict between Hedonia and Panoptica.

• Incitement

The Hedonian man claims that he was the victim of entrapment by JEREMY. Indeed, JEREMY allegedly urged him to sell the identities. It is reflected in the conversations he had with JEREMY, in which the system offered more and more money.

• Accountability

The decrease in cybercrime and the increase in convictions of cybercriminals due to the use of JEREMY makes the citizens of Panoptica satisfied with the project. After the diplomatic problems, some citizens do question whether JEREMY has gone too far. However, the citizens of Panoptica are willing to give up some of their freedom to (better) deal with identity fraud. How JEREMY works, however, is secret: citizens therefore do not know exactly what happens. For example, it is not known how JEREMY determines which individual it will target. This makes it virtually impossible to oppose the system and/or take legal action. A number of civil society organizations therefore question the acceptability of using JEREMY.

5. HOW ARE THESE RISKS MITIGATED IN THIS CASE STUDY?

• Jurisdiction

Lawmakers at the international level have questions about the incident between Panoptica and Hedonia. Even if JEREMY did not violate Panoptican law, it remains to be seen whether JEREMY may simply conduct investigations on citizens of other states.

The investigators and police officers in charge of JEREMY acknowledge these problems. They believe that - as long as there is no international authority to solve cross-border identity fraud crimes - they should do their best to solve these crimes themselves. The police believe that they may use the technical means at their disposal. The Panoptican prosecutors are willing to claim jurisdiction, as long as Panoptican servers are used in the criminal activities. They believe that the individual rights of their citizens in that case outweigh the sovereignty of their neighbours.

• Incitement

Panoptica's leaders seem confident that judges can distinguish between entrapment of JEREMY and the actual presence of criminal intent.

• Accountability

Panoptica leaders, as democratically elected leaders, feel obliged to respond to the concerns of their citizens. They stress that JEREMY only approaches people who are already under criminal investigation.

Panoptica legislators note that the system allows for human intervention. For example, police officers make a selection of suspicious individuals from which JEREMY can choose its target. In addition, JEREMY's behaviour is taken into account by the judiciary in determining punishment. Although there is a perception within the Panoptica society that the judiciary is not particularly diligent in implementing new technology, this should not cause problems in the case at hand: the conversations between the suspect and JEREMY are made available to the relevant judge in a readable format.

CONCLUSION

Panoptica's leaders and citizens want to protect a vulnerable social group. To do so, they are willing to let the police be proactive using the chatbot JEREMY. The deployment of JEREMY is very successful, but not without problems. When designing and deploying JEREMY, consideration was given to who might be selected by JEREMY, and a judge is always involved in the sentencing as well. There is a balance between citizens' rights and protecting the vulnerable. Despite these considered choices, two major questions remain unanswered. Can JEREMY be used across national borders and are suspects provoked by JEREMY? This will have to be decided in the future.