For example, after the Christchurch terrorist attack in New Zealand in 2019, hundreds of visually different versions of the livestream video of the atrocity circulated.
Therefore, to address this, matching-based tools typically use perceptual hashes rather than cryptographic hashes. A hash is similar to a digital fingerprint, and a cryptographic hash acts like a secure, unique ID tag. Changing just one pixel in an image significantly alters its fingerprint and prevents false matches.
Perceptual hashing, on the other hand, focuses on similarity. Small changes such as pixel color adjustments will be missed, but it will identify images that have the same core content. This makes perceptual hashing more resilient to small changes to a piece of content. But this also means that the hash is not completely random and can be used to recreate the original image.
2. Classification
The second approach relies on content classification. Machine learning and other forms of AI such as natural language processing are used. To achieve this, the AI needs many examples, such as text that has been labeled as terrorist content or text that has not been labeled by human content or his moderators. By analyzing these examples, AI learns the features that distinguish different types of content and can uniquely classify new content.
Once trained, the algorithm will be able to predict whether a new content item belongs to one of the specified categories. These items may be removed or flagged for human review.
However, this approach also has its challenges. Collecting and preparing large datasets of terrorist content to train algorithms is time and resource intensive.
Training data can also quickly become outdated as terrorists use new terminology or discuss new world events and current events. Algorithms also have difficulty understanding context, such as subtlety or sarcasm. There is also a lack of cultural sensitivity, with different groups having different dialects and language usage.
These limitations can have important offline implications. Countries such as Ethiopia and Romania have documented failures to remove hate speech, while free speech activists in countries such as Egypt, Syria and Tunisia have reported having their content removed.
Human moderators still needed
So, despite advances in AI, human input remains essential. This is important for maintaining databases and datasets, evaluating content flagged for review, and operating an appeals process should you dispute a decision.
But it’s a grueling, back-breaking job, and many tech companies like Meta outsource this work to third-party vendors, leading to damning reports about moderators’ working conditions.
To address this, we recommend that companies hiring content moderators develop a set of minimum standards that include mental health provision. There is also the possibility of developing AI tools to protect the health of moderators. This works, for example, by blurring areas of the image, allowing moderators to reach decisions without directly looking at the disturbing content.
But at the same time, platforms that have the resources necessary to develop automated content moderation tools and hire a sufficient number of human reviewers with the necessary expertise are There are very few.
Many platforms are moving to off-the-shelf products. The content moderation solutions market is estimated to be worth $32 billion by 2031.
But you need to be careful here. Third-party providers currently do not receive the same level of oversight as the technology platforms themselves. Insufficient human input and lack of transparency regarding the datasets used to train algorithms can lead to over-reliance on automated tools.
Therefore, collaborative efforts between government and the private sector are essential. For example, the EU-funded Tech Against Terrorism Europe project has developed valuable resources for technology companies. In some cases, automated content moderation tools have been made publicly available, such as Meta’s Hasher-Matcher-Actioner, which companies can use to build their own databases of hashed terrorist content.
International organizations, governments, and technology platforms must prioritize the development of such collaborative resources. Without this, effectively combating online terrorist content will remain difficult.
This article is republished from The Conversation under a Creative Commons license. Read the original article.
Stuart McDonald received funding from the EU Internal Security Fund for the Tech Against Terrorism Europe (ISF-2021-AG-TCO-101080101) project.
Ashley A. Mattheis receives funding from the EU Internal Security Fund for the Tech Against Terrorism Europe project (ISF-2021-AG-TCO-101080101).
David Wells received funding from the Council of Europe to carry out analysis of emerging patterns of misuse of technology by terrorists (ongoing)
