Content Moderation: What It Is, How It Works, and the Best APIs

In 2017, several major brands were up in arms when they found their advertising content had been placed next to videos about terrorism on a major video sharing platform. They quickly pulled their ads but were understandably concerned about any long term impacts this mistake would have on their company’s image.

Obviously, this poor ad placement is something brands want to avoid—then and now. But with the explosion of online communication through videos, blog posts, social media, and more, ensuring crises like the one mentioned above don’t happen again is harder than one would think.

Many platforms turned to human content moderators to try and get ahead of this problem. But not only is it impossible for humans to manually sift through and vet each piece of content—there are around 500 million tweets sent on X (formerly Twitter) each day alone—many moderators have found their mental health being negatively affected by the content they examine.

Thankfully, recent major advances in Artificial Intelligence research have made significantly more accurate, automated Content Moderation a reality today.

This article will look at what AI-powered Content Moderation is, how it works, some of the best APIs for performing Content Moderation, and a few of its top use cases.

What is Content Moderation?

Content Moderation models use AI to detect sensitive content in bodies of texts, including those shared via online platforms or social media. Content Moderation can also be performed on audio and video data using top Speech-to-Text APIs. This allows video platforms like YouTube and podcast platforms like Spotify to also use AI-powered Content Moderation. 

Typically, the sensitive content Content Moderation models can detect includes topics related to drugs, alcohol, violence, sensitive social issues, and hate speech.

Here’s an example of what might be included as “sensitive content” by a Content Moderation model:

Once detected, platforms can use this information to automate decision making regarding ad placements, content acceptance, and more. The definition of what is acceptable or not acceptable may vary across platforms and industries, as each comes with its own set of rules, users, and needs.

How Does Content Moderation Work?

Content Moderation models are typically designed using one of three methods: generative, classifier, and text analysis.

A generative model takes an input text and generates a list of Content Moderation topics that may or may not be included in the original text. For example, a generative model might label the input text He had a cigarette after dinner as containing references to tobacco.

A classifier model would take an input text and then output a probability that the text conforms with a predetermined list of sensitive content categories. For example, a simple classifier Content Moderation model could be designed with three possible outputs– hate speech, violence and profanity. Then, the model would output a probability that the text conformed to each of the above possibilities.

Finally, a general text analysis model could be used for Content Moderation. With this method, one would use a “blacklist” approach and create a mini dictionary of blacklisted words for each predefined category such as crime or drugs. If the input text fed through the model contained one of these listed words, it would categorize the text according to the category that word was listed under. This approach has its limitations given that, for example, creating exhaustive lists for each category may prove challenging. A text analysis model may also miss out on important context that could help more accurately categorize the word.

Content Moderation Use Cases

Content Moderation has significant value across a wide range of brand suitability and brand safety use cases.

For example, smart media monitoring platforms use Content Moderation to help brands see if their name is mentioned next to any sensitive content, so they can take appropriate action, if needed.

Brands looking to advertise on YouTube can use Content Moderation to ensure that their ads aren’t placed next to videos containing sensitive content.

Content Moderation APIs also help:

• Protect advertisers
• Protect brand reputation
• Increase brand loyalty
• Increase brand engagement
• Protect communities

Top Content Moderation APIs: A Comparative Overview 

In the next section, we compare six popular Content Moderation APIs across type, capabilities and pricing, as well as discuss some of the pros and cons to using each. 

Ultimately, choosing a Content Moderation API depends on your use case—some APIs interact purely with text inputs, like social media feeds, while others are adept at handling audio and video inputs, like YouTube. Other models can identify potentially harmful content in images as well.

The sensitivity of the model, as well as the accuracy, will also be important determining factors depending on your use case. An open forum may need more strict content moderation than a private one, for example.

The table below provides a brief overview of each of the seven Content Moderation APIs compares across type, capabilities, and pricing. 

Top APIs for Content Moderation

Now that we’ve examined what Content Moderation is and how Content Moderation models work, let’s dig into the top Content Moderation APIs available today.

1. AssemblyAI’s Content Moderation API

AssemblyAI offers advanced AI-powered Speech-to-Text and Audio Intelligence APIs, including Content Moderation, Entity Detection, Text Summarization, Sentiment Analysis, PII Redaction, and more.

Its Content Moderation API lets product teams and developers pinpoint exactly what sensitive content was spoken and where it occurs in an audio or video file. Teams also receive a severity score and confidence score for each topic flagged.

For example, the AssemblyAI Content Moderation API found health_issues to be present in the following transcription text segment:

Yes, that's it. Why does that happen? By calling off the Hunt, your 
brain can stop persevering on the ugly sister, giving the correct set 
of neurons a chance to be activated. Tip of the tongue, especially 
blocking on a person's name, is totally normal. 25 year olds can 
experience several tip of the tongues a week, but young people don't 
sweat them, in part because old age, memory loss, and Alzheimer's are 
nowhere on their radars.

Pricing starts at $0.12 per hour for the pay as you go plan, which allow unlimited access to AssemblyAI’s Speech-to-Text, Audio Intelligence, LeMUR, and Streaming Speech-to-Text models. Developers looking to prototype with Speech AI can also get started with $50 in free credits. Volume discounts are also available for teams building at scale.

2. Azure AI Content Safety

AI Content Safety is part of Azure’s Cognitive Services suite of products. Its API can detect sensitive or offensive content in text, images, and video. Users can also use its Human Review tool to aid confidence in a real-world context.  

Pricing for the Azure AI Content Safety starts at $.75 per 1,000 images, $.38 per 1,000 text records, with limited free tier available. Human moderation is included in its standard API pricing. Those looking to try the API should review the Start Guide here.

3. Amazon Rekognition

Amazon Rekognition offers Content Moderation for image, text, and video analysis, in addition to other Audio Intelligence features such as Sentiment Analysis, Text Detection, and more. The Content Moderation API identifies and labels sensitive and offensive content in videos and texts along with an accompanying confidence score.

You will need an AWS account, an AWS account ID, and IAM user profile to use Amazon Rekognition. Pricing varies based on usage. This guide can get you started.

4. Hive Moderation

The Hive Moderation API performs Content Moderation on all media types, including images, videos, GIFs, and live streams. The API detects more than 25 subclasses across 5 distinct classes of offensive or sensitive content, including NSFW, violence, drugs, hate, and attributes, along with a confidence score. Hive’s documentation can be found here, but developers looking to test the API will have to sign up for a demo here.

5. Sightengine

Sightengine’s Content Moderation API lets users moderate and filter images, videos, and texts in real time. Users can pick and choose which models they wish to apply and create their own custom moderation rules.

Pricing ranges from $29 to $399 per month depending on usage and audio/video streams needed, with a free tier and enterprise custom pricing also available. 

6. OpenAI Content Moderation API

OpenAI’s recently updated Content Moderation API lets developers identify harmful content in text and images and then take appropriate corrective action if needed. The API can classify content across six categories: violence, violence/graphic, self-harm, self-harm/intent, self-harm/instruction, and sexual. While free to use, the API is aimed toward developer-use and does not provide a user-friendly dashboard interface like some of the other APIs discussed.

Content Moderation Tutorial

Want to learn how to do Content Moderation on audio files in Python? Check out this YouTube Tutorial:

Suggested Reads

• Top free speech-to-text APIs and open source engines
• Best APIs for sentiment analysis
• What are the top PII Redaction APIs and AI models?
• Text Summarization NLP: 5 Best APIs

Source: Read More