Content Moderation Overview

Delivering a safe and appropriate chat environment to your users is essential. Agora Chat gives you multiple options to intelligently moderate the chat content and handle inappropriate user behavior.

• The following message management tools are available:
  • Message reporting.
  • Message moderation, including text and image moderation, profanity filtering, and domain filtering.
• Message moderation can be applied globally or to selected chat groups and chat rooms.

Prerequisites

• You have created a valid Agora developer account.
• Moderation is not enabled by default. To use this feature, you need to subscribe to the Pro or Enterprise pricing plan and enable it in Agora Console.

Enable the moderation feature

1. Log in to Agora Console.
2. In the left navigation menu, click Project Management > Config for the project for which you want to enable the moderation feature > Config of Chat > Features > Overview.
3. On the Chat Overview page, turn on the switch of a specific moderation option, as shown in the following figure:
   

Message management

The following table summarizes the message management tools provided by Agora Chat:

Implement the Reporting Feature

To use the reporting feature, you need to call the reporting API when you develop your application. For details, see the following documents:

• Implement the reporting feature (Android)
• Implement the reporting feature (iOS)
• Implement the reporting feature (Web)
• Implement the reporting feature (React Native)
• Implement the reporting feature (Flutter)

After a user reports a message from the application, moderators can check and deal with the report on Agora Console:

1. To enter the Message Report page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Message Report > History, as shown in the following figure:

   

2. On the Message Report page, moderators can filter the message report items by time period, session type, or handling method. They can also search for a specific report item by user ID, group ID, or chat room ID. For the reports, Agora Chat supports two handling methods: withdrawing the message or asking the sender to process the message.

Text and image moderation

Introduction

Powered by Microsoft Azure Moderator, Agora Chat's text and image moderation can scan messages for illegal text and image content and mark the content for moderation. Microsoft Azure Moderator uses the following three categories to moderate the message and returns a confidence score between 1 and 5 for a message:

• Adult: The content may be considered sexually explicit or adult in certain situations.
• Racial: The content may be considered racial discrimination.
• Offensive: The content may be considered offensive or abusive.

After enabling the text and image moderation feature on Agora Console, you can set a threshold for each moderation category. When the score returned by Microsoft Azure Moderator reaches the threshold, Agora Chat blocks the message. You can also impose a penalty on users who reach the violation limit within a time period. The moderation penalties include the following: banning the user, forcing the user to go offline, or deleting the user.

To see how moderation works and determine which moderation settings suit your needs, you can test different text strings and images on Agora Console.

Create a moderation rule

Taking a one-to-one chat text as an example, follow these steps to create a text moderation rule:

1. To enter the Rule Config page, in the left navigation menu, click Project Management > Config for the project for which you want > Config of Chat > Text Moderation or Image Moderation > Rule Config, as shown in the following figure:

   

2. To create a text moderation rule, click Add:

   

   The following table lists the fields that a text moderation rule supports:

   Field	Description
   Rule name	The rule name.
   Conversation type	The moderation scope, which can be one of the following: a one-to-one chat, chat group or chat room, chat groups or chat rooms. If you set a rule for a specific chat group or room, the global moderation rules for chat groups and rooms are overwritten.
   Enable	Determines whether to turn a rule on or off.
   Message handling	When the moderation result is Rejected, you can set the action on the moderated message to one of the following options: Blocks the message. Agora recommends using this setting for online messages after you fully test this moderation rule and ensure it suits your needs. (Default) Sends the message. Agora recommends using this setting when you are testing this moderation rule and do not want this rule to affect online messages. Replaces the message with ***. When the moderation action fails (for example, the timeout period of the text audit interface is 200 milliseconds; if no result is returned within 200 milliseconds, this moderation action times out), the message processing policy can be set to one of the following options: Blocks the message. (Default) Sends the message. If the action on the moderated message is set as Blocks the message, the 508, MESSAGE_EXTERNAL_LOGIC_BLOCKED error is returned after the message is blocked. You can set whether to indicate this error in the application. If you choose to indicate this error in the application, a red exclamation mark is displayed before the blocked message.
   Rule	Sets the threshold for each moderation category.
   User management	Imposes a penalty on users who reach the violation limit within a time period. The moderation penalties include the following: banning the user, forcing the user to go offline, or deleting the user.

3. After creating a rule, you can edit or delete the rule:

   

Conduct a text or image moderation rule test

1. To enter the Rule Test page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Text Moderation or Image Moderation > Rule Test, as shown in the following figure:

   

2. Select a rule, fill in the text to moderate, and click Check to test the rule. The moderation result is displayed on the same page.

Profanity filter

The profanity filter can detect and filter out profanities contained words in messages according to configurations you set.

Follow these steps to specify a profanity filter configuration:

1. To enter the Rule Config page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Word List > Rule Config, as shown in the following figure:

   

2. On the Rule Config page, you can add or delete words and determine which filtering method to apply to messages that contain the specified keywords. You can replace the word with *** or simply not send the word.

Domain filter

The domain filter can detect and filter out certain domains contained in messages according to configurations you set.

Follow these steps to specify a domain filter configuration:

1. To enter the Rule Config page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Domain Filter > Rule Config, as shown in the following figure:

   

2. To create a domain filtering rule, click Add:

   

   The following table lists the fields that a domain filtering rule supports:

   Field	Description
   Rule name	The rule name.
   Conversation type	The moderation scope, which can be one of the following: a one-to-one chat, chat group or chat room, chat groups or chat rooms. If you set a rule for a specific chat group or room, the global moderation rules for chat groups and rooms are overwritten.
   Enable	Determines whether to turn a rule on or off.
   Message handling	Blocks messages containing the domain. Only allows messages containing the domain to pass. Replaces the domain in the message with ***. Takes no action on the moderated message.
   Domain name	Adds a domain to the rule.
   User management	Imposes a penalty on users who reach the violation limit within a time period. The moderation penalties include the following: banning the user, forcing the user to go offline, or deleting the user.

3. After creating a rule, you can edit or delete the rule:

   

Check the moderation history

You can check the history of the text moderation, image moderation, profanity filter, and domain filter on Agora Console. You can filter the moderation items by the time period, session type, or moderation result. You can also search for a specific item by the sender ID or receiver ID.

User management

You can impose a penalty on users for repeated violations. The penalties can be applied as a global application setting, or only to a specific chat group or room. The following table lists all the user moderation options that Agora Chat supports:

Take global actions on users

1. To enter the User Management page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Operation Management > User, as shown in the following figure:

   

2. To take action on a user (banning a user, deleting a user, or forcing a user to go offline), search for the user ID, and click More:

   

Take actions on chat group members

1. To enter the Chat Group Management page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Operation Management > Group, as shown in the following figure:

   

2. To take action on a group member (removing a member from the group or adding a user to the group blocklist), search for the group ID, and click More:

   

3. You can also click the group ID to enter the group's moderation dashboard, where you can manage the group info, group members, and messages in real-time:

   To use this feature, you need to enable it on the Features > Overview page.

   

Take actions on chat room members

1. To enter the Chat Room Management page, in the left navigation menu, click Project Management > Config for the project that you want > Config of Chat > Chat Room Management, as shown in the following figure:

   

2. To take action on a room member (removing a member from the room or adding a user to the room blocklist), search for the room ID, and click More:

   

3. You can also click the room ID to enter the room's moderation dashboard, where you can manage the room info, room members, and messages in real-time:

   To use this feature, you need to enable it on the Features > Overview page.