Moderation Integration Guide

Complete step-by-step guide to integrate AI-powered content moderation

Quick Reference for AI Agents & Developers

// Moderation is automatic once enabled in Dashboard
// 1. Enable moderation in CometChat Dashboard → Moderation → Rules

// 2. Check moderation status when sending
const message = await CometChat.sendMessage(textMessage);
if (message.getModerationStatus() === CometChat.ModerationStatus.PENDING) {
  console.log("Message under review");
}

// 3. Listen for moderation results
CometChat.addMessageListener("MOD_LISTENER", new CometChat.MessageListener({
  onMessageModerated: (msg) => {
    const status = msg.getModerationStatus();
    // APPROVED, DISAPPROVED, or PENDING
  }
}));

// 4. Manual flagging by users
await CometChat.flagMessage(message);

This guide walks you through integrating CometChat’s AI-powered moderation to automatically review messages for inappropriate content and allow users to flag messages manually.

Prerequisites

• CometChat SDK already integrated (see Chat Integration Guide)
• Moderation enabled in your CometChat plan
• Access to the CometChat Dashboard

---

Overview

CometChat Moderation provides two types of content review:

Type	Description	Trigger
AI Moderation	Automatic review of text, images, and videos	Automatic on send
User Flagging	Manual reporting by users	User action

---

Step 1: Enable Moderation in Dashboard

1. Go to app.cometchat.com
2. Select your app
3. Navigate to Moderation → Rules
4. Enable moderation and configure rules:
   • Profanity filter
   • Spam detection
   • Image/video content analysis
   • Custom keywords

Moderation rules are configured in the dashboard, not in code. The SDK automatically applies these rules to messages.

---

Step 2: Understand Moderation Status

When moderation is enabled, every text, image, and video message goes through review:

// Moderation statuses
CometChat.ModerationStatus.PENDING      // Under review
CometChat.ModerationStatus.APPROVED     // Passed moderation
CometChat.ModerationStatus.DISAPPROVED  // Blocked by moderation

Message Flow with Moderation

User sends message
       │
       ▼
┌─────────────────┐
│ Message sent    │
│ Status: PENDING │
└─────────────────┘
       │
       ▼
┌─────────────────┐
│ AI Moderation   │
│ Reviews content │
└─────────────────┘
       │
       ├──────────────────┐
       ▼                  ▼
┌─────────────┐    ┌──────────────┐
│  APPROVED   │    │ DISAPPROVED  │
│ Message     │    │ Message      │
│ visible     │    │ blocked      │
└─────────────┘    └──────────────┘

---

Step 3: Check Moderation Status on Send

When moderation is enabled, messages are reviewed before being delivered. Check the moderation status after sending to know if the message is pending review, approved, or blocked.

JavaScript

/**
 * Send a message and handle moderation status
 * Messages may be immediately approved, pending review, or blocked
 * 
 * @param {string} receiverUID - Recipient's user ID
 * @param {string} text - Message content
 * @returns {Object} - Message and moderation status
 */
async function sendModeratedMessage(receiverUID, text) {
  // Create a standard text message
  const textMessage = new CometChat.TextMessage(
    receiverUID,
    text,
    CometChat.RECEIVER_TYPE.USER
  );

  try {
    // Send the message - it goes through moderation automatically
    const message = await CometChat.sendMessage(textMessage);
    
    // Check the moderation status
    // This tells you what happened to the message
    const status = message.getModerationStatus();

    switch (status) {
      case CometChat.ModerationStatus.PENDING:
        // Message is being reviewed by AI moderation
        // It will be delivered if approved, blocked if not
        // Listen for onMessageModerated to get the final result
        console.log("Message under moderation review");
        return { message, pending: true };
        
      case CometChat.ModerationStatus.APPROVED:
        // Message passed moderation immediately
        // This happens when content is clearly safe
        console.log("Message approved immediately");
        return { message, pending: false };
        
      case CometChat.ModerationStatus.DISAPPROVED:
        // Message was blocked immediately
        // This is rare on send - usually happens via listener
        // Content clearly violated moderation rules
        console.log("Message blocked");
        return { message, blocked: true };
    }
  } catch (error) {
    console.log("Send failed:", error);
    throw error;
  }
}

TypeScript

interface ModerationResult {
  message: CometChat.TextMessage;
  pending?: boolean;
  blocked?: boolean;
}

/**
 * Send a message with moderation status handling
 * Returns the message and its moderation state
 */
async function sendModeratedMessage(
  receiverUID: string,
  text: string
): Promise<ModerationResult> {
  const textMessage = new CometChat.TextMessage(
    receiverUID,
    text,
    CometChat.RECEIVER_TYPE.USER
  );

  const message = await CometChat.sendMessage(textMessage);
  const status = message.getModerationStatus();

  switch (status) {
    case CometChat.ModerationStatus.PENDING:
      // Message queued for AI review
      return { message, pending: true };
    case CometChat.ModerationStatus.APPROVED:
      // Message passed moderation
      return { message, pending: false };
    case CometChat.ModerationStatus.DISAPPROVED:
      // Message blocked by moderation
      return { message, blocked: true };
    default:
      return { message };
  }
}

What This Code Does

1. Sends Message: sendMessage() transmits the message through CometChat
2. Triggers Moderation: AI automatically reviews the content
3. Returns Status: getModerationStatus() tells you the result
4. Enables UI Updates: You can show pending indicators or blocked notices

Moderation Status Values

Status	Meaning	UI Action
PENDING	Under AI review	Show “reviewing…” indicator
APPROVED	Passed moderation	Display normally
DISAPPROVED	Blocked by moderation	Show blocked notice or hide

---

Step 4: Listen for Moderation Results

Register a listener to receive real-time moderation decisions. This is essential for updating your UI when pending messages are approved or blocked.

JavaScript

const listenerID = "MODERATION_LISTENER";

// addMessageListener() registers callbacks for message events
// including moderation results
CometChat.addMessageListener(
  listenerID,
  new CometChat.MessageListener({
    // ==================== MODERATION RESULTS ====================
    // Called when a pending message gets a final moderation decision
    // This fires for YOUR messages that were pending review
    onMessageModerated: (message) => {
      const messageId = message.getId();
      const status = message.getModerationStatus();

      switch (status) {
        case CometChat.ModerationStatus.APPROVED:
          // Message passed moderation - it will be delivered
          console.log(`Message ${messageId} approved`);
          // Update UI - remove pending indicator, show as sent
          updateMessageStatus(messageId, "approved");
          break;

        case CometChat.ModerationStatus.DISAPPROVED:
          // Message was blocked - it won't be delivered
          console.log(`Message ${messageId} blocked`);
          // Update UI - show blocked notice or remove message
          handleBlockedMessage(messageId, message);
          break;
      }
    },
    
    // ==================== INCOMING MESSAGES ====================
    // Called when you receive a message from another user
    // These have already passed moderation (or are pending)
    onTextMessageReceived: (textMessage) => {
      const status = textMessage.getModerationStatus();
      
      if (status === CometChat.ModerationStatus.APPROVED) {
        // Message passed moderation - safe to display
        displayMessage(textMessage);
      } else if (status === CometChat.ModerationStatus.PENDING) {
        // Message is still being reviewed
        // Optionally show with a pending indicator
        displayMessage(textMessage, { pending: true });
      }
      // Don't display DISAPPROVED messages from others
      // They shouldn't reach you, but handle just in case
    },
    
    // Also handle media messages the same way
    onMediaMessageReceived: (mediaMessage) => {
      const status = mediaMessage.getModerationStatus();
      if (status === CometChat.ModerationStatus.APPROVED) {
        displayMessage(mediaMessage);
      }
    }
  })
);

// IMPORTANT: Remove listener when done to prevent memory leaks
// CometChat.removeMessageListener(listenerID);

TypeScript

const listenerID: string = "MODERATION_LISTENER";

CometChat.addMessageListener(
  listenerID,
  new CometChat.MessageListener({
    // Handle moderation decisions for your sent messages
    onMessageModerated: (message: CometChat.BaseMessage): void => {
      const messageId = message.getId();
      const status = message.getModerationStatus();

      switch (status) {
        case CometChat.ModerationStatus.APPROVED:
          // Message approved - update UI to show as sent
          updateMessageStatus(messageId, "approved");
          break;
        case CometChat.ModerationStatus.DISAPPROVED:
          // Message blocked - show notice or remove
          handleBlockedMessage(messageId, message);
          break;
      }
    },
    
    // Handle incoming messages (already moderated)
    onTextMessageReceived: (textMessage: CometChat.TextMessage): void => {
      const status = textMessage.getModerationStatus();
      
      if (status === CometChat.ModerationStatus.APPROVED) {
        displayMessage(textMessage);
      }
      // Skip DISAPPROVED messages
    }
  })
);

What This Code Does

1. Listens for Moderation Results: onMessageModerated fires when pending messages get a decision
2. Handles Incoming Messages: onTextMessageReceived includes moderation status
3. Filters Blocked Content: Only displays approved messages
4. Updates UI: Removes pending indicators or shows blocked notices

Event Flow

Event	When It Fires	Your Action
onMessageModerated	Your pending message gets approved/blocked	Update message status in UI
onTextMessageReceived	You receive a message	Check status before displaying
onMediaMessageReceived	You receive media	Check status before displaying

Incoming Messages: Messages you receive from others have usually already passed moderation. The DISAPPROVED status on incoming messages is rare but possible if moderation rules change.

---

Step 5: Handle Blocked Messages

When a message is disapproved by moderation, decide how to handle it in your UI. The approach differs based on whether it’s your message or someone else’s.

/**
 * Handle a message that was blocked by moderation
 * Different handling for sender vs receiver
 * 
 * @param {string} messageId - The blocked message's ID
 * @param {CometChat.BaseMessage} message - The blocked message object
 */
function handleBlockedMessage(messageId, message) {
  const sender = message.getSender();
  const currentUser = CometChat.getLoggedinUser();

  if (sender.getUid() === currentUser.getUid()) {
    // ==================== YOUR MESSAGE WAS BLOCKED ====================
    // This means your message violated moderation rules
    // You should inform the user why their message wasn't sent
    
    // Option 1: Show a notification explaining the block
    // Best for: Transparent communication with users
    showNotification("Your message was blocked due to policy violation");
    
    // Option 2: Replace the message content with a notice
    // Best for: Keeping message in timeline but showing it was blocked
    replaceMessageContent(messageId, "This message was blocked");
    
    // Option 3: Remove the message from the UI entirely
    // Best for: Clean UI, but user might be confused
    removeMessageFromUI(messageId);
    
  } else {
    // ==================== SOMEONE ELSE'S MESSAGE WAS BLOCKED ====================
    // This is rare - blocked messages usually don't reach you
    // But handle it just in case (e.g., moderation rules changed)
    
    // Simply don't display it or remove if already shown
    removeMessageFromUI(messageId);
  }
}

/**
 * Update a message's visual status in the UI
 * Used to remove pending indicators when moderation completes
 * 
 * @param {string} messageId - The message ID
 * @param {string} status - New status ("approved", "blocked", etc.)
 */
function updateMessageStatus(messageId, status) {
  // Find the message element in your UI
  const messageElement = document.querySelector(`[data-message-id="${messageId}"]`);
  
  if (messageElement) {
    // Remove pending indicator
    messageElement.classList.remove("pending");
    
    // Add new status class for styling
    messageElement.classList.add(status);
    
    // If approved, you might want to remove any "reviewing..." text
    const pendingBadge = messageElement.querySelector(".pending-badge");
    if (pendingBadge) {
      pendingBadge.remove();
    }
  }
}

What This Code Does

1. Identifies Sender: Checks if the blocked message is yours or someone else’s
2. Handles Your Blocked Messages: Shows notification, replaces content, or removes
3. Handles Others’ Blocked Messages: Simply removes from UI
4. Updates Visual Status: Removes pending indicators when moderation completes

Handling Strategies

Strategy	Pros	Cons	Best For
Notification	Clear communication	Interrupts user	Transparent apps
Replace Content	Message stays in timeline	Takes up space	Audit trails
Remove Message	Clean UI	User confusion	Minimal UI

User Experience: Be transparent with users about why their message was blocked. Vague messages like “Something went wrong” lead to frustration. Clear messages like “Your message was blocked due to community guidelines” help users understand and adjust their behavior.

---

Step 6: User Flagging (Manual Reports)

In addition to AI moderation, allow users to manually flag inappropriate messages. This creates a community-driven moderation layer where users can report content that AI might miss.

JavaScript

/**
 * Flag a message for manual review
 * This reports the message to moderators for human review
 * 
 * @param {CometChat.BaseMessage} message - The message to flag
 * @returns {boolean} - true if flagged successfully
 */
async function flagMessage(message) {
  try {
    // flagMessage() reports this message to CometChat's moderation system
    // The message will appear in Dashboard → Moderation → Flagged Messages
    // Moderators can then review and take action
    await CometChat.flagMessage(message);
    console.log("Message flagged successfully");
    
    // Update UI to show the message was reported
    // This provides feedback to the user who reported it
    showNotification("Message reported. Thank you for helping keep our community safe.");
    
    return true;
  } catch (error) {
    console.log("Flag failed:", error);
    
    // Handle specific error cases
    if (error.code === "ERR_ALREADY_FLAGGED") {
      // User already reported this message
      showNotification("You've already reported this message");
    } else if (error.code === "ERR_CANNOT_FLAG_OWN_MESSAGE") {
      // Can't flag your own messages
      showNotification("You cannot report your own messages");
    } else {
      showNotification("Failed to report message. Please try again.");
    }
    
    return false;
  }
}

// ==================== USAGE EXAMPLE ====================
// Typically triggered from a context menu or report button

// Add click handler to report button
const reportButton = document.getElementById("report-btn");
reportButton.addEventListener("click", () => {
  // selectedMessage is the message the user right-clicked or long-pressed
  flagMessage(selectedMessage);
});

React

import { useState } from "react";
import { CometChat } from "@cometchat/chat-sdk-javascript";

/**
 * MessageActions Component
 * Provides action buttons for a message, including report
 * 
 * @param {Object} props
 * @param {CometChat.BaseMessage} props.message - The message to act on
 */
function MessageActions({ message }) {
  const [flagged, setFlagged] = useState(false);
  const [loading, setLoading] = useState(false);

  /**
   * Handle the flag/report action
   * Flags the message and updates UI state
   */
  const handleFlag = async () => {
    setLoading(true);
    
    try {
      // Flag the message for moderator review
      await CometChat.flagMessage(message);
      setFlagged(true);
      
      // Show success feedback
      alert("Message reported successfully. Thank you for helping keep our community safe.");
    } catch (error) {
      if (error.code === "ERR_ALREADY_FLAGGED") {
        // Already reported - update UI to reflect this
        setFlagged(true);
        alert("You've already reported this message");
      } else {
        alert("Failed to report message. Please try again.");
      }
    } finally {
      setLoading(false);
    }
  };

  return (
    <div className="message-actions">
      <button 
        onClick={handleFlag} 
        disabled={flagged || loading}
        className={flagged ? "reported" : ""}
      >
        {loading ? "Reporting..." : flagged ? "Reported" : "Report Message"}
      </button>
    </div>
  );
}

export default MessageActions;

What This Code Does

1. Reports Message: flagMessage() sends the message to CometChat’s moderation queue
2. Handles Errors: Catches duplicate reports and other error cases
3. Updates UI: Shows feedback to the user who reported
4. Prevents Duplicates: Tracks flagged state to prevent re-reporting

Flagging Best Practices

Practice	Description
Confirm Action	Consider showing a confirmation dialog before flagging
Provide Feedback	Always tell users their report was received
Prevent Abuse	Track flagged messages to prevent spam reporting
Easy Access	Make the report option easy to find but not accidental

Moderator Review: Flagged messages appear in the CometChat Dashboard under Moderation → Flagged Messages. Moderators can review each report and take action (approve, delete, or ban user).

---

Step 7: Review Flagged Messages (Admin)

Flagged messages can be reviewed in the CometChat Dashboard:

1. Go to Moderation → Flagged Messages
2. Review each flagged message
3. Take action: Approve, Delete, or Ban User

For programmatic access to flagged messages, use the CometChat REST API.

---

Complete Integration Example

import { CometChat } from "@cometchat/chat-sdk-javascript";

class ModeratedChatService {
  constructor() {
    this.pendingMessages = new Map();
    this.listenerID = "MODERATION_LISTENER";
  }

  setupModerationListeners(callbacks) {
    CometChat.addMessageListener(
      this.listenerID,
      new CometChat.MessageListener({
        onTextMessageReceived: (message) => {
          this.handleIncomingMessage(message, callbacks);
        },
        
        onMediaMessageReceived: (message) => {
          this.handleIncomingMessage(message, callbacks);
        },
        
        onMessageModerated: (message) => {
          this.handleModerationResult(message, callbacks);
        }
      })
    );
  }

  handleIncomingMessage(message, callbacks) {
    const status = message.getModerationStatus();

    switch (status) {
      case CometChat.ModerationStatus.APPROVED:
        callbacks.onMessageReceived?.(message);
        break;
        
      case CometChat.ModerationStatus.PENDING:
        // Optionally show with indicator
        callbacks.onMessageReceived?.(message, { pending: true });
        break;
        
      case CometChat.ModerationStatus.DISAPPROVED:
        // Don't display blocked messages from others
        console.log("Blocked message not displayed");
        break;
    }
  }

  handleModerationResult(message, callbacks) {
    const messageId = message.getId();
    const status = message.getModerationStatus();

    // Remove from pending tracking
    this.pendingMessages.delete(messageId);

    if (status === CometChat.ModerationStatus.APPROVED) {
      callbacks.onMessageApproved?.(message);
    } else if (status === CometChat.ModerationStatus.DISAPPROVED) {
      callbacks.onMessageBlocked?.(message);
    }
  }

  async sendMessage(receiverUID, text) {
    const textMessage = new CometChat.TextMessage(
      receiverUID,
      text,
      CometChat.RECEIVER_TYPE.USER
    );

    const message = await CometChat.sendMessage(textMessage);
    const status = message.getModerationStatus();

    if (status === CometChat.ModerationStatus.PENDING) {
      this.pendingMessages.set(message.getId(), message);
    }

    return {
      message,
      status,
      isPending: status === CometChat.ModerationStatus.PENDING
    };
  }

  async flagMessage(message) {
    try {
      await CometChat.flagMessage(message);
      return { success: true };
    } catch (error) {
      return { 
        success: false, 
        alreadyFlagged: error.code === "ERR_ALREADY_FLAGGED",
        error 
      };
    }
  }

  cleanup() {
    CometChat.removeMessageListener(this.listenerID);
  }
}

// Usage
const moderatedChat = new ModeratedChatService();

moderatedChat.setupModerationListeners({
  onMessageReceived: (message, options) => {
    console.log("New message:", message.getText());
    if (options?.pending) {
      console.log("(pending moderation)");
    }
    // Add to UI
  },
  
  onMessageApproved: (message) => {
    console.log("Message approved:", message.getId());
    // Update UI - remove pending indicator
  },
  
  onMessageBlocked: (message) => {
    console.log("Message blocked:", message.getId());
    // Update UI - show blocked notice or remove
  }
});

// Send a message
const result = await moderatedChat.sendMessage("user_456", "Hello!");
if (result.isPending) {
  console.log("Message sent, awaiting moderation...");
}

// Flag a message
const flagResult = await moderatedChat.flagMessage(someMessage);
if (flagResult.success) {
  console.log("Message reported");
}

---

Moderation Best Practices

Show Pending Status

Let users know their message is being reviewed:

// Show a subtle indicator
<div className={`message ${isPending ? 'pending' : ''}`}>
  {messageText}
  {isPending && <span className="pending-badge">Reviewing...</span>}
</div>

Handle Blocked Messages Gracefully

Don’t just silently remove messages - inform the sender:

if (message.getSender().getUid() === currentUser.getUid()) {
  showNotification("Your message couldn't be sent due to community guidelines");
}

Provide Clear Flagging UI

Make it easy for users to report inappropriate content:

// Add to message context menu
<MenuItem onClick={() => flagMessage(message)}>
  Report Message
</MenuItem>

Don't Over-Moderate

Configure rules carefully in the dashboard to avoid false positives. Start with basic rules and adjust based on your community’s needs.

---

Supported Message Types

Message Type	AI Moderation	User Flagging
Text Messages	✅	✅
Image Messages	✅	✅
Video Messages	✅	✅
Audio Messages	❌	✅
Custom Messages	❌	✅

---

Next Steps

Block Users

Let users block other users

Groups

Moderate group conversations

Webhooks

Get notified of moderation events

REST API

Programmatic moderation management