Moderation System Architecture

Overview

The moderation system provides automated content moderation for user-generated content including chat messages and reviews. It uses a multi-layered approach combining OpenAI’s Moderation API, custom LLM analysis, and Google’s Perspective API to ensure content safety while minimizing false positives.

System Flow

flowchart TD
    A[Message Created] --> B[InstantDB Subscriber]
    B --> C{Already Moderated?}
    C -->|Yes| Z[Skip]
    C -->|No| D[Add to Moderation Queue]
    D --> E[ChatModerationProcessor]
    E --> F[ModerationService.runModeration]
    F --> G[OpenAI Moderation API]
    G --> H{Flagged?}

    H -->|No| I[Enqueue Notification]
    I --> J[Mark as Moderated]

    H -->|Yes| K[LLM Flagging Analysis]
    K --> L[Determine Action]
    L --> M{Room Type?}
    M -->|Category Room| N[Perspective API Scores]
    M -->|Group/Private| O[LLM Sentiment Analysis]
    N --> P[Update Message with Results]
    O --> P
    P --> Q[Skip Notification]

Components

1. InstantDB Subscriber Service

Location: server/src/instantdb/subscriber.service.ts

Subscribes to real-time message updates from InstantDB and queues unmoderated messages for processing.

Responsibility	Description
Subscription Management	Creates and maintains WebSocket subscriptions to InstantDB
Idempotency	Skips already-moderated messages
Job Deduplication	Uses `messageId` as BullMQ `jobId` to prevent duplicates
Health Monitoring	Cron job checks subscription health every minute

2. Moderation Service

Location: server/src/moderation/moderation.service.ts

Orchestrates the moderation pipeline and determines final actions.

async runModeration(type: ModerationType, data: any): Promise<ModerationResultDto>

Moderation Types:

CHAT - Chat messages
REVIEW - Product/service reviews

3. Analysis Service

Location: server/src/moderation/analysis.service.ts

Provides AI/ML analysis capabilities:

Method	Provider	Purpose
`checkOpenAIModeration()`	OpenAI	Binary flag detection (fast, first-pass)
`getFlaggingDetailsFromLLM()`	OpenRouter	Detailed severity analysis for flagged content
`getSentimentFromLLM()`	OpenRouter	Sentiment scoring (-1 to 1)
`getPerspectiveScores()`	Google Perspective API	Toxicity, profanity, and sexual content scores

4. Queue Processors

Location: server/src/moderation/processors/

Processor	Queue	Job Type
`ChatModerationProcessor`	`chat-moderation`	`MODERATE_CHAT`
`ReviewModerationProcessor`	`review-moderation`	`MODERATE_REVIEW`
`CommentModerationProcessor`	`comment-moderation`	`MODERATE_COMMENT`

Decision Flow

Step 1: OpenAI Moderation (Gate)

All content first passes through OpenAI’s Moderation API. This is a fast, binary check.

Not Flagged: Immediately enqueue notification, skip detailed analysis
Flagged: Proceed to detailed LLM analysis

Step 2: LLM Flagging Analysis

For flagged content, the LLM provides:

reason: Human-readable explanation
recommendedAction: RETAIN, WARN, or REMOVE
needsHumanReview: Boolean for edge cases

Step 3: Context-Specific Analysis

Additional analysis based on content type and context:

Context	Analysis
Category Room Chat	Perspective API (toxicity, profanity, sexual)
Group/Private Chat	LLM Sentiment Analysis
Reviews	LLM Sentiment + Perspective API

Data Model

ModerationResultDto

{
  messageId?: string;
  reviewId?: string;
  isFlagged: boolean;
  action: ModerationAction;
  needsHumanReview: boolean;
  moderationReason?: string;
  sentimentScore?: number;
  sentimentLabel?: SentimentLabel;
  toxicityScore?: number;
  profanityScore?: number;
  sexualScore?: number;
  moderatedAt: Date;
}

ModerationAction Enum

RETAIN - Content is safe
WARN - Content is borderline, may need review
REMOVE - Content should be hidden/removed

SentimentLabel Enum

POSITIVE
NEUTRAL
NEGATIVE

Configuration

Environment variables:

Variable	Description
`OPENAI_API_KEY`	OpenAI API key for moderation
`OPENROUTER_API_KEY`	OpenRouter API key for LLM
`OPENROUTER_BASE_URL`	OpenRouter base URL
`PERSPECTIVE_API_KEY`	Google Perspective API key
`MODERATION_LLM_MODEL`	Model ID for LLM analysis
`MODERATION_SENTIMENT_PROMPT`	Prompt template for sentiment
`MODERATION_FLAGGING_PROMPT`	Prompt template for flagging

Moderation Flow Implementation & Cleanup

Implemented a robust, sequential moderation-to-notification flow and performed comprehensive code cleanup.

Changes Made

1. Moderation→Notification Flow Restructure

Sequential Processing: Modified ModerationService to trigger notifications only after OpenAI moderation passes.
Notification Gate: OpenAI moderation acts as the first filter. If content is flagged, detailed LLM analysis runs but notifications are skipped.
Decoupled Subscriber: Removed notification logic from instantdb.service.ts and subscriber.service.ts, making them purely responsible for queuing moderation.
Idempotency: Leveraged BullMQ jobId for robust deduplication.

2. Code Cleanup & Optimization

Debug Logs: Removed all this.logger.debug statements from moderation-related services.
Console Logs: Removed console.log statements from analysis.service.ts.
Comment Stripping: Removed all JSDoc and internal comments from:
- moderation.service.ts
- analysis.service.ts
- instantdb.service.ts
Protocol Fix: Verified and restored URLs corrupted during comment stripping.

Verification Results

Integration Flow

sequenceDiagram
    participant IDB as InstantDB
    participant SUB as Subscriber
    participant MOD as ModerationService
    participant NOTIF as NotificationService

    IDB->>SUB: New Message (moderated: false)
    SUB->>MOD: Add Moderation Job
    MOD->>MOD: OpenAI Moderation Check
    alt Content is Clean
        MOD->>NOTIF: Enqueue Notification Job
        MOD->>IDB: Mark as Moderated (isFlagged: false)
    else Content is Flagged
        MOD->>MOD: LLM Severity Analysis
        MOD->>IDB: Mark as Moderated (isFlagged: true)
        Note over MOD: Skips Notification
    end

Clean Code Verification

Ran grep checks confirming 0 instances of debug logs and comments in the modified service files.

Queue Configuration

Moderation queues are configured with:

removeOnComplete: false (for debugging/audit)
attempts: 3 with exponential backoff
jobId: chat-mod-{messageId} for deduplication

Integration Points

Inbound

InstantDB Subscriptions: Real-time message stream
Direct API Calls: For batch moderation

Outbound

Notification Service: Enqueues notifications for clean content
InstantDB Updates: Marks messages as moderated with results

Notification Flow

Notifications are only sent for content that passes OpenAI moderation:

OpenAI check returns !flagged
enqueueNotificationForChat() fetches message details
Validates sender/room metadata
Queues notification job
Marks message as notificationProcessedAt

Flagged content skips notification entirely.