A comprehensive system for moderating chat messages using multiple AI models and detection methods. This system provides real-time content moderation by analyzing text for various types of inappropriate content, personal information, and potential risks.

The Chat Moderation System uses a multi-model approach to provide thorough content analysis:

1. PII Detection: Identifies personal information like emails, phone numbers, addresses
2. Harassment Detection: Detects harassment, bullying, and toxic content
3. Hybrid Model: Combines multiple approaches for comprehensive analysis
4. Text Moderation: Uses DeBERTa model for general content moderation

• Real-time Analysis: Process messages instantly with WebSocket support
• Multi-Model Approach: Combines multiple AI models for comprehensive analysis
• Configurable Scoring: Flexible scoring system with customizable thresholds
• Model Caching: Efficient model loading and caching system
• Worker Pool: Distributed processing with Redis-based worker system
• Modern Dashboard: React-based UI for real-time monitoring
• Docker Support: Easy deployment with Docker containers

chat_moderation/
├── backend/                 # Python FastAPI backend
│   ├── app/
│   │   ├── api/            # API endpoints
│   │   │   └── endpoints.py # Main API routes
│   │   ├── core/           # Core configuration
│   │   │   ├── config.py   # Settings management
│   │   │   └── logging.py  # Logging setup
│   │   ├── models/         # AI models
│   │   │   ├── pii.py      # PII detection
│   │   │   ├── harassment.py # Harassment detection
│   │   │   └── hybrid.py   # Hybrid model
│   │   ├── services/       # Business logic
│   │   │   ├── evaluator.py # Message evaluation
│   │   │   └── preprocessing.py # Text preprocessing
│   │   └── utils/          # Utilities
│   ├── tests/              # Backend tests
│   └── requirements.txt    # Python dependencies
├── frontend/               # React TypeScript frontend
│   ├── src/
│   │   ├── components/     # React components
│   │   ├── pages/         # Page components
│   │   ├── services/      # API services
│   │   ├── styles/        # CSS styles
│   │   └── utils/         # Utilities
│   ├── public/            # Static assets
│   └── package.json       # Node.js dependencies
└── docker/                # Docker configuration
    ├── backend.Dockerfile
    ├── frontend.Dockerfile
    └── nginx.conf

• FastAPI Framework: High-performance async web framework
• Redis Message Queue: For worker communication
• Model Caching: Efficient model loading and memory management
• WebSocket Support: Real-time updates and notifications
• Pydantic Models: Strong type checking and validation

1. PII Detection Model

   • Based on DeBERTa architecture
   • Detects personal information in text
   • Configurable sensitivity levels

2. Harassment Detection

   • Fine-tuned BERT model
   • Specialized in detecting toxic content
   • Multiple category classification

3. Hybrid Model

   • Ensemble approach combining multiple models
   • Improved accuracy through model combination
   • Configurable weighting system

• Real-time Updates: WebSocket connection for instant results
• Material-UI Components: Modern and responsive design
• TypeScript: Type-safe development
• React Router: Multiple page support
• Axios: API communication

• Python 3.9+
• Node.js 18+
• Redis
• Docker and Docker Compose (optional)

1. Clone the repository:

   git clone <repository-url>
   cd chat-moderation

2. Set up environment files:

   cp backend/.env.example backend/.env
   cp frontend/.env.example frontend/.env

3. Configure environment variables:

   # backend/.env
   OPENAI_API_KEY=your-key-here
   MODEL_WORKERS=4
   MODEL_CACHE_SIZE=2
   
   # frontend/.env
   VITE_API_URL=http://localhost:8000
   VITE_WS_URL=ws://localhost:8000/ws

4. Start services:

   docker-compose up -d

1. Create virtual environment:

   cd backend
   python -m venv venv
   source venv/bin/activate

2. Install dependencies:

   pip install -r requirements.txt

3. Start Redis:

   redis-server

4. Start worker and API:

   # Terminal 1
   python -m app.worker
   
   # Terminal 2
   uvicorn app.main:app --reload

1. Install dependencies:

   cd frontend
   npm install

2. Start development server:

   npm run dev

• POST /api/evaluate: Evaluate a message

  {
    "message": "text to analyze"
  }

• WS /api/ws: WebSocket endpoint for real-time updates

{
  "overall_category_score": 5,
  "final_status": "rejected",
  "label": "harassment",
  "detailed_results": {
    "pii": { ... },
    "harassment": { ... },
    "hybrid": { ... }
  }
}

cd backend
pytest

cd frontend
npm test

• Logs are available in backend/logs/
• Model cache in backend/model_cache/
• Redis persistence in redis_data/

• API keys should be properly secured
• CORS settings should be configured for production
• WebSocket connections should be authenticated
• Model weights should be verified for tampering

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request

This project is licensed under the MIT License - see the LICENSE file for details.

• HuggingFace Transformers library
• OpenAI API
• FastAPI framework
• React and Material-UI