It’s up to you to decide how to utilize the output within your broader moderation system.

Understanding the Output: For each piece of content submitted, PolicyAI provides a binary Clear or Flag decision and a classification label (the determined policy category). Additional outputs are available, including a reason, an analysis, a severity score indicating the model's classification of how severe the violation is, and a **risk score ****indicating how certain the content contains a policy violation. You can also create custom outputs tailored to your needs!

Severity and risk scores can be additional helpful signals for triage, human review/ oversight, and double-checking.

Potential Workflow Integrations:

• Automated Actions: Integrate PolicyAI's API to trigger actions based on classifications and risk scores.
• Human Review Queueing: Use PolicyAI outputs to populate specific queues for human moderators, prioritizing content based on predicted category and severity.
• Data Analysis and Reporting: Store PolicyAI outputs to analyze trends in violations, understand the volume of different content types, and report on moderation effectiveness.
• Feedback Loops: Integrate human review decisions back into your system to update golden sets and inform policy refinements.