Evaluations measure how well a policy version performs against a labeled dataset: run the policy over the dataset, and the app compares its decisions to your labels, row by row. This is how you check a policy before it goes live and how you confirm a change made things better, not worse.
On any policy, click Test Policy and pick a dataset to run the current version against. Run Results opens right there on the policy page with a quick read of how the run went — covered in the next section. The run is also saved to the Evaluations page, where the deeper analysis lives.
Run Results is the quick read that opens as soon as a test finishes, with two tabs.
The Results tab shows an overview (agreement rate, total items, correct and incorrect counts), a metrics chart you can switch between Agreement, Precision, Recall, F1, False Positives, and False Negatives to compare against your recent runs, per-category metrics, and every individual row with its reason, severity and risk scores, and media outputs — filterable, and exportable as a CSV. A row counts as correct when the policy and the expected label agree on whether the content is flagged, regardless of which flagged category each chose.
The Thresholds tab shows what would happen at different risk-score cutoffs: precision and recall curves across the 0–10 scale, the score distribution split into should-flag and should-clear, and the rows sitting just above and just below the line — the ones a small threshold move would change. Drag the slider (or the chart) to move the threshold and watch precision, recall, F1, and the false positive and false negative counts update. This is how you pick the cutoff that matches your tolerance — see Outputs and custom outputs for how to think about the risk score itself.
From Run Results you can also Rerun Test to check consistency, or click Diagnose to get feedback from our AI about how to improve the policy.
You can start the same kind of run from the other end: click New Evaluation on the Evaluations page and pick a policy version and a dataset.
The Evaluations list shows every run: which policy version ran, against which dataset, its agreement rate and F1 score, and when it ran. Filter by policy or sort any column to find the run you need. Reruns and refine runs appear in the list too, so you can trace how a policy's numbers moved as you iterated.
Assessment Agreement answers the binary question: did the policy flag what your labels say should be flagged, and clear what should be clear?
The top of the page summarizes the run:
The confusion matrix breaks the same run into four boxes: correctly cleared, false alarms (clear content the policy flagged), missed (violations the policy cleared), and caught. The two red boxes are where your attention goes.