Writing effective policies

The app is built on a simple idea: your policy is the prompt. The model doesn't bring its own opinions about what should be banned, escalated, or allowed. Instead, it reads your policy, looks at the content, and applies your rules. If the policy is sharp, the output is sharp. If the policy is vague, the model fills in the gaps with its own priors, and you get inconsistency.

A few principles that shape the product:

The LLM is a literal reader. It does what you ask it to do. It doesn't know your platform's history, what exists outside of the content and prompt you send it, or your unwritten norms. If something matters, it has to be in the policy.
Policies are software. They get versions, tests, and iteration loops. They are not one-and-done documents.
Specificity beats hedging. Vague language ("egregious," "obvious") feels safe but produces inconsistent results. Concrete criteria (even if you have to be slightly wrong sometimes) produce a system you can debug.
Optimize for the boundary. The interesting cases are the ones at the edge of a category. That's where examples earn their keep.

Why policy quality matters

Output quality is downstream of policy quality. Here's what that looks like in practice:

Symptom you see	Usually caused by
Same kind of content classified differently across runs	Vague thresholds; missing examples at the boundary
Clean content getting flagged	"Clear" category named or scoped wrong; rules overlap
The wrong label applied to a clear violation	Categories defined by adjacency to each other rather than by their own criteria
Inconsistent behavior on satire, news, or educational content	Exceptions not spelled out; implicit cultural context
Reviewers disagreeing with the model on appeals	Policy says one thing, internal practice does another

A useful test: if a brand-new reviewer applied your written policy literally, would they reach the same answers as your tenured team? If no, then you'll have an LLM problem too, because the model will use whatever the written policy says, plus its own priors, to fill the gap. Closing that gap by tightening the policy text usually fixes both groups at once.

The elements of a policy each pull on outcomes differently:

Categories are the labels the model returns. They set the resolution of your enforcement decisions.
Descriptions tell the model when each label applies. This is where most of your precision lives.
Examples calibrate the boundary. They're how the model learns what violations look like for you.
Exceptions ("Not a violation" lists) prevent over-enforcement on content that looks like a violation but isn't.
Instructions (optional) tell the model how to approach the whole policy: tone, fallback behavior, what to do with edge cases that don't match cleanly.

Anatomy of a good policy

Categories that don't overlap

Each category should be defined by what it covers, not by contrast with another category. If two categories require you to think about each other to apply correctly, the model will trip on the boundary. When you find yourself writing "this category covers X except when it's actually category Y," that's a sign the underlying taxonomy needs work.

A rough test: cover up the category name and read the description alone. Can you tell which behavior it's describing? If not, the description is leaning on the label.