What you are building
This tutorial builds a content moderation system for user-generated content — the pipeline that decides whether a post, comment, image caption, or message is safe to publish. The reliable architecture is not a single model verdict but a series of layers, each cheaper and faster than the last, that together auto-approve the obviously fine, auto-block the obviously harmful, and route the uncertain middle to human reviewers. Built well, the system handles the overwhelming volume automatically while concentrating scarce human attention on the genuinely ambiguous cases where mistakes matter most.
The layered pipeline
Start with a fast pre-filter: cheap heuristics and a blocklist catch the trivial cases (known spam patterns, banned terms) before any model spend. Next, a classification layer runs the content through the OpenAI Moderation API and, where you have policy-specific needs, a custom classifier you have fine-tuned on your own labelled data. These return per-category scores — hate, harassment, self-harm, sexual, violence, plus your own categories like fraud or off-platform solicitation. A decision layer applies two thresholds per category: above the high cutoff, auto-block; below the low cutoff, auto-allow; the band in between goes to a human-review queue. Reviewers resolve the hard cases, and their labels flow back to improve the classifiers over time.
Thresholds, errors, and the cost of being wrong
The heart of the system is the threshold policy, because moderation is a balance of two opposing errors. A false positive blocks legitimate content and frustrates real users; a false negative lets genuine harm reach the platform. You cannot minimise both at once, so you tune cutoffs against a labelled sample according to your risk tolerance — a children’s platform leans toward over-blocking, a developer forum toward under-blocking. Categories carry different stakes too: you might auto-block high-confidence self-harm or CSAM signals immediately while sending borderline harassment to review. Treat these cutoffs as explicit, documented policy decisions, not defaults you inherited from an API.
Humans, audit logs, and appeals
AI shrinks the queue; it does not eliminate the need for people. Humans resolve the ambiguous band, handle context the model misses — sarcasm, reclaimed slurs, cultural nuance, coded language — and own the appeals process. Log every decision with its inputs, model scores, threshold, action, and reviewer, so you can audit accuracy, defend individual calls, retrain on real data, and meet regulatory expectations under regimes like the EU Digital Services Act. Give users a clear appeal route that reaches a human, and feed every overturned decision back as a training label. A moderation system is ultimately judged not only on how much harm it catches but on how fairly it treats the people it gets wrong — build the recourse path in from day one.