The Illegal Comment Problem

The problem

German criminal law takes online speech more seriously than most jurisdictions. §130 (Volksverhetzung — incitement to hatred), §185 (Beleidigung — insult), §186 (üble Nachrede — defamation), and §187 (Verleumdung — calumny) are four of twelve statutes the system I built and operate assesses against, and they apply to a comment on a politician's TikTok video the same way they apply to a pamphlet handed out in a market square. There are real consequences, real prosecutions, and real removal obligations under the EU's Digital Services Act.

The operational problem is not, as the press sometimes frames it, that hate speech is hard to spot. Any fluent German speaker can read a thread and tell you what crossed the line. The problem is scale, context, and the cost of being wrong in either direction.

A mid-sized political account can attract thousands of comments on a single post. Across the accounts my system monitors, that adds up to 5.4M+ comments and 26,000+ posts processed to date. A small legal-review team has neither the time nor the throughput to read each one carefully, let alone build a case for removal. Meanwhile every false positive — flagging legal speech as illegal — erodes the team's ability to act on the real violations. Over-flagging is worse than under-flagging, because it poisons the review queue and trains the humans to stop trusting it.

So the problem reframes: how do you build a system that finds the highest-reach violations, classifies them against a fact-specific legal standard, and produces a review queue a human will actually work through — and act on — on a Tuesday afternoon?

The constraint

Four constraints shape everything:

1. The statutes are fact-specific. §185 depends on intent and context. §187 requires a false assertion of fact. §130 has a public-order threshold. A model that classifies by keyword is not just wrong, it's wrong in a way that gets the user sued for over-removal.
2. The reviewer is not a lawyer. The output has to make sense to a press officer or a community manager, not a paralegal. If the interface requires reading five pages of reasoning to act on a flag, it will not be acted on.
3. The goal is removal, not detection. Detection is a vanity metric. What the platform owner actually cares about is: did the illegal comment come down? Everything upstream of that question is overhead.
4. Inference is not free at this scale. 1.4 billion tokens run through the system per cycle. Routing every one of them through a frontier model would make the product too expensive to operate — and a civic-tech tool nobody can afford to run doesn't protect anyone.

The approach

The system is eight stages, not two. Discovery narrows an unbounded platform down to a watchlist before ingestion ever starts. Ingestion writes an immutable capture record for every comment. Classification is staged — cheap first, expensive only where it has to be. Evaluation checks model output against a structured legal test and a benchmark gate, not a vibe. Action turns a confirmed violation into a filed report with evidence attached. Outcome captures what actually happened, on the platform and on appeal. Every outcome feeds a data flywheel that makes the next model version better. And the whole thing runs on infrastructure I own, end to end.

Discovery, not brute force. Broad social platforms are too large to monitor comment-by-comment from a standing start. Network analysis, graph analysis, and community detection identify which sources actually carry the reach that matters — in the community this system watches, a bounded watchlist built this way carries roughly 81.6% of measured reach. Discovery is the difference between monitoring a platform and monitoring everyone on it.

Staged classification, not one expensive pass. Every comment that survives ingestion runs through a cheap, fast model first. Its only job is to decide probably clean / probably worth a second look — and it resolves 49 to 72% of traffic on its own, depending on the account and the cycle. What survives escalates to a self-hosted, open-weight model served across eight A100/H100/H200 GPUs with vLLM, tensor parallelism, and prefix caching pushing cache utilization to roughly 99%. That architecture, plus routing the majority of volume away from the expensive tier, is what keeps inference around 65% cheaper than running a frontier model on everything — about €0.0035 per unit, blended, across the full 1.4B-token cycle.

Score against statute elements, don't emit a label. The fine-tier output is not "legal" or "illegal." It's a structured record per statute: which elements of the offense are present, which are absent, which are unclear, and how confident the model is. §186 asks about factual assertions, so the model extracts the factual claim and checks whether it's presented as fact or opinion. §185 asks about the target of the insult, so the model identifies who's being addressed and whether the expression is directed at them personally. Structured, grammar-constrained decoding makes the output shape deterministic — the model can be wrong about the law, but it can't be wrong about the schema.

No model version ships without clearing a gate. Every candidate prompt or model version is benchmarked against a held-out set of real decisions from a certified EU dispute-settlement body before it's allowed anywhere near production traffic. Below 90% agreement with those reference decisions, the version doesn't promote — full stop. That threshold turned evaluation from a one-time sanity check into a standing regression suite, the same discipline a compiler team applies to a language change.

The principle underneath all of it: optimize for removal outcomes. A review queue that produces 100 flags, of which 60 get removed, is better than one that produces 500 flags, of which 80 get removed. Every decision in the system — the triage threshold, the severity sort, the promotion gate, the UI's default ordering — is tuned to maximize acted-on flags per unit of reviewer attention, not the number of flags in total.

The build

The architecture, end to end:

Two pieces are worth calling out beyond the diagram. The first is the operator console — the internal application the legal team actually works in, built on Next.js and NestJS with role-based access control, signed evidence streaming, live pipeline telemetry, and an audit log that keeps what the machine decided structurally separate from what a human decided. That separation isn't a UI nicety; it's the record that has to hold up if a removal is ever challenged.

The second is that I own the infrastructure this system runs on — not as a dedicated specialism, but because a production AI system needs someone accountable for it. Nineteen services across Hetzner and GCP, Docker and Kubernetes, Lambda for the edges, Redis/BullMQ for the queues, with the monitoring, deployment, and rollback discipline that keeps an inference pipeline this large from becoming someone's 2am problem.

The outcome

The system produces a legally coherent, evidence-backed removal request without a human having to build the case from scratch. Confirmed violations get filed through automated Art. 16 DSA reporting — including on-device evidence capture via physical-device orchestration, so the report is defensible on its own. Of the cases filed, 64.4% result in platform removal, and of the subset independently reviewed under Art. 21, 73.2% are upheld. Those numbers describe this system's cases, not a claim about platforms or the DSA regime generally.

The quieter outcome is the data flywheel. Every classification carries its model output, the platform's response, the QA decision, and — where it happens — legal review, and all of it becomes a growing German-language labelled corpus. That corpus is what the next evaluation benchmark and the next model version are built on. The system doesn't just get faster over time; it gets more accurate, because production is where the hardest cases actually show up.

Aftermath

Three things I'd do differently, or sooner.

Ground the statutes in a living glossary. The first version of the fine classifier's prompts hardcoded the statute definitions. The second version pulled them from a version-controlled glossary that legal advisors could edit without touching code. That should have been there from day one.

Make the assessment record queryable. The first schema treated each assessment as a blob. The second version flattened the per-element decisions into columns. Being able to ask "show me all flags where §185's target element was ambiguous" turned out to be the most useful operational query I had. Design for that from the start.

Treat the promotion gate as infrastructure, not a one-off script. The benchmark started as a notebook someone ran before a deploy. It's now a standing suite that blocks promotion automatically. Build the gate before you need to trust it under pressure, not after the first bad version almost ships.

The larger lesson, for anyone building high-stakes classification: the model is the easy part. Staged inference, a benchmark gate, an evidence trail, and infrastructure someone actually owns — that's the system. If you build the model first and the system second, you will spend the next year rebuilding the model.