Guardrails
Checks around a model that screen inputs and outputs for harmful, private or off-topic content.
Also: content filter, moderation, safety classifier
Guardrails are separate from the model's own training. A moderation model may screen prompts before they reach the main model and scan its answers before they reach the user, blocking categories such as violent threats, self-harm or personal data. Meta's Llama Guard, released in 2023, and Google's ShieldGemma are open examples; Anthropic has described constitutional classifiers that screen for weapons-related content. Businesses also add rules to keep assistants on topic. Guardrails add delay and can block legitimate requests.