Jailbreak
A prompt crafted to get around a model's safety training and make it produce what it would normally refuse.
Jailbreaks exploit the gap between what a model was trained to refuse and the endless ways a request can be phrased. Early ones asked the model to role-play an unrestricted character; others use hypothetical framing, unusual languages or encodings, very long chains of examples, or automatically searched strings of odd tokens, as a 2023 Carnegie Mellon paper showed. Labs patch known jailbreaks, add classifiers that screen inputs and outputs, and run bounty programmes, but no model has proved fully resistant.