Red-teaming
Deliberately attacking a model to find ways it can be made to misbehave, before others find them.
Also: red team, adversarial testing
Red-teaming borrows its name from military and security exercises. Testers, from in-house staff to outside experts and automated attacker models, try to get a model to produce harmful content, leak data, write malware, assist with weapons or act unsafely as an agent. Findings shape safety training and are summarised in system cards. Labs give government safety institutes pre-release access for such testing, and public events, such as the generative AI red-teaming exercise at the DEF CON hacking conference in 2023, have widened participation.