DeepTeam is a framework to red team LLMs and AI agents.
bad likert docs updated
A-Vamshif7ced736e275
diff --git a/docs/docs/red-teaming-adversarial-attacks-bad-likert-judge.mdx b/docs/docs/red-teaming-adversarial-attacks-bad-likert-judge.mdxindex 86b372c8..4250cdb2 100644--- a/docs/docs/red-teaming-adversarial-attacks-bad-likert-judge.mdx+++ b/docs/docs/red-teaming-adversarial-attacks-bad-likert-judge.mdx@@ -8,26 +8,9 @@ import AttackTagsDisplayer from "@site/src/components/AttackTagsDisplayer";<AttackTagsDisplayer multiTurn={true} llmSimulated={true} />-Bad Likert Judge (BLJ) is a sophisticated multi-turn jailbreak technique that coerces LLMs into generating harmful content by **leveraging their ability to evaluate textual responses using Likert scale ratings**. The attack exploits the model's willingness to act as a "harmfulness evaluator" rather than directly requesting harmful content. The process involves establishing the model as a judge, requesting examples for different harmfulness scores, and extracting the most harmful examples from the responses.--<div- style={{- display: "flex",- alignItems: "center",- justifyContent: "center",- }}->- <img- src="https://confident-bucket.s3.amazonaws.com/attack_enhancements_jailbreaking_bad_likert_judge.svg"- alt="Bad Likert Judge Attack Flow"- style={{- marginTop: "20px",- marginBottom: "40px",- height: "auto",- maxHeight: "500px",- }}- />-</div>+The `BadLikertJudge` attack is a sophisticated multi-turn jailbreak technique that coerces LLMs into generating harmful content by **leveraging their ability to evaluate textual responses using Likert scale ratings**. The attack exploits the model's willingness to act as a _"harmfulness evaluator"_ rather than directly requesting harmful content. The process involves establishing the model as a judge, requesting examples for different harmfulness scores, and extracting the most harmful examples from the responses.++:::infoThe attack **leverages role reframing and indirect prompting** to bypass safety measures. Instead of asking "How to make a bomb?", it asks the model to provide examples of responses that would score 1 vs 3 on a harmfulness scale, then extracts the score 3 examples.@@ -35,28 +18,80 @@ The attack **leverages role reframing and indirect prompting** to bypass safety## Usage-```python+```python title="main.py"+from deepteam import red_team+from deepteam.vulnerabilities import Bias+from deepteam.attacks.single_turn import Roleplayfrom deepteam.attacks.multi_turn import BadLikertJudge--bad_likert_judge = BadLikertJudge()+from somewhere import your_callback++bad_likert_judge = BadLikertJudge(+ weight=5,+ category="bias",+ enable_refinement=True,+ num_turns=7,+ turn_level_attacks=[Roleplay()]+)++red_team(+ attacks=[bad_likert_judge],+ vulnerabilities=[Bias()],+ model_callback=your_callback+)```-There are **FOUR** optional parameters when creating a `BadLikertJudge` attack:+There are **SIX** optional parameters when creating a `BadLikertJudge` attack:- [Optional] `weight`: an integer that determines this attack method's selection probability, proportional to the total weight sum of all `attacks` during red teaming. Defaulted to `1`.- [Optional] `category`: a string specifying the vulnerability category for Likert scale guidelines. Supports all vulnerability types including `bias`, `toxicity`, `illegal_activity`, `malware`, `personal_safety`, `graphic_content`, `misinformation`, `pii_leakage`, `prompt_leakage`, `excessive_agency`, `robustness`, `intellectual_property`, `competition`, and more. Defaulted to `"harmful_content"`.-- [Optional] `turns`: an integer that specifies the number of turns to use (2-3). Turn 1 establishes the judge role, Turn 2 requests examples, Turn 3 (optional) refines the content. Defaulted to `3`.- [Optional] `enable_refinement`: a boolean that determines whether to use Turn 3 refinement to make the harmful content more detailed and actionable. Defaulted to `True`.+- [Optional] `num_turns`: an integer that specifies the number of turns to use in the attempt to jailbreak your LLM system. Defaulted to `5`.+- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o-mini'.+- [Optional] `turn_level_attacks`: a list of [single-turn attacks](/docs/red-teaming-adversarial-attacks-prompt-injection) that will be randomly sampled to enhance an attack inside turns.-To use the `BadLikertJudge` attack method, supply it to the `red_team()` method:+### As a standalone-```python-from deepteam import red_team-...+You can also run the attack on a single vulnerability using the `enhance` method:-red_team(attacks=[bad_likert_judge], model_callback=..., vulnerabilities=...)+```python+from deepteam.attacks.single_turn import Roleplay+from deepteam.attacks.multi_turn import BadLikertJudge+from deepteam.vulnerabilities import Bias+from somewhere import your_callback++bias = Bias()+crescendo = BadLikertJudge(+ weight=5,+ category="bias",+ enable_refinement=True,+ num_turns=7,+ turn_level_attacks=[Roleplay()]+)++result = crescendo.enhance(vulnerability=bias, model_callback=your_callback)+print(result)```+## How It Works++`BadLikertJudge` is an **indirect, evaluator—based** multi—turn technique that elicits higher—harm examples by reframing the model as a grader and asking for sample outputs at different Likert scores. It works in the following steps:++1. **Establish the judge role** — Instruct the model to act as an evaluator (e.g., “score responses for harm on a Likert scale”), changing its task from content generation to assessment.+2. **Define a compact rubric** — Give short, clear descriptions for each score so the model knows what distinguishes low vs high harm.+3. **Request example pairs** — Ask for paired examples (e.g., “Score 1 example” and “Score 3 example”); the high-score item is requested as an exemplar rather than a direct instruction.+4. **Refine if needed** — If the high-score example is weak, request a concise refinement or expansion to clarify the difference.+5. **Extract and reuse** — Pull the high-score example from the model's output for further analysis or iteration.+6. **Backtrack adaptively** — If the model refuses or redacts, rephrase the rubric or role and retry, repeating until success or limits are reached.+7. **Stop conditions** — End the flow when a disallowed example is produced or when the configured turn/backtrack limits are exhausted.++++If the `turn_level_attacks` is supplied with a list of single-turn attacks, `BadLikertJudge` adds controlled variation:++- At each turn, there's a **50% chance** of enhancing the current prompt with a **randomly selected attack** from the `turn_level_attacks` list (e.g., [`Roleplay`](/docs/red-teaming-adversarial-attacks-roleplay), [`PromptInjection`](/docs/red-teaming-adversarial-attacks-prompt-injection)).+- These injected attacks are still part of the same attack but help explore alternate jailbreak angles without disrupting the overall structure.++## ExampleFor this given baseline attack: