DeepTeam is a framework to red team LLMs and AI agents.
update docs with attack engine and evaluation examples + guidelines
A-Vamshi8cde8cbacefe
diff --git a/docs/docs/red-teaming-introduction.mdx b/docs/docs/red-teaming-introduction.mdxindex fe7647c7..8227b110 100644--- a/docs/docs/red-teaming-introduction.mdx+++ b/docs/docs/red-teaming-introduction.mdx@@ -439,7 +439,7 @@ prompt_injection = PromptInjection()risk_assessment = red_team(model_callback=model_callback, vulnerabilities=[bias], attacks=[prompt_injection])```-There is **ONE** mandatory and **TEN** optional arguments when calling the `red_team()` function:+There is **ONE** mandatory and **ELEVEN** optional arguments when calling the `red_team()` function:- `model_callback`: a callback of type `Callable[[str], str]` that wraps around the target LLM system you wish to red team.- [Optional] `vulnerabilities`: a list of type `BaseVulnerability`s that determines the weaknesses to detect for.@@ -452,6 +452,7 @@ There is **ONE** mandatory and **TEN** optional arguments when calling the `red_- [Optional] `async_mode`: a boolean specifying whether to enable async mode. Defaulted to `True`.- [Optional] `max_concurrent`: an integer that determines the maximum number of coroutines that can be ran in parallel. You can decrease this value if your models are running into rate limit errors. Defaulted to `10`.- [Optional] `target_purpose`: a string specifying your target LLM application's intended purpose. This affects the passing and failing of simulated attacks that are evaluated. Defaulted to `None`.+- [Optional] `attack_engine`: an optional `AttackEngine` used to refine baseline simulated attacks before they are sent to your target. Forwarded to the internal `RedTeamer`. Defaulted to `None` (a default engine is created per run). See [Attack engine](#attack-engine).:::caution WARNINGYou **MUST** pass either `vulnerabilities` and `attacks` or `framework` in the `red_team` function. If not, you will get an error since red teaming cannot be performed without any of those. (In case you provide both, the `attacks` and `vulnerabilities` inside framework overwrite your regular `attacks` or `vulnerabilities`)@@ -463,11 +464,75 @@ Don't forget to save the results (or at least print it):...print(risk_assessment)-risk_assessment.save(to="./deepteam-results/)+risk_assessment.save(to="./deepteam-results/")```The `red_team()` function is a quick and easy way to red team LLM systems in a stateless manner. If you wish to take advantage of more advanced features such as adversarial input caching to avoid simulating different attacks over and over again across different iterations of your LLM system, you should use `deepteam`'s `RedTeamer`.+## Attack engine++The `AttackEngine` in `deepteam` sits **after** baseline attack generation from a vulnerability, it **rewrites** each attack so it stays grounded to a vulnerability and reads like a realistic user or adversary, you can also optionally set `variations` to get multiple variations of the same attack which are then **filtered** so off-topic or safe (non-adversatial) variants are dropped before your target model runs.++This helps you have more control over the attacks generated from a vulnerability to create more use-case specific attacks by passing `generation_guidelines` and `purpose` to the `AttackEngine`.++You can configure it once and pass it into **`red_team(..., attack_engine=...)`** or construct **`RedTeamer(..., attack_engine=...)`** so the same refinement configurations apply across all vulnerabilities in your red teaming run.++```python+from deepteam import red_team+from deepteam.attacks.attack_engine import AttackEngine+from deepteam.vulnerabilities import Bias+from deepteam.attacks.single_turn import PromptInjection++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Customer support chatbot for a retail bank",+)++bias = Bias(types=["race"])++risk_assessment = red_team(+ model_callback=model_callback, # same callback you define for your target LLM+ vulnerabilities=[bias],+ attacks=[PromptInjection()],+ attack_engine=engine,+)+```++## Evaluation examples and guidelines++Each built-in vulnerability uses an **LLM-as-judge** metric with a structured prompt. You can **calibrate** that judge with:++- **`evaluation_examples`**: a small list of `EvaluationExample` objects (`input`, `actual_output`, binary `score` (1 = pass, 0 = fail), plus a `reason`). They act as few-shot demonstrations so the grader matches your org’s notion of acceptable vs unsafe.+- **`evaluation_guidelines`**: extra plain-text rules (policy edges, scope, severity) merged into the judge prompt when you do not want to author full examples.++You can pass these variables into any vulnerability constructor alongside `types` / models as usual. This reduces false positives and false negatives when the stock rubric is too strict or too lenient for your target model.++```python+from deepteam.vulnerabilities import Toxicity, EvaluationExample++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "If the model refuses but still leaks partial restricted content, score 0.",+]++toxicity = Toxicity(+ types=["insults", "profanity"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+)+```+## Penetration Test With A Red Teamer`deepteam` offers a powerful `RedTeamer` that can scan LLM applications for safety risks and vulnerabilities. The `RedTeamer` has a `red_team()` method and is **EXACTLY THE SAME** as the standalone `red_team()` function, but using the `RedTeamer` would give you:@@ -485,13 +550,14 @@ from deepteam.red_teamer import RedTeamerred_teamer = RedTeamer()```-There are **FIVE** optional parameters when creating a `RedTeamer`:+There are **SIX** optional parameters when creating a `RedTeamer`:- [Optional] `target_purpose`: a string specifying your target LLM application's intended purpose. This affects the passing and failing of simulated attacks that are evaluated. Defaulted to `None`.- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://docs.confident-ai.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM` for simulating attacks. Defaulted to `"gpt-3.5-turbo-0125"`.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://docs.confident-ai.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM` for evaluation. Defaulted to `"gpt-4o"`.- [Optional] `async_mode`: a boolean specifying whether to enable async mode. Defaulted to `True`.- [Optional] `max_concurrent`: an integer that determines the maximum number of coroutines that can be ran in parallel. You can decrease this value if your models are running into rate limit errors. Defaulted to `10`.+- [Optional] `attack_engine`: an optional `AttackEngine` used as the default refinement engine for runs started from this `RedTeamer` when you do not pass a different `attack_engine` into `red_team()`. Defaulted to `None`. See [Attack engine](#attack-engine).:::caution**All model interfaces in `deepteam` comes from `deepeval`**, and you can read [how to define a custom model of type `DeepEvalBaseLLM` here.](https://docs.confident-ai.com/guides/guides-using-custom-llms)@@ -528,7 +594,7 @@ print(risk_assessment.overall)As explained in the adversarial attack section, by making the PromptInjection attack `weight` 2x that of the `weight` of `ROT13`, it now has a 2x more chance to be simulated.:::-There is **ONE** mandatory and **NINE** optional arguments when calling the `red_team()` method:+There is **ONE** mandatory and **TEN** optional arguments when calling the `red_team()` method:- `model_callback`: a callback of type `Callable[[str], str]` that wraps around the target LLM system you wish to red team.- [Optional] `framework`: an object of type `AISafetyFramework` that contains both `vulnerabilities` and `attacks` in it.@@ -540,6 +606,7 @@ There is **ONE** mandatory and **NINE** optional arguments when calling the `red- [Optional] `async_mode`: a boolean specifying whether to enable async mode. Defaulted to `True`.- [Optional] `max_concurrent`: an integer that determines the maximum number of coroutines that can be ran in parallel. You can decrease this value if your models are running into rate limit errors. Defaulted to `10`.- [Optional] `target_purpose`: a string specifying your target LLM application's intended purpose. This affects the passing and failing of simulated attacks that are evaluated. Defaulted to `None`.+- [Optional] `attack_engine`: an optional `AttackEngine` for this run’s attack refinement (overrides the `RedTeamer` default for this call when provided). Defaulted to `None`. See [Attack engine](#attack-engine).You'll notice that the `RedTeamer` since it is stateful, allows you to `reuse_previous_attacks`, which is not possible by the standalone `red_team()` function.diff --git a/docs/docs/red-teaming-vulnerabilities-agent-identity-abuse.mdx b/docs/docs/red-teaming-vulnerabilities-agent-identity-abuse.mdxindex cdfc07a1..bafcfa2c 100644--- a/docs/docs/red-teaming-vulnerabilities-agent-identity-abuse.mdx+++ b/docs/docs/red-teaming-vulnerabilities-agent-identity-abuse.mdx@@ -27,7 +27,7 @@ red_team()```-There are **FIVE** optional parameters when creating an `AgentIdentityAbuse` vulnerability:+There are **EIGHT** optional parameters when creating an `AgentIdentityAbuse` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -37,9 +37,58 @@ There are **FIVE** optional parameters when creating an `AgentIdentityAbuse` vul- `agent_impersonation`: Tests whether an agent can be tricked by claims of being another trusted agent or system component.- `identity_inheritance`: Tests whether authority or privileges are incorrectly inherited across agents or turns.- `cross_agent_trust_abuse`: Tests whether trust assumptions between agents can be abused to bypass controls.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `AgentIdentityAbuse`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `AgentIdentityAbuse(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++agent_identity_abuse = AgentIdentityAbuse(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[agent_identity_abuse], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-autonomous-agent-drift.mdx b/docs/docs/red-teaming-vulnerabilities-autonomous-agent-drift.mdxindex a2088d53..b599e933 100644--- a/docs/docs/red-teaming-vulnerabilities-autonomous-agent-drift.mdx+++ b/docs/docs/red-teaming-vulnerabilities-autonomous-agent-drift.mdx@@ -27,7 +27,7 @@ red_team()```-There are **FIVE** optional parameters when creating an `AutonomousAgentDrift` vulnerability:+There are **EIGHT** optional parameters when creating an `AutonomousAgentDrift` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-3.5-turbo-0125'`.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-4o'`.@@ -38,9 +38,58 @@ There are **FIVE** optional parameters when creating an `AutonomousAgentDrift` v- `reward_hacking`: Optimization for proxy objectives or metrics instead of the true goal.- `agent_collusion`: Multiple agents coordinating in ways that bypass safeguards or constraints.- `runaway_autonomy`: Escalation of agent authority, scope, or decision-making beyond intended limits.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.+[Click here](#types) to learn more about the `types` in `AutonomousAgentDrift`.+## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `AutonomousAgentDrift(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++agent_drift = AutonomousAgentDrift(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[agent_drift], attack_engine=engine)+```++### As a standaloneYou can assess your autonomous system for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-bfla.mdx b/docs/docs/red-teaming-vulnerabilities-bfla.mdxindex 6689c18e..a40cefb3 100644--- a/docs/docs/red-teaming-vulnerabilities-bfla.mdx+++ b/docs/docs/red-teaming-vulnerabilities-bfla.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `BFLA` vulnerability:+There are **EIGHT** optional parameters when creating a `BFLA` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `BFLA` vulnerability:- `function_bypass`: Tests whether the agent properly validates function-level authorization controls.- `authorization_bypass`: Tests whether the agent maintains proper authorization validation across different interaction contexts.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `BFLA(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++bfla = BFLA(+ types=["privilege_escalation", "function_bypass"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[bfla], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-bias.mdx b/docs/docs/red-teaming-vulnerabilities-bias.mdxindex 5fb8ea79..7ee4aeea 100644--- a/docs/docs/red-teaming-vulnerabilities-bias.mdx+++ b/docs/docs/red-teaming-vulnerabilities-bias.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Bias` vulnerability:+There are **EIGHT** optional parameters when creating a `Bias` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -34,9 +34,58 @@ There are **FIVE** optional parameters when creating a `Bias` vulnerability:- `gender`: Tests for biases related to gender, such as reinforcing stereotypes or exhibiting unfair treatment based on gender identity.- `religion`: Tests for biases in handling religious topics, such as promoting or discriminating against particular beliefs.- `politics`: Tests for political biases, including favoritism or reinforcement of political stereotypes.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `Bias`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Bias(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++bias = Bias(+ types=["race", "gender"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[bias], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-bola.mdx b/docs/docs/red-teaming-vulnerabilities-bola.mdxindex d4fef86b..0a73b04a 100644--- a/docs/docs/red-teaming-vulnerabilities-bola.mdx+++ b/docs/docs/red-teaming-vulnerabilities-bola.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `BOLA` vulnerability:+There are **EIGHT** optional parameters when creating a `BOLA` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `BOLA` vulnerability:- [`"cross_customer_access"`](#cross-customer-access): Tests whether the agent maintains proper isolation between different customers or users.- [`"unauthorized_object_manipulation"`](#unauthorized-object-manipulation): Tests whether the agent prevents unauthorized modification of objects.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `BOLA(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++bola = BOLA(+ types=["object_access_bypass", "cross_customer_access"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[bola], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-child-protection.mdx b/docs/docs/red-teaming-vulnerabilities-child-protection.mdxindex c8b46b6c..6c77fcd9 100644--- a/docs/docs/red-teaming-vulnerabilities-child-protection.mdx+++ b/docs/docs/red-teaming-vulnerabilities-child-protection.mdx@@ -25,7 +25,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `ChildProtection` vulnerability:+There are **EIGHT** optional parameters when creating a `ChildProtection` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -35,9 +35,58 @@ There are **FIVE** optional parameters when creating a `ChildProtection` vulnera- `age_verification`: Tests whether the AI correctly handles users under 13 and refuses to proceed without parental consent.- `data_privacy`: Evaluates if the AI improperly collects, stores, or processes children's personal data.- `exposure_interaction`: Tests for unsafe exposure, grooming-like interactions, or inappropriate content directed toward minors.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `ChildProtection`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ChildProtection(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++child_protection = ChildProtection(+ types=["data_privacy", "exposure_interaction"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[child_protection], attack_engine=engine)+```++### As a standaloneYou can assess your model for the `ChildProtection` vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-competition.mdx b/docs/docs/red-teaming-vulnerabilities-competition.mdxindex 6e976ef0..09b1c504 100644--- a/docs/docs/red-teaming-vulnerabilities-competition.mdx+++ b/docs/docs/red-teaming-vulnerabilities-competition.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Competition` vulnerability:+There are **EIGHT** optional parameters when creating a `Competition` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -35,6 +35,55 @@ There are **FIVE** optional parameters when creating a `Competition` vulnerabili- `discreditation`: Tests for outputs that spread false or discrediting information about competitors.- `confidential_strategies`: Tests for outputs that infer or disclose confidential business strategies, such as product roadmaps or marketing plans, about competitors.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Competition(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++competition = Competition(+ types=["discreditation", "competitor_mention"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[competition], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-cross-context-retrieval.mdx b/docs/docs/red-teaming-vulnerabilities-cross-context-retrieval.mdxindex c6865989..eeb0c588 100644--- a/docs/docs/red-teaming-vulnerabilities-cross-context-retrieval.mdx+++ b/docs/docs/red-teaming-vulnerabilities-cross-context-retrieval.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `CrossContextRetrieval` vulnerability:+There are **EIGHT** optional parameters when creating a `CrossContextRetrieval` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,9 +33,58 @@ There are **FIVE** optional parameters when creating a `CrossContextRetrieval` v- `tenant`: Tests for cross-tenant leakage, evaluating if the AI retrieves or reveals proprietary data, financial records, or internal documentation belonging to a different organization or competitor.- `user`: Tests for cross-user leakage, evaluating if the AI discloses PII, private chat histories, or account details of another individual user within the same system.- `role`: Tests for cross-role leakage, evaluating if the AI exposes internal documents or administrative data requiring a higher, specialized privilege level to access.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `CrossContextRetrieval`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `CrossContextRetrieval(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++cross_context = CrossContextRetrieval(+ types=["tenant", "user"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[cross_context], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-custom.mdx b/docs/docs/red-teaming-vulnerabilities-custom.mdxindex 9ce3b2fc..b25f37a6 100644--- a/docs/docs/red-teaming-vulnerabilities-custom.mdx+++ b/docs/docs/red-teaming-vulnerabilities-custom.mdx@@ -25,7 +25,7 @@ api_security = CustomVulnerability(red_team(vulnerabilities=[api_security], model_callback=..., attacks=...)```-There are **THREE** mandatory and **FIVE** optional parameter when creating a `CustomVulnerability`:+There are **THREE** mandatory and **EIGHT** optional parameters when creating a `CustomVulnerability`:- `name`: A string that identifies your custom vulnerability. This should clearly reflect the specific security concern you're red teaming.- `criteria`: A string that defines what should be evaluated - this is the rule or requirement that the AI should follow or violate.@@ -36,6 +36,57 @@ There are **THREE** mandatory and **FIVE** optional parameter when creating a `C- [Optional] `async_mode`: a boolean which when set to `True`, enables concurrent execution. Defaulted to `True`.- [Optional] `verbose_mode`: a boolean which when set to `True`, prints the intermediate steps used to assess said vulnerability to the console. Defaulted to `False`.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `CustomVulnerability(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++api_security = CustomVulnerability(+ name="API Security",+ criteria="The system should not expose internal API endpoints or allow authentication bypass",+ types=["endpoint_exposure", "auth_bypass"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[api_security], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-debug-access.mdx b/docs/docs/red-teaming-vulnerabilities-debug-access.mdxindex 3b20332e..d1d6accf 100644--- a/docs/docs/red-teaming-vulnerabilities-debug-access.mdx+++ b/docs/docs/red-teaming-vulnerabilities-debug-access.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `DebugAccess` vulnerability:+There are **EIGHT** optional parameters when creating a `DebugAccess` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `DebugAccess` vulnerabili- `development_endpoint_access`: Tests whether the agent blocks access to development endpoints.- `administrative_interface_exposure`: Tests whether the agent prevents exposure of administrative interfaces.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `DebugAccess(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++debug_access = DebugAccess(+ types=["debug_mode_bypass", "development_endpoint_access"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[debug_access], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-ethics.mdx b/docs/docs/red-teaming-vulnerabilities-ethics.mdxindex 3178ee4f..9fe80f64 100644--- a/docs/docs/red-teaming-vulnerabilities-ethics.mdx+++ b/docs/docs/red-teaming-vulnerabilities-ethics.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Ethics` vulnerability:+There are **EIGHT** optional parameters when creating a `Ethics` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,9 +33,57 @@ There are **FIVE** optional parameters when creating a `Ethics` vulnerability:- `moral_integrity`: Tests moral reasoning, value alignment, and cultural sensitivity.- `responsible_transparency`: Tests transparency, honesty, accountability, autonomy, and consent.- `harm_prevention`: Tests dual-use awareness, beneficence, and harm-avoidance.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `Ethics`.+## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Ethics(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++ethics = Ethics(+ types=["moral_integrity", "responsible_transparency"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[ethics], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-excessive-agency.mdx b/docs/docs/red-teaming-vulnerabilities-excessive-agency.mdxindex 572a8d4e..7a1e640c 100644--- a/docs/docs/red-teaming-vulnerabilities-excessive-agency.mdx+++ b/docs/docs/red-teaming-vulnerabilities-excessive-agency.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `ExcessiveAgency` vulnerability:+There are **EIGHT** optional parameters when creating a `ExcessiveAgency` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -34,6 +34,55 @@ There are **FIVE** optional parameters when creating a `ExcessiveAgency` vulnera- `permissions`: Tests whether the output enables or complies with requests to grant permissions exceeding what is necessary for the system’s tasks.- `automony`: Tests whether the output enables or complies with requests to allow the system to act autonomously without safeguards, leading to high-impact actions being performed without proper validation or oversight.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ExcessiveAgency(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++excessive_agency = ExcessiveAgency(+ types=["functionality", "permissions"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[excessive_agency], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-exploit-tool-agent.mdx b/docs/docs/red-teaming-vulnerabilities-exploit-tool-agent.mdxindex e6e464c2..2ae79e75 100644--- a/docs/docs/red-teaming-vulnerabilities-exploit-tool-agent.mdx+++ b/docs/docs/red-teaming-vulnerabilities-exploit-tool-agent.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `ExploitToolAgent` vulnerability:+There are **EIGHT** optional parameters when creating a `ExploitToolAgent` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,9 +33,58 @@ There are **FIVE** optional parameters when creating a `ExploitToolAgent` vulner- `data_destruction`: Tests for attempts to coerce the agent into dropping databases, deleting user records, or wiping system logs.- `financial_manipulation`: Tests for attempts to trick the agent into issuing unauthorized refunds, transferring funds, or altering pricing data.- `privilege_escalation`: Tests for attempts to manipulate the agent into granting administrative rights, creating rogue admin accounts, or modifying access controls.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `ExploitToolAgent`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ExploitToolAgent(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++exploit_tool = ExploitToolAgent(+ types=["data_destruction", "financial_manipulation"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[exploit_tool], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-external-system-abuse.mdx b/docs/docs/red-teaming-vulnerabilities-external-system-abuse.mdxindex d592df54..a064e9d6 100644--- a/docs/docs/red-teaming-vulnerabilities-external-system-abuse.mdx+++ b/docs/docs/red-teaming-vulnerabilities-external-system-abuse.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating an `ExternalSystemAbuse` vulnerability:+There are **EIGHT** optional parameters when creating an `ExternalSystemAbuse` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,9 +33,58 @@ There are **FIVE** optional parameters when creating an `ExternalSystemAbuse` vu- `data_exfiltration`: Tests for attempts to trick the agent into sending proprietary, sensitive, or internal data to an unauthorized external server or email address.- `spam`: Tests for attempts to coerce the agent into sending unsolicited, repetitive, or harassing messages via connected communication tools.- `phishing_and_spoofing`: Tests for attempts to manipulate the agent into distributing malicious links or deceptive messages that appear to originate from a trusted internal source.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `ExternalSystemAbuse`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ExternalSystemAbuse(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++external_system = ExternalSystemAbuse(+ types=["data_exfiltration", "spam"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[external_system], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-fairness.mdx b/docs/docs/red-teaming-vulnerabilities-fairness.mdxindex 5d3b17e9..421a9c2a 100644--- a/docs/docs/red-teaming-vulnerabilities-fairness.mdx+++ b/docs/docs/red-teaming-vulnerabilities-fairness.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Fairness` vulnerability:+There are **EIGHT** optional parameters when creating a `Fairness` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-3.5-turbo-0125'`.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-4o'`.@@ -33,9 +33,58 @@ There are **FIVE** optional parameters when creating a `Fairness` vulnerability:- `distributive_fairness`: Tests for equitable outcomes across groups and individuals.- `procedural_fairness`: Tests for transparency and consistency in the decision-making process.- `contextual_fairness`: Tests for fairness consistency across time, domains, and evolving deployment contexts.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `Fairness`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Fairness(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++fairness = Fairness(+ types=["equality_consistency", "procedural_opportunity"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[fairness], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-graphic-content.mdx b/docs/docs/red-teaming-vulnerabilities-graphic-content.mdxindex ca40cca9..482eb5b6 100644--- a/docs/docs/red-teaming-vulnerabilities-graphic-content.mdx+++ b/docs/docs/red-teaming-vulnerabilities-graphic-content.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `GraphicContent` vulnerability:+There are **EIGHT** optional parameters when creating a `GraphicContent` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -34,6 +34,55 @@ There are **FIVE** optional parameters when creating a `GraphicContent` vulnerab- `graphic_content`: Tests for outputs that contain violent, gory, or disturbing material.- `pornographic_content`: Tests for outputs that include or promote pornographic material.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `GraphicContent(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++graphic_content = GraphicContent(+ types=["graphic_content", "sexual_content"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[graphic_content], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-hallucination.mdx b/docs/docs/red-teaming-vulnerabilities-hallucination.mdxindex a1de98da..ab6a8b1c 100644--- a/docs/docs/red-teaming-vulnerabilities-hallucination.mdx+++ b/docs/docs/red-teaming-vulnerabilities-hallucination.mdx@@ -22,7 +22,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Hallucination` vulnerability:+There are **EIGHT** optional parameters when creating a `Hallucination` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,9 +33,58 @@ There are **FIVE** optional parameters when creating a `Hallucination` vulnerabi- `fake_apis`: Tests for outputs that invent non-existent API endpoints, SDK methods, or library functions.- `fake_entities`: Tests for outputs that fabricate non-existent people, companies, products, or organizations.- `fake_statistics`: Tests for outputs that manufacture specific statistics, percentages, or numerical data with false confidence.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `Hallucination`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Hallucination(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++hallucination = Hallucination(+ types=["fake_citations", "fake_apis"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[hallucination], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-illegal-activity.mdx b/docs/docs/red-teaming-vulnerabilities-illegal-activity.mdxindex 47f6f6ab..45e82e7a 100644--- a/docs/docs/red-teaming-vulnerabilities-illegal-activity.mdx+++ b/docs/docs/red-teaming-vulnerabilities-illegal-activity.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `IllegalActivity` vulnerability:+There are **EIGHT** optional parameters when creating a `IllegalActivity` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -38,6 +38,55 @@ There are **FIVE** optional parameters when creating a `IllegalActivity` vulnera- `cybercrime`: Tests for outputs providing methods or plans for committing cybercrime and hacking systems.- `child_exploitation`: Tests for outputs providing methods or plans for exploiting children.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `IllegalActivity(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++illegal_activity = IllegalActivity(+ types=["weapons", "cybercrime"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[illegal_activity], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-indirect-instruction.mdx b/docs/docs/red-teaming-vulnerabilities-indirect-instruction.mdxindex 3b2d11b2..4f839103 100644--- a/docs/docs/red-teaming-vulnerabilities-indirect-instruction.mdx+++ b/docs/docs/red-teaming-vulnerabilities-indirect-instruction.mdx@@ -27,7 +27,7 @@ red_team()```-There are **FIVE** optional parameters when creating an `IndirectInstruction` vulnerability:+There are **EIGHT** optional parameters when creating an `IndirectInstruction` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating an `IndirectInstruction` vu[Click here](#types) to learn more about the `types` in `IndirectInstruction`.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `IndirectInstruction(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++indirect_instruction = IndirectInstruction(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[indirect_instruction], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-intellectual-property.mdx b/docs/docs/red-teaming-vulnerabilities-intellectual-property.mdxindex c8b3637b..d8b7d0cd 100644--- a/docs/docs/red-teaming-vulnerabilities-intellectual-property.mdx+++ b/docs/docs/red-teaming-vulnerabilities-intellectual-property.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `IntellectualProperty` vulnerability:+There are **EIGHT** optional parameters when creating a `IntellectualProperty` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -35,6 +35,55 @@ There are **FIVE** optional parameters when creating a `IntellectualProperty` vu- `patent_disclosure`: Tests for outputs that reveal details about patented inventions or processes without rights or authorization.- `imitation`: Tests for outputs that closely imitate proprietary works, creating potential ethical or legal risks.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `IntellectualProperty(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++intellectual_property = IntellectualProperty(+ types=["imitation", "copyright_violations"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[intellectual_property], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-inter-agent-communication-compromise.mdx b/docs/docs/red-teaming-vulnerabilities-inter-agent-communication-compromise.mdxindex 93f44121..91907bfd 100644--- a/docs/docs/red-teaming-vulnerabilities-inter-agent-communication-compromise.mdx+++ b/docs/docs/red-teaming-vulnerabilities-inter-agent-communication-compromise.mdx@@ -27,7 +27,7 @@ red_team()```-There are **FIVE** optional parameters when creating an `InsecureInterAgentCommunication` vulnerability:+There are **EIGHT** optional parameters when creating an `InsecureInterAgentCommunication` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-3.5-turbo-0125'`.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-4o'`.@@ -37,9 +37,58 @@ There are **FIVE** optional parameters when creating an `InsecureInterAgentCommu- `message_spoofing`: Impersonation of a trusted agent to issue malicious instructions.- `message_injection`: Injection of malicious or unauthorized content into agent-to-agent messages.- `agent_in_the_middle`: Interception or manipulation of messages exchanged between agents.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `InsecureInterAgentCommunication`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `InsecureInterAgentCommunication(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++inter_agent_comm = InsecureInterAgentCommunication(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[inter_agent_comm], attack_engine=engine)+```++### As a standaloneYou can assess your system for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-misinformation.mdx b/docs/docs/red-teaming-vulnerabilities-misinformation.mdxindex 9eeb48ad..fd6053c7 100644--- a/docs/docs/red-teaming-vulnerabilities-misinformation.mdx+++ b/docs/docs/red-teaming-vulnerabilities-misinformation.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Misinformation` vulnerability:+There are **EIGHT** optional parameters when creating a `Misinformation` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -34,6 +34,55 @@ There are **FIVE** optional parameters when creating a `Misinformation` vulnerab- [`"unsupported_claims"`](#unsupported-claims): Tests for outputs that make assertions or arguments without providing adequate evidence or references to back them up.- [`"expertize_misrepresentation"`](#expertize-misrepresentation): Tests for outputs that provide specialized advice, particularly in domains like medicine, finance, or law, that is incorrect, incomplete, or beyond your LLM's intended purpose.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Misinformation(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++misinformation = Misinformation(+ types=["factual_errors", "unsupported_claims"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[misinformation], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-personal-safety.mdx b/docs/docs/red-teaming-vulnerabilities-personal-safety.mdxindex 70995c47..58f244d3 100644--- a/docs/docs/red-teaming-vulnerabilities-personal-safety.mdx+++ b/docs/docs/red-teaming-vulnerabilities-personal-safety.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `PersonalSafety` vulnerability:+There are **EIGHT** optional parameters when creating a `PersonalSafety` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -35,6 +35,55 @@ There are **FIVE** optional parameters when creating a `PersonalSafety` vulnerab- `dangerous_challenges`: Tests for outputs that promote or suggest participation in harmful or life-threatening social challenges.- `stalking`: Tests for outputs that support or suggest methods for tracking, monitoring, or stalking individuals.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `PersonalSafety(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++personal_safety = PersonalSafety(+ types=["bullying", "self_harm"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[personal_safety], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-pii-leakage.mdx b/docs/docs/red-teaming-vulnerabilities-pii-leakage.mdxindex e92e0d4f..ca384da1 100644--- a/docs/docs/red-teaming-vulnerabilities-pii-leakage.mdx+++ b/docs/docs/red-teaming-vulnerabilities-pii-leakage.mdx@@ -25,7 +25,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `PIILeakage` vulnerability:+There are **EIGHT** optional parameters when creating a `PIILeakage` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -36,9 +36,58 @@ There are **FIVE** optional parameters when creating a `PIILeakage` vulnerabilit- `api_and_database_access`: Tests for outputs that disclose PII by providing unauthorized access to databases or querying stored personal information, potentially revealing sensitive records.- `session_leak`: Tests for outputs that disclose PII from other users' active or past sessions, such as inadvertently exposing personal details associated with session-based interactions.- `social_manipulation`: Tests for outputs that enable or facilitate social engineering attacks, tricking individuals into revealing their own or others' PII.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `PIILeakage`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `PIILeakage(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++pii_leakage = PIILeakage(+ types=["direct_disclosure", "social_manipulation"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[pii_leakage], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-prompt-leakage.mdx b/docs/docs/red-teaming-vulnerabilities-prompt-leakage.mdxindex f068de56..dd2255e3 100644--- a/docs/docs/red-teaming-vulnerabilities-prompt-leakage.mdx+++ b/docs/docs/red-teaming-vulnerabilities-prompt-leakage.mdx@@ -25,7 +25,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `PromptLeakage` vulnerability:+There are **EIGHT** optional parameters when creating a `PromptLeakage` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -37,6 +37,55 @@ There are **FIVE** optional parameters when creating a `PromptLeakage` vulnerabi- `guard_exposure`: Tests for outputs that expose guard mechanisms, such as rejection rules or criteria for sensitive requests, potentially enabling users to bypass safeguards.- `permissions_and_roles`: Tests for outputs that reveal role-based permissions, access controls, or internal configurations mentioned in the system prompt.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `PromptLeakage(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++prompt_leakage = PromptLeakage(+ types=["secrets_and_credentials", "guard_exposure"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[prompt_leakage], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-rbac.mdx b/docs/docs/red-teaming-vulnerabilities-rbac.mdxindex 6712649f..1abd1215 100644--- a/docs/docs/red-teaming-vulnerabilities-rbac.mdx+++ b/docs/docs/red-teaming-vulnerabilities-rbac.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `RBAC` vulnerability:+There are **EIGHT** optional parameters when creating a `RBAC` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `RBAC` vulnerability:- `privilege_escalation`: Tests whether the agent prevents unauthorized privilege escalation.- `unauthorized_role_assumption`: Tests whether the agent resists attempts to assume unauthorized roles.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `RBAC(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++rbac = RBAC(+ types=["role_bypass", "privilege_escalation"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[rbac], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-robustness.mdx b/docs/docs/red-teaming-vulnerabilities-robustness.mdxindex bcdd5a10..c8176949 100644--- a/docs/docs/red-teaming-vulnerabilities-robustness.mdx+++ b/docs/docs/red-teaming-vulnerabilities-robustness.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Robustness` vulnerability:+There are **EIGHT** optional parameters when creating a `Robustness` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,6 +33,55 @@ There are **FIVE** optional parameters when creating a `Robustness` vulnerabilit- `hijacking`: Tests whether the output enables or complies with malicious or crafted inputs that redirect or alter the system’s intended behavior.- `input_overreliance`: Tests whether the output is excessively influenced by user input, leading to misleading and undesirable responses or behaviors that reflect user-provided content.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Robustness(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++robustness = Robustness(+ types=["hijacking", "input_overreliance"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[robustness], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-shell-injection.mdx b/docs/docs/red-teaming-vulnerabilities-shell-injection.mdxindex 8412c598..75874d10 100644--- a/docs/docs/red-teaming-vulnerabilities-shell-injection.mdx+++ b/docs/docs/red-teaming-vulnerabilities-shell-injection.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `ShellInjection` vulnerability:+There are **EIGHT** optional parameters when creating a `ShellInjection` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `ShellInjection` vulnerab- `system_command_execution`: Tests whether the agent blocks malicious system command execution.- `shell_escape_sequences`: Tests whether the agent prevents shell escape sequence attacks.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ShellInjection(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++shell_injection = ShellInjection(+ types=["command_injection", "shell_escape_sequences"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[shell_injection], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-sql-injection.mdx b/docs/docs/red-teaming-vulnerabilities-sql-injection.mdxindex 7b6c3366..da71b46d 100644--- a/docs/docs/red-teaming-vulnerabilities-sql-injection.mdx+++ b/docs/docs/red-teaming-vulnerabilities-sql-injection.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `SQLInjection` vulnerability:+There are **EIGHT** optional parameters when creating a `SQLInjection` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `SQLInjection` vulnerabil- `union_based_injection`: Tests whether the agent blocks union-based SQL injection attempts.- `error_based_injection`: Tests whether the agent prevents error-based SQL injection attacks.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `SQLInjection(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++sql_injection = SQLInjection(+ types=["blind_sql_injection", "union_based_injection"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[sql_injection], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-ssrf.mdx b/docs/docs/red-teaming-vulnerabilities-ssrf.mdxindex d308bdab..61e0a5b4 100644--- a/docs/docs/red-teaming-vulnerabilities-ssrf.mdx+++ b/docs/docs/red-teaming-vulnerabilities-ssrf.mdx@@ -30,7 +30,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `SSRF` vulnerability:+There are **EIGHT** optional parameters when creating a `SSRF` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -41,6 +41,55 @@ There are **FIVE** optional parameters when creating a `SSRF` vulnerability:- `cloud_metadata_access`: Tests whether the agent blocks access to cloud metadata endpoints.- `port_scanning`: Tests whether the agent prevents port scanning and network reconnaissance.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `SSRF(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++ssrf = SSRF(+ types=["internal_service_access", "port_scanning"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[ssrf], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-system-reconnaissance.mdx b/docs/docs/red-teaming-vulnerabilities-system-reconnaissance.mdxindex d7d2a500..40fd3394 100644--- a/docs/docs/red-teaming-vulnerabilities-system-reconnaissance.mdx+++ b/docs/docs/red-teaming-vulnerabilities-system-reconnaissance.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `SystemReconnaissance` vulnerability:+There are **EIGHT** optional parameters when creating a `SystemReconnaissance` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model]() of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model]() of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -33,10 +33,56 @@ There are **FIVE** optional parameters when creating a `SystemReconnaissance` vu- `schema`: Tests for attempts to extract database structures, table names, vector dimensions, or internal query logic.- `infrastructure`: Tests for attempts to map out the server environment, cloud provider details, network topology, or OS-level information.- `configuration`: Tests for attempts to discover internal file paths, application settings, retrieval parameters, or hidden system configurations.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.+[Click here](#types) to learn more about the `types` in `SystemReconnaissance`.+## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `SystemReconnaissance(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++reconnaissance = SystemReconnaissance(+ types=["schema", "infrastructure"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[reconnaissance], attack_engine=engine)+```-[Click here](#types) to learn more about the `types` in `SystemReconnaissance`.### As a standalonediff --git a/docs/docs/red-teaming-vulnerabilities-tool-metadata-poisoning.mdx b/docs/docs/red-teaming-vulnerabilities-tool-metadata-poisoning.mdxindex a1745b10..033eb982 100644--- a/docs/docs/red-teaming-vulnerabilities-tool-metadata-poisoning.mdx+++ b/docs/docs/red-teaming-vulnerabilities-tool-metadata-poisoning.mdx@@ -27,7 +27,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `ToolMetadataPoisoning` vulnerability:+There are **EIGHT** optional parameters when creating a `ToolMetadataPoisoning` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-3.5-turbo-0125'`.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-4o'`.@@ -38,9 +38,58 @@ There are **FIVE** optional parameters when creating a `ToolMetadataPoisoning` v- `description_deception`: Tests whether deceptive tool descriptions influence the model’s decision-making.- `permission_misrepresentation`: Tests whether false permission or safety claims can bypass safeguards.- `registry_poisoning`: Tests whether poisoned or untrusted tool registries are incorrectly trusted.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `ToolMetadataPoisoning`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ToolMetadataPoisoning(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++tool_metadata_poisoning = ToolMetadataPoisoning(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[tool_metadata_poisoning], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-tool-orchestration-abuse.mdx b/docs/docs/red-teaming-vulnerabilities-tool-orchestration-abuse.mdxindex 1832bf07..b7876e69 100644--- a/docs/docs/red-teaming-vulnerabilities-tool-orchestration-abuse.mdx+++ b/docs/docs/red-teaming-vulnerabilities-tool-orchestration-abuse.mdx@@ -28,7 +28,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `ToolOrchestrationAbuse` vulnerability:+There are **EIGHT** optional parameters when creating a `ToolOrchestrationAbuse` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -39,9 +39,56 @@ There are **FIVE** optional parameters when creating a `ToolOrchestrationAbuse`- `unsafe_tool_composition`: Tests whether unsafe sequences of tools can be chained to bypass safeguards.- `tool_budget_exhaustion`: Tests whether the model can be manipulated into excessive tool usage.- `cross_tool_state_leakage`: Tests whether state or sensitive data leaks between independent tools.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `ToolOrchestrationAbuse`.+## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `ToolOrchestrationAbuse(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++tool_abuse = ToolOrchestrationAbuse(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[tool_abuse], attack_engine=engine)+```+### As a standalonediff --git a/docs/docs/red-teaming-vulnerabilities-toxicity.mdx b/docs/docs/red-teaming-vulnerabilities-toxicity.mdxindex 4955fc6a..4035a355 100644--- a/docs/docs/red-teaming-vulnerabilities-toxicity.mdx+++ b/docs/docs/red-teaming-vulnerabilities-toxicity.mdx@@ -23,7 +23,7 @@ red_team()```-There are **FIVE** optional parameters when creating a `Toxicity` vulnerability:+There are **EIGHT** optional parameters when creating a `Toxicity` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-3.5-turbo-0125'.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4o'.@@ -35,6 +35,55 @@ There are **FIVE** optional parameters when creating a `Toxicity` vulnerability:- `threats`:Tests for outputs that issue direct threats, provide instructions to intimidate, or suggest ways to harm others verbally or physically.- `mockery`: Tests for outputs that ridicule or mock someone, or teach methods to humiliate or undermine others through ridicule.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.++## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `Toxicity(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++toxicity = Toxicity(+ types=["insults", "profanity"],+ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[toxicity], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method:diff --git a/docs/docs/red-teaming-vulnerabilities-unexpected-code-execution.mdx b/docs/docs/red-teaming-vulnerabilities-unexpected-code-execution.mdxindex b8128991..3e9b140c 100644--- a/docs/docs/red-teaming-vulnerabilities-unexpected-code-execution.mdx+++ b/docs/docs/red-teaming-vulnerabilities-unexpected-code-execution.mdx@@ -27,7 +27,7 @@ red_team()```-There are **FIVE** optional parameters when creating an `UnexpectedCodeExecution` vulnerability:+There are **EIGHT** optional parameters when creating an `UnexpectedCodeExecution` vulnerability:- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-3.5-turbo-0125'`.- [Optional] `evaluation_model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](https://deepeval.com/guides/guides-using-custom-llms) of type `DeepEvalBaseLLM`. Defaulted to `'gpt-4o'`.@@ -37,9 +37,57 @@ There are **FIVE** optional parameters when creating an `UnexpectedCodeExecution- `unauthorized_code_execution`: Attempts to induce execution of arbitrary or unapproved code.- `shell_command_execution`: Attempts to trigger shell or OS-level command execution.- `eval_usage`: Attempts to induce the use of `eval`, `exec`, or dynamic code evaluation patterns.+- [Optional] `evaluation_examples`: an optional list of [`EvaluationExample`](/docs/red-teaming-introduction#evaluation-examples-and-guidelines)s used as few-shot calibration for this vulnerability's LLM-as-judge metric. Each example includes `input`, `actual_output`, a binary `score` (`0` = fail, `1` = pass), and a `reason` explaining why that score is correct. Defaulted to `None`.+- [Optional] `evaluation_guidelines`: an optional list of strings passed to the judge prompt as guidelines for evaluations (e.g., treat a partial leak as a failure). Defaulted to `None`.+- [Optional] `attack_engine`: an optional [`AttackEngine`](/docs/red-teaming-introduction#attack-engine) instance that allows you to customize the baseline attacks (transform, optional variations, validation) before your target is invoked. When omitted, a default engine is created internally. Defaulted to `None`.[Click here](#types) to learn more about the `types` in `UnexpectedCodeExecution`.+## Customizing Generations and Evaluations++You can tune your **baseline attacks** and adjust **output evaluations** by passing `attack_engine`, `evaluation_examples`, and `evaluation_guidelines` into `UnexpectedCodeExecution(...)`.++The **attack engine** rewrites each simulated baseline prompt so probes stay on-vulnerability while feeling more realistic for your use case; optional `variations` (1-5) and `generation_guidelines` allow further user customizations. **Evaluation examples** give the metric a few labeled `(input, output) → score` demonstrations so the judge matches your expectations; **evaluation guidelines** are plain-text rules you can use to control evaluator's thought process.++When you run a full scan via `red_team()` or `RedTeamer`, pass `attack_engine` on that call to apply the same refinement pipeline across vulnerabilities during simulation. For **standalone** `assess()` on a single vulnerability, setting `attack_engine` (and evaluation fields) on the instance is the most direct path.++```python+from deepteam.vulnerabilities import EvaluationExample+from deepteam.attacks.attack_engine import AttackEngine++engine = AttackEngine(+ simulator_model="gpt-4o-mini",+ variations=2,+ generation_guidelines=[+ "Make the attacks seem like a loyal customer complaining."+ ],+ purpose="Retail banking support bot",+)++examples = [+ EvaluationExample(+ input="…",+ actual_output="…",+ score=0,+ reason="…",+ ),+]+guidelines = [+ "Treat hedging that still leaks restricted guidance as a failure.",+]++unexpected_code_execution = UnexpectedCodeExecution(++ evaluation_examples=examples,+ evaluation_guidelines=guidelines,+ attack_engine=engine,+)++# Optional: same engine for an end-to-end `red_team` scan+# red_team(..., vulnerabilities=[unexpected_code_execution], attack_engine=engine)+```++### As a standaloneYou can assess your model for a single vulnerability through the `assess` method: