Skip to content

Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.

fix(flow): learn from resumed human feedback and keep the reviewed output's type (#8142)

João Moura75808b0eca0c
diff --git a/docs/edge/ar/learn/human-feedback-in-flows.mdx b/docs/edge/ar/learn/human-feedback-in-flows.mdx
index 11aeffcf6..89a74d8c4 100644
--- a/docs/edge/ar/learn/human-feedback-in-flows.mdx
+++ b/docs/edge/ar/learn/human-feedback-in-flows.mdx
@@ -77,6 +77,12 @@ class ReviewFlow(Flow):
معامل `learn=True` يمكّن حلقة تغذية راجعة بين المراجعين البشريين ونظام الذاكرة. عند تمكينه، يحسّن النظام مخرجاته تدريجيًا بالتعلم من التصحيحات البشرية السابقة.
+تُستخلص الدروس من كل رد، سواء وصل أثناء التشغيل أو استأنف لاحقًا Flow متوقفًا عبر `provider` غير متزامن. قبل المراجعة التالية تُطبَّق الدروس السابقة على المخرجات: يُعاد كتابة النص نصًا، ونموذج Pydantic بالنموذج نفسه ما لم يحتوِ على ما لا يراه LLM: قيمة سرية أو حقل مستبعد أو سمة خاصة تغيّرت عن قيمتها الافتراضية، وتُعرض المخرجات الأخرى مثل `dict` أو `list` دون تغيير.
+
+<Note>
+يوافق الإنسان على النسخة التي رُوجعت، وقد تختلف عمّا أعادته الدالة. في الخطوات التالية اقرأها من `self.last_human_feedback.output` (أو من وسيط المستمع)، فالحالة التي ضبطتها الدالة قبل المراجعة ما زالت تحتفظ بالنسخة الأصلية. وإذا توقف Flow بانتظار تغذية راجعة غير متزامنة، تُستعاد المخرجات من JSON عند الاستئناف، فيصل نموذج Pydantic على شكل `dict`.
+</Note>
+
## أفضل الممارسات
1. **اكتب رسائل طلب واضحة**
diff --git a/docs/edge/en/learn/human-feedback-in-flows.mdx b/docs/edge/en/learn/human-feedback-in-flows.mdx
index 0c3792bca..180c3b80c 100644
--- a/docs/edge/en/learn/human-feedback-in-flows.mdx
+++ b/docs/edge/en/learn/human-feedback-in-flows.mdx
@@ -74,7 +74,8 @@ When this flow runs, it will:
| `metadata` | `dict` | No | Additional data for enterprise integrations |
| `provider` | `HumanFeedbackProvider` | No | Custom provider for async/non-blocking feedback. See [Async Human Feedback](#async-human-feedback-non-blocking) |
| `learn` | `bool` | No | Enable HITL learning: distill lessons from feedback and pre-review future output. Default `False`. See [Learning from Feedback](#learning-from-feedback) |
-| `learn_limit` | `int` | No | Max past lessons to recall for pre-review. Default `5` |
+| `learn_source` | `str` | No | Memory `source` that lessons are stored under and recalled from. Default `"hitl"` |
+| `learn_strict` | `bool` | No | Raise when pre-review or lesson distillation fails, instead of falling back. Default `False` |
### Basic Usage (No Routing)
@@ -645,11 +646,15 @@ The `learn=True` parameter enables a feedback loop between human reviewers and t
### How It Works
-1. **After feedback**: The LLM extracts generalizable lessons from the output + feedback and stores them in memory with `source="hitl"`. If the feedback is just approval (e.g. "looks good"), nothing is stored.
-2. **Before next review**: Past HITL lessons are recalled from memory and applied by the LLM to improve the output before the human sees it.
+1. **After feedback**: The LLM extracts generalizable lessons from the output + feedback and stores them in memory under `learn_source` (default `"hitl"`). This happens whether the reply arrives during the run or later resumes a paused flow through an [async provider](#async-human-feedback-non-blocking). If the feedback is just approval (e.g. "looks good"), nothing is stored.
+2. **Before next review**: Past HITL lessons are recalled from memory and applied by the LLM to improve the output before the human sees it. Text output is rewritten as text, and a Pydantic model is rewritten as the same model unless it holds something the LLM can't see: a secret, an excluded field, or a private attribute changed from its default. Any other output, such as a `dict` or a `list`, is shown unchanged.
Over time, the human sees progressively better pre-reviewed output because each correction informs future reviews.
+<Note>
+The human approves the reviewed version, which can differ from what the method returned. Downstream, read it from `self.last_human_feedback.output` (or the listener's argument). State the method set before the review still holds the original. If the flow paused for async feedback, the output is restored from JSON on resume, so a Pydantic model arrives as a `dict`.
+</Note>
+
### Example
```python Code
@@ -682,14 +687,15 @@ class ArticleReviewFlow(Flow):
| Parameter | Default | Description |
|-----------|---------|-------------|
| `learn` | `False` | Enable HITL learning |
-| `learn_limit` | `5` | Max past lessons to recall for pre-review |
+| `learn_source` | `"hitl"` | Memory source that lessons are stored under and recalled from |
+| `learn_strict` | `False` | Raise on pre-review or distillation failures instead of falling back |
### Key Design Decisions
- **Same LLM for everything**: The `llm` parameter on the decorator is shared by outcome collapsing, lesson distillation, and pre-review. No need to configure multiple models.
- **Structured output**: Both distillation and pre-review use function calling with Pydantic models when the LLM supports it, falling back to text parsing otherwise.
- **Non-blocking storage**: Lessons are stored via `remember_many()` which runs in a background thread -- the flow continues immediately.
-- **Graceful degradation**: If the LLM fails during distillation, nothing is stored. If it fails during pre-review, the raw output is shown. Neither failure blocks the flow.
+- **Graceful degradation**: If the LLM fails during distillation, nothing is stored. If it fails during pre-review, the raw output is shown. Neither failure blocks the flow unless `learn_strict=True`, which raises instead.
- **No scope/categories needed**: When storing lessons, only `source` is passed. The encoding pipeline infers scope, categories, and importance automatically.
<Note>
diff --git a/docs/edge/ko/learn/human-feedback-in-flows.mdx b/docs/edge/ko/learn/human-feedback-in-flows.mdx
index a6305ca8a..fc490f201 100644
--- a/docs/edge/ko/learn/human-feedback-in-flows.mdx
+++ b/docs/edge/ko/learn/human-feedback-in-flows.mdx
@@ -74,7 +74,8 @@ flow.kickoff()
| `metadata` | `dict` | 아니오 | 엔터프라이즈 통합을 위한 추가 데이터 |
| `provider` | `HumanFeedbackProvider` | 아니오 | 비동기/논블로킹 피드백을 위한 커스텀 프로바이더. [비동기 인간 피드백](#비동기-인간-피드백-논블로킹) 참조 |
| `learn` | `bool` | 아니오 | HITL 학습 활성화: 피드백에서 교훈을 추출하고 향후 출력을 사전 검토합니다. 기본값 `False`. [피드백에서 학습하기](#피드백에서-학습하기) 참조 |
-| `learn_limit` | `int` | 아니오 | 사전 검토를 위해 불러올 최대 과거 교훈 수. 기본값 `5` |
+| `learn_source` | `str` | 아니오 | 교훈을 저장하고 불러올 메모리 `source`. 기본값 `"hitl"` |
+| `learn_strict` | `bool` | 아니오 | 사전 검토나 교훈 추출이 실패하면 대체 동작 대신 예외를 발생시킵니다. 기본값 `False` |
### 기본 사용법 (라우팅 없음)
@@ -635,11 +636,15 @@ async def on_slack_feedback_async(flow_id: str, slack_message: str):
### 작동 방식
-1. **피드백 후**: LLM이 출력 + 피드백에서 일반화 가능한 교훈을 추출하고 `source="hitl"`로 메모리에 저장합니다. 피드백이 단순한 승인(예: "좋아 보입니다")인 경우 아무것도 저장하지 않습니다.
-2. **다음 검토 전**: 과거 HITL 교훈을 메모리에서 불러와 LLM이 인간이 보기 전에 출력을 개선하는 데 적용합니다.
+1. **피드백 후**: LLM이 출력 + 피드백에서 일반화 가능한 교훈을 추출하고 `learn_source`(기본값 `"hitl"`)로 메모리에 저장합니다. 응답이 실행 중에 도착하든, 나중에 [비동기 프로바이더](#비동기-인간-피드백-논블로킹)를 통해 일시 중지된 Flow를 재개하든 동일하게 동작합니다. 피드백이 단순한 승인(예: "좋아 보입니다")인 경우 아무것도 저장하지 않습니다.
+2. **다음 검토 전**: 과거 HITL 교훈을 메모리에서 불러와 LLM이 인간이 보기 전에 출력을 개선하는 데 적용합니다. 텍스트 출력은 텍스트로, Pydantic 모델은 같은 모델로 다시 작성됩니다. 단, LLM이 볼 수 없는 값(시크릿, 제외된 필드, 기본값에서 변경된 private 속성)이 있는 모델은 그대로 표시됩니다. `dict`나 `list` 같은 그 밖의 출력도 그대로 표시됩니다.
시간이 지남에 따라 각 수정 사항이 향후 검토에 반영되므로 인간은 점진적으로 더 나은 사전 검토된 출력을 보게 됩니다.
+<Note>
+인간이 승인하는 것은 검토된 버전이며, 메서드가 반환한 값과 다를 수 있습니다. 이후 단계에서는 `self.last_human_feedback.output`(또는 리스너의 인자)에서 읽으세요. 메서드가 검토 전에 설정한 상태에는 여전히 원본이 남아 있습니다. Flow가 비동기 피드백을 위해 일시 중지되었다면 재개 시 출력이 JSON에서 복원되므로 Pydantic 모델은 `dict`로 전달됩니다.
+</Note>
+
### 예제
```python Code
@@ -672,14 +677,15 @@ class ArticleReviewFlow(Flow):
| 매개변수 | 기본값 | 설명 |
|-----------|--------|------|
| `learn` | `False` | HITL 학습 활성화 |
-| `learn_limit` | `5` | 사전 검토를 위해 불러올 최대 과거 교훈 수 |
+| `learn_source` | `"hitl"` | 교훈을 저장하고 불러올 메모리 source |
+| `learn_strict` | `False` | 사전 검토나 추출 실패 시 대체 동작 대신 예외 발생 |
### 주요 설계 결정
- **모든 것에 동일한 LLM 사용**: 데코레이터의 `llm` 매개변수는 outcome 매핑, 교훈 추출, 사전 검토에 공유됩니다. 여러 모델을 구성할 필요가 없습니다.
- **구조화된 출력**: 추출과 사전 검토 모두 LLM이 지원하는 경우 Pydantic 모델과 함께 function calling을 사용하고, 그렇지 않으면 텍스트 파싱으로 폴백합니다.
- **논블로킹 저장**: 교훈은 백그라운드 스레드에서 실행되는 `remember_many()`를 통해 저장됩니다 -- Flow는 즉시 계속됩니다.
-- **우아한 저하**: 추출 중 LLM이 실패하면 아무것도 저장하지 않습니다. 사전 검토 중 실패하면 원시 출력이 표시됩니다. 어느 쪽의 실패도 Flow를 차단하지 않습니다.
+- **우아한 저하**: 추출 중 LLM이 실패하면 아무것도 저장하지 않습니다. 사전 검토 중 실패하면 원시 출력이 표시됩니다. `learn_strict=True`로 예외를 발생시키지 않는 한 어느 쪽의 실패도 Flow를 차단하지 않습니다.
- **범위/카테고리 불필요**: 교훈을 저장할 때 `source`만 전달됩니다. 인코딩 파이프라인이 범위, 카테고리, 중요도를 자동으로 추론합니다.
<Note>
diff --git a/docs/edge/pt-BR/learn/human-feedback-in-flows.mdx b/docs/edge/pt-BR/learn/human-feedback-in-flows.mdx
index ad4d068cd..74693caea 100644
--- a/docs/edge/pt-BR/learn/human-feedback-in-flows.mdx
+++ b/docs/edge/pt-BR/learn/human-feedback-in-flows.mdx
@@ -74,7 +74,8 @@ Quando este flow é executado, ele irá:
| `metadata` | `dict` | Não | Dados adicionais para integrações enterprise |
| `provider` | `HumanFeedbackProvider` | Não | Provider customizado para feedback assíncrono/não-bloqueante. Veja [Feedback Humano Assíncrono](#feedback-humano-assíncrono-não-bloqueante) |
| `learn` | `bool` | Não | Habilitar aprendizado HITL: destila lições do feedback e pré-revisa saídas futuras. Padrão `False`. Veja [Aprendendo com Feedback](#aprendendo-com-feedback) |
-| `learn_limit` | `int` | Não | Máximo de lições passadas para recuperar na pré-revisão. Padrão `5` |
+| `learn_source` | `str` | Não | `source` da memória em que as lições são armazenadas e de onde são recuperadas. Padrão `"hitl"` |
+| `learn_strict` | `bool` | Não | Lança um erro quando a pré-revisão ou a destilação de lições falha, em vez de usar o fallback. Padrão `False` |
### Uso Básico (Sem Roteamento)
@@ -635,11 +636,15 @@ O parâmetro `learn=True` habilita um ciclo de feedback entre revisores humanos
### Como Funciona
-1. **Após o feedback**: O LLM extrai lições generalizáveis da saída + feedback e as armazena na memória com `source="hitl"`. Se o feedback for apenas aprovação (ex: "parece bom"), nada é armazenado.
-2. **Antes da próxima revisão**: Lições HITL passadas são recuperadas da memória e aplicadas pelo LLM para melhorar a saída antes que o humano a veja.
+1. **Após o feedback**: O LLM extrai lições generalizáveis da saída + feedback e as armazena na memória sob `learn_source` (padrão `"hitl"`). Isso acontece tanto quando a resposta chega durante a execução quanto quando ela retoma depois um flow pausado por meio de um [provider assíncrono](#feedback-humano-assíncrono-não-bloqueante). Se o feedback for apenas aprovação (ex: "parece bom"), nada é armazenado.
+2. **Antes da próxima revisão**: Lições HITL passadas são recuperadas da memória e aplicadas pelo LLM para melhorar a saída antes que o humano a veja. Uma saída de texto é reescrita como texto, e um modelo Pydantic é reescrito como o mesmo modelo, a menos que contenha algo que o LLM não consegue ver: um secret, um campo excluído ou um atributo privado alterado em relação ao padrão. Qualquer outra saída, como um `dict` ou uma `list`, é mostrada sem alterações.
Com o tempo, o humano vê saídas pré-revisadas progressivamente melhores porque cada correção informa revisões futuras.
+<Note>
+O humano aprova a versão revisada, que pode ser diferente do que o método retornou. Nas etapas seguintes, leia-a de `self.last_human_feedback.output` (ou do argumento do listener). O estado que o método definiu antes da revisão ainda guarda o original. Se o flow pausou para feedback assíncrono, a saída é restaurada a partir de JSON na retomada, então um modelo Pydantic chega como um `dict`.
+</Note>
+
### Exemplo
```python Code
@@ -672,14 +677,15 @@ class ArticleReviewFlow(Flow):
| Parâmetro | Padrão | Descrição |
|-----------|--------|-----------|
| `learn` | `False` | Habilitar aprendizado HITL |
-| `learn_limit` | `5` | Máximo de lições passadas para recuperar na pré-revisão |
+| `learn_source` | `"hitl"` | Source da memória em que as lições são armazenadas e de onde são recuperadas |
+| `learn_strict` | `False` | Lança um erro em falhas de pré-revisão ou destilação, em vez de usar o fallback |
### Decisões de Design Principais
- **Mesmo LLM para tudo**: O parâmetro `llm` no decorador é compartilhado pelo mapeamento de outcome, destilação de lições e pré-revisão. Não é necessário configurar múltiplos modelos.
- **Saída estruturada**: Tanto a destilação quanto a pré-revisão usam function calling com modelos Pydantic quando o LLM suporta, com fallback para parsing de texto caso contrário.
- **Armazenamento não-bloqueante**: Lições são armazenadas via `remember_many()` que executa em uma thread em segundo plano -- o flow continua imediatamente.
-- **Degradação graciosa**: Se o LLM falhar durante a destilação, nada é armazenado. Se falhar durante a pré-revisão, a saída bruta é mostrada. Nenhuma falha bloqueia o flow.
+- **Degradação graciosa**: Se o LLM falhar durante a destilação, nada é armazenado. Se falhar durante a pré-revisão, a saída bruta é mostrada. Nenhuma falha bloqueia o flow, a menos que `learn_strict=True`, que lança um erro.
- **Sem escopo/categorias necessários**: Ao armazenar lições, apenas `source` é passado. O pipeline de codificação infere escopo, categorias e importância automaticamente.
<Note>
diff --git a/lib/crewai/src/crewai/flow/flow_definition.py b/lib/crewai/src/crewai/flow/flow_definition.py
index cad4da79b..0a65fd7ef 100644
--- a/lib/crewai/src/crewai/flow/flow_definition.py
+++ b/lib/crewai/src/crewai/flow/flow_definition.py
@@ -320,7 +320,7 @@ class FlowHumanFeedbackDefinition(BaseModel):
)
learn: bool = Field(
default=False,
- description="Whether feedback should be recorded for later learning workflows.",
+ description="Whether to distill lessons from feedback into memory and apply past lessons to the output before it is reviewed.",
examples=[True],
)
learn_source: str = Field(
@@ -330,7 +330,7 @@ class FlowHumanFeedbackDefinition(BaseModel):
)
learn_strict: bool = Field(
default=False,
- description="Whether learning should enforce strict validation of feedback data.",
+ description="Whether a failed pre-review or lesson distillation raises instead of falling back to the unreviewed output and storing nothing. Lessons are written to memory in the background, so a storage failure after that is not raised.",
examples=[False],
)
diff --git a/lib/crewai/src/crewai/flow/human_feedback.py b/lib/crewai/src/crewai/flow/human_feedback.py
index dd107604c..52767bbcf 100644
--- a/lib/crewai/src/crewai/flow/human_feedback.py
+++ b/lib/crewai/src/crewai/flow/human_feedback.py
@@ -64,7 +64,7 @@ from datetime import datetime
import logging
from typing import TYPE_CHECKING, Any, TypeVar
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, Secret, SecretBytes, SecretStr
if TYPE_CHECKING:
@@ -76,6 +76,7 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
F = TypeVar("F", bound=Callable[..., Any])
+_StructuredT = TypeVar("_StructuredT", bound=BaseModel)
__all__ = ["HumanFeedbackCollapseError", "HumanFeedbackResult", "human_feedback"]
@@ -154,8 +155,12 @@ class HumanFeedbackResult:
collapsed outcome for routing purposes.
Attributes:
- output: The original return value from the decorated method that was
- shown to the human for review.
+ output: The output shown to the human for review: the decorated
+ method's return value or, with ``learn=True``, that value after
+ past lessons were applied (same type). Downstream steps should read
+ the approved output from here, not from state the method set. After
+ a flow pauses for async feedback and resumes, the pending review is
+ restored from JSON, so a Pydantic output arrives as a ``dict``.
feedback: The raw text feedback provided by the human. Empty string
if no feedback was provided.
outcome: The collapsed outcome string when emit is specified.
@@ -276,6 +281,39 @@ def _resolve_llm_instance(llm: Any) -> Any:
return llm
+def _as_structured(response: Any, model_type: type[_StructuredT]) -> _StructuredT:
+ if isinstance(response, model_type):
+ return response
+ if isinstance(response, str):
+ # LiteLLM-routed models return structured output as a JSON string.
+ return model_type.model_validate_json(response)
+ return model_type.model_validate(response)
+
+
+def _has_hidden_values(value: Any) -> bool:
+ if isinstance(value, (Secret, SecretStr, SecretBytes)):
+ return True
+ if isinstance(value, BaseModel):
+ private = value.__pydantic_private__ or {}
+ # A rebuilt model starts private attributes from their defaults, so any
+ # private value that differs from its default would be lost.
+ if any(
+ private[name] != attr.get_default()
+ for name, attr in type(value).__private_attributes__.items()
+ if name in private
+ ):
+ return True
+ return any(
+ info.exclude or _has_hidden_values(getattr(value, name))
+ for name, info in type(value).model_fields.items()
+ )
+ if isinstance(value, dict):
+ return any(_has_hidden_values(item) for item in value.values())
+ if isinstance(value, (list, tuple, set, frozenset)):
+ return any(_has_hidden_values(item) for item in value)
+ return False
+
+
def _pre_review_with_lessons(
flow_instance: Flow[Any],
method_name: str,
@@ -288,10 +326,23 @@ def _pre_review_with_lessons(
from crewai.hooks.dispatch import HookAborted
try:
+ # Only text and Pydantic outputs can be rewritten without changing their
+ # type, and a model only when the LLM sees all of it: secrets are masked,
+ # excluded fields and private attributes dropped, so rebuilding from the
+ # review would lose them. Anything else reaches the reviewer unchanged.
+ if isinstance(method_output, BaseModel):
+ if _has_hidden_values(method_output):
+ return method_output
+ rendered = method_output.model_dump_json()
+ elif isinstance(method_output, str):
+ rendered = method_output
+ else:
+ return method_output
+
mem = flow_instance.memory
if mem is None:
return method_output
- query = f"human feedback lessons for {method_name}: {method_output!s}"
+ query = f"human feedback lessons for {method_name}: {rendered}"
matches = mem.recall(query, source=learn_source)
if not matches:
return method_output
@@ -299,7 +350,7 @@ def _pre_review_with_lessons(
lessons = "\n".join(f"- {m.record.content}" for m in matches)
llm_inst = _resolve_llm_instance(llm)
prompt = _get_hitl_prompt("hitl_pre_review_user").format(
- output=str(method_output),
+ output=rendered,
lessons=lessons,
)
messages = [
@@ -309,11 +360,14 @@ def _pre_review_with_lessons(
},
{"role": "user", "content": prompt},
]
+ if isinstance(method_output, BaseModel):
+ output_type = type(method_output)
+ return _as_structured(
+ llm_inst.call(messages, response_model=output_type), output_type
+ )
if getattr(llm_inst, "supports_function_calling", lambda: False)():
response = llm_inst.call(messages, response_model=PreReviewResult)
- if isinstance(response, PreReviewResult):
- return response.improved_output
- return PreReviewResult.model_validate(response).improved_output
+ return _as_structured(response, PreReviewResult).improved_output
reviewed = llm_inst.call(messages)
return reviewed if isinstance(reviewed, str) else str(reviewed)
except HookAborted:
@@ -367,10 +421,7 @@ def _distill_and_store_lessons(
lessons: list[str] = []
if getattr(llm_inst, "supports_function_calling", lambda: False)():
response = llm_inst.call(messages, response_model=DistilledLessons)
- if isinstance(response, DistilledLessons):
- lessons = response.lessons
- else:
- lessons = DistilledLessons.model_validate(response).lessons
+ lessons = _as_structured(response, DistilledLessons).lessons
else:
response = llm_inst.call(messages)
if isinstance(response, str):
diff --git a/lib/crewai/src/crewai/flow/runtime/__init__.py b/lib/crewai/src/crewai/flow/runtime/__init__.py
index 9113cbf18..8b7a3cf10 100644
--- a/lib/crewai/src/crewai/flow/runtime/__init__.py
+++ b/lib/crewai/src/crewai/flow/runtime/__init__.py
@@ -1498,6 +1498,7 @@ class Flow(BaseModel, Generic[T], metaclass=FlowMeta):
# The serialized context carries the full LLM config (a dict, or a
# legacy model string) — the single source for cross- and
# same-process resume.
+ previous_feedback = self.last_human_feedback
result = await self._finalize_human_feedback(
method_name=context.method_name,
method_output=context.method_output,
@@ -1514,6 +1515,27 @@ class Flow(BaseModel, Generic[T], metaclass=FlowMeta):
else result
)
+ feedback_definition = self._definition.methods[
+ context.method_name
+ ].human_feedback
+ if feedback_definition is not None:
+ # A reply that resumes a paused flow (e.g. email review on AMP)
+ # teaches the same lessons as one that arrives during the run.
+ try:
+ await self._store_feedback_lessons(
+ feedback_definition,
+ context.method_name,
+ context.method_output,
+ feedback,
+ context.llm,
+ )
+ except Exception:
+ # learn_strict re-raised and the review stays pending: undo the
+ # recorded reply so a retry on this instance records it once.
+ self.human_feedback_history.pop()
+ self.last_human_feedback = previous_feedback
+ raise
+
self._completed_methods.add(FlowMethodName(context.method_name))
await asyncio.to_thread(
@@ -3671,17 +3693,9 @@ class Flow(BaseModel, Generic[T], metaclass=FlowMeta):
metadata=metadata or {},
)
- if learn and raw_feedback.strip():
- await asyncio.to_thread(
- _distill_and_store_lessons,
- self,
- method_name,
- method_output,
- raw_feedback,
- llm=llm_instance,
- learn_source=feedback_definition.learn_source,
- learn_strict=feedback_definition.learn_strict,
- )
+ await self._store_feedback_lessons(
+ feedback_definition, method_name, method_output, raw_feedback, llm_instance
+ )
if emit:
# Stash the feedback result: the collapsed outcome routes listeners,
@@ -3691,6 +3705,31 @@ class Flow(BaseModel, Generic[T], metaclass=FlowMeta):
return result.outcome
return result
+ async def _store_feedback_lessons(
+ self,
+ feedback_definition: FlowHumanFeedbackDefinition,
+ method_name: str,
+ method_output: Any,
+ raw_feedback: str,
+ llm: Any,
+ ) -> None:
+ if not (
+ feedback_definition.learn
+ and self.memory is not None
+ and raw_feedback.strip()
+ ):
+ return
+ await asyncio.to_thread(
+ _distill_and_store_lessons,
+ self,
+ method_name,
+ method_output,
+ raw_feedback,
+ llm=llm,
+ learn_source=feedback_definition.learn_source,
+ learn_strict=feedback_definition.learn_strict,
+ )
+
async def _finalize_human_feedback(
self,
*,
diff --git a/lib/crewai/tests/test_async_human_feedback.py b/lib/crewai/tests/test_async_human_feedback.py
index 8fba9d113..48732a536 100644
--- a/lib/crewai/tests/test_async_human_feedback.py
+++ b/lib/crewai/tests/test_async_human_feedback.py
@@ -1464,3 +1464,359 @@ class TestResumeLLMFromSerializedContext:
from crewai.llms.base_llm import BaseLLM as BaseLLMClass
assert isinstance(captured_llm[0], BaseLLMClass)
assert captured_llm[0].model == "gpt-4o-mini"
+
+
+# learn=True on resumed feedback
+
+
+def _pause(tmpdir: str, flow_cls: type[Flow], flow_id: str, **context: Any) -> Flow:
+ """Persist a pending review for ``flow_cls.draft`` and restore it, as AMP does."""
+ persistence = SQLiteFlowPersistence(os.path.join(tmpdir, "flows.db"))
+ persistence.save_pending_feedback(
+ flow_uuid=flow_id,
+ context=PendingFeedbackContext(
+ flow_id=flow_id,
+ flow_class=f"test.{flow_cls.__name__}",
+ method_name="draft",
+ method_output="reviewed draft",
+ message="Review:",
+ llm="gpt-4o-mini",
+ **context,
+ ),
+ state_data={"id": flow_id},
+ )
+ flow = flow_cls.from_pending(flow_id, persistence)
+ flow.memory = MagicMock()
+ flow.memory.recall.return_value = []
+ return flow
+
+
+def _distill_llm(*responses: Any) -> MagicMock:
+ llm = MagicMock()
+ llm.supports_function_calling.return_value = True
+ llm.call.side_effect = list(responses)
+ return llm
+
+
+class TestLearnOnResume:
+ """A reply that resumes a paused flow teaches lessons like an in-run reply."""
+
+ def test_resume_stores_lessons_distilled_from_the_reply(self) -> None:
+ from crewai.flow.human_feedback import DistilledLessons
+
+ class LearnFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", llm="gpt-4o-mini", learn=True)
+ def draft(self):
+ return "draft"
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, LearnFlow, "learn-resume-1")
+ llm = _distill_llm(DistilledLessons(lessons=["Cite a source for every claim"]))
+ with patch("crewai.llm.LLM", return_value=llm):
+ flow.resume("Always cite your sources")
+
+ flow.memory.remember_many.assert_called_once_with(
+ ["Cite a source for every claim"], source="hitl"
+ )
+ distill_prompt = llm.call.call_args.args[0][1]["content"]
+ assert "reviewed draft" in distill_prompt
+ assert "Always cite your sources" in distill_prompt
+ assert flow.last_human_feedback.feedback == "Always cite your sources"
+
+ def test_resume_async_stores_lessons(self) -> None:
+ import asyncio
+
+ from crewai.flow.human_feedback import DistilledLessons
+
+ class LearnFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", llm="gpt-4o-mini", learn=True)
+ def draft(self):
+ return "draft"
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, LearnFlow, "learn-resume-async")
+ llm = _distill_llm(DistilledLessons(lessons=["Keep intros under 50 words"]))
+ with patch("crewai.llm.LLM", return_value=llm):
+ asyncio.run(flow.resume_async("The intro is too long"))
+
+ flow.memory.remember_many.assert_called_once_with(
+ ["Keep intros under 50 words"], source="hitl"
+ )
+
+ def test_resume_routes_on_the_reply_and_stores_lessons(self) -> None:
+ from crewai.flow.human_feedback import DistilledLessons
+
+ class RoutedFlow(Flow):
+ published: str = ""
+
+ @start()
+ @human_feedback(
+ message="Review:",
+ emit=["approved", "needs_revision"],
+ llm="gpt-4o-mini",
+ learn=True,
+ )
+ def draft(self):
+ return "draft"
+
+ @listen("approved")
+ def publish(self):
+ self.published = self.last_human_feedback.output
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(
+ tmpdir, RoutedFlow, "learn-resume-emit", emit=["approved", "needs_revision"]
+ )
+ llm = _distill_llm(DistilledLessons(lessons=["Prefer active voice"]))
+ with (
+ patch.object(flow, "_collapse_to_outcome", return_value="approved"),
+ patch("crewai.llm.LLM", return_value=llm),
+ ):
+ flow.resume("Approved, but prefer active voice next time")
+
+ assert flow.published == "reviewed draft"
+ flow.memory.remember_many.assert_called_once_with(
+ ["Prefer active voice"], source="hitl"
+ )
+
+ def test_resume_stores_lessons_under_the_configured_source(self) -> None:
+ from crewai.flow.human_feedback import DistilledLessons
+
+ class SourcedFlow(Flow):
+ @start()
+ @human_feedback(
+ message="Review:",
+ llm="gpt-4o-mini",
+ learn=True,
+ learn_source="editorial",
+ )
+ def draft(self):
+ return "draft"
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, SourcedFlow, "learn-resume-source")
+ llm = _distill_llm(DistilledLessons(lessons=["Use the house style guide"]))
+ with patch("crewai.llm.LLM", return_value=llm):
+ flow.resume("Follow the house style")
+
+ assert flow.memory.remember_many.call_args.kwargs == {"source": "editorial"}
+
+ def test_resume_with_empty_reply_stores_nothing(self) -> None:
+ class LearnFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", llm="gpt-4o-mini", learn=True)
+ def draft(self):
+ return "draft"
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, LearnFlow, "learn-resume-empty")
+ llm = _distill_llm()
+ with patch("crewai.llm.LLM", return_value=llm):
+ flow.resume("")
+
+ llm.call.assert_not_called()
+ flow.memory.remember_many.assert_not_called()
+
+ def test_resume_without_learn_leaves_memory_untouched(self) -> None:
+ class PlainFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", llm="gpt-4o-mini")
+ def draft(self):
+ return "draft"
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, PlainFlow, "learn-resume-off")
+ llm = _distill_llm()
+ with patch("crewai.llm.LLM", return_value=llm):
+ flow.resume("Always cite your sources")
+
+ llm.call.assert_not_called()
+ flow.memory.remember_many.assert_not_called()
+
+ def test_failed_distillation_still_completes_the_resume(self) -> None:
+ class LearnFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", llm="gpt-4o-mini", learn=True)
+ def draft(self):
+ return "draft"
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, LearnFlow, "learn-resume-lenient")
+ llm = _distill_llm(RuntimeError("distillation model unavailable"))
+ with patch("crewai.llm.LLM", return_value=llm):
+ flow.resume("Always cite your sources")
+
+ assert flow.last_human_feedback.feedback == "Always cite your sources"
+ flow.memory.remember_many.assert_not_called()
+
+ def test_strict_distillation_failure_can_be_retried_cleanly(self) -> None:
+ class StrictFlow(Flow):
+ @start()
+ @human_feedback(
+ message="Review:", llm="gpt-4o-mini", learn=True, learn_strict=True
+ )
+ def draft(self):
+ return "draft"
+
+ from crewai.flow.human_feedback import DistilledLessons
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, StrictFlow, "learn-resume-strict")
+ llm = _distill_llm(
+ RuntimeError("distillation model unavailable"),
+ DistilledLessons(lessons=["Cite a source for every claim"]),
+ )
+ with patch("crewai.llm.LLM", return_value=llm):
+ with pytest.raises(
+ RuntimeError, match="distillation model unavailable"
+ ):
+ flow.resume("Always cite your sources")
+
+ assert (
+ flow.persistence.load_pending_feedback("learn-resume-strict")
+ is not None
+ )
+ assert flow.human_feedback_history == []
+ assert flow.last_human_feedback is None
+ flow.memory.remember_many.assert_not_called()
+
+ flow.resume("Always cite your sources")
+
+ assert [r.feedback for r in flow.human_feedback_history] == [
+ "Always cite your sources"
+ ]
+ flow.memory.remember_many.assert_called_once_with(
+ ["Cite a source for every claim"], source="hitl"
+ )
+
+ def test_retry_after_a_failed_completion_offers_the_lessons_again(self) -> None:
+ # Lessons are stored before the review is durably completed, which strict
+ # mode needs, so a retry after a later persistence failure stores them again
+ # and memory consolidation absorbs the repeat.
+ from crewai.flow.human_feedback import DistilledLessons
+
+ class LearnFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", llm="gpt-4o-mini", learn=True)
+ def draft(self):
+ return "draft"
+
+ lesson = ["Cite a source for every claim"]
+ with tempfile.TemporaryDirectory() as tmpdir:
+ flow = _pause(tmpdir, LearnFlow, "learn-resume-retry")
+ persistence = flow.persistence
+ with (
+ patch(
+ "crewai.llm.LLM",
+ return_value=_distill_llm(DistilledLessons(lessons=lesson)),
+ ),
+ patch.object(
+ SQLiteFlowPersistence,
+ "clear_pending_feedback",
+ side_effect=RuntimeError("database is locked"),
+ ),
+ pytest.raises(RuntimeError, match="database is locked"),
+ ):
+ flow.resume("Always cite your sources")
+
+ flow.memory.remember_many.assert_called_once_with(lesson, source="hitl")
+ assert persistence.load_pending_feedback("learn-resume-retry") is not None
+
+ retried = LearnFlow.from_pending("learn-resume-retry", persistence)
+ retried.memory = MagicMock()
+ with patch(
+ "crewai.llm.LLM",
+ return_value=_distill_llm(DistilledLessons(lessons=lesson)),
+ ):
+ retried.resume("Always cite your sources")
+
+ retried.memory.remember_many.assert_called_once_with(lesson, source="hitl")
+ assert persistence.load_pending_feedback("learn-resume-retry") is None
+
+ @patch("crewai.flow.runtime.crewai_event_bus.emit")
+ def test_resumed_pydantic_output_arrives_as_a_dict(self, mock_emit: MagicMock) -> None:
+ # The pending review is persisted as JSON, so the docs and the
+ # HumanFeedbackResult.output docstring say a model comes back as a dict.
+ class Draft(BaseModel):
+ title: str
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ persistence = SQLiteFlowPersistence(os.path.join(tmpdir, "flows.db"))
+
+ class PauseProvider:
+ def request_feedback(
+ self, context: PendingFeedbackContext, flow: Flow
+ ) -> str:
+ persistence.save_pending_feedback(
+ flow_uuid=context.flow_id,
+ context=context,
+ state_data=flow.state
+ if isinstance(flow.state, dict)
+ else flow.state.model_dump(),
+ )
+ raise HumanFeedbackPending(context=context)
+
+ class DraftFlow(Flow):
+ @start()
+ @human_feedback(message="Review:", provider=PauseProvider())
+ def draft(self):
+ return Draft(title="Launch day")
+
+ pending = DraftFlow(persistence=persistence).kickoff()
+ assert isinstance(pending.context.method_output, Draft)
+
+ resumed = DraftFlow.from_pending(pending.context.flow_id, persistence)
+ resumed.resume("looks good")
+
+ assert resumed.last_human_feedback.output == {"title": "Launch day"}
+
+ @patch("crewai.flow.runtime.crewai_event_bus.emit")
+ def test_reply_after_a_paused_kickoff_is_learned(self, mock_emit: MagicMock) -> None:
+ from crewai.flow.human_feedback import DistilledLessons
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ persistence = SQLiteFlowPersistence(os.path.join(tmpdir, "flows.db"))
+
+ class EmailProvider:
+ def request_feedback(
+ self, context: PendingFeedbackContext, flow: Flow
+ ) -> str:
+ persistence.save_pending_feedback(
+ flow_uuid=context.flow_id,
+ context=context,
+ state_data=flow.state
+ if isinstance(flow.state, dict)
+ else flow.state.model_dump(),
+ )
+ raise HumanFeedbackPending(context=context)
+
+ class ReviewFlow(Flow):
+ @start()
+ @human_feedback(
+ message="Review:",
+ llm="gpt-4o-mini",
+ learn=True,
+ provider=EmailProvider(),
+ )
+ def draft(self):
+ return "first draft"
+
+ paused = ReviewFlow(persistence=persistence)
+ paused.memory = MagicMock()
+ paused.memory.recall.return_value = []
+ pending = paused.kickoff()
+ assert isinstance(pending, HumanFeedbackPending)
+ paused.memory.remember_many.assert_not_called()
+
+ resumed = ReviewFlow.from_pending(pending.context.flow_id, persistence)
+ resumed.memory = MagicMock()
+ llm = _distill_llm(DistilledLessons(lessons=["Lead with the customer outcome"]))
+ with patch("crewai.llm.LLM", return_value=llm):
+ resumed.resume("Lead with what the customer gets")
+
+ resumed.memory.remember_many.assert_called_once_with(
+ ["Lead with the customer outcome"], source="hitl"
+ )
+ assert "first draft" in llm.call.call_args.args[0][1]["content"]
diff --git a/lib/crewai/tests/test_human_feedback_decorator.py b/lib/crewai/tests/test_human_feedback_decorator.py
index 17557c7e2..6f872d9fe 100644
--- a/lib/crewai/tests/test_human_feedback_decorator.py
+++ b/lib/crewai/tests/test_human_feedback_decorator.py
@@ -12,6 +12,14 @@ from typing import Any
from unittest.mock import MagicMock, patch
import pytest
+from pydantic import (
+ BaseModel,
+ ConfigDict,
+ Field,
+ PrivateAttr,
+ SecretStr,
+ ValidationError,
+)
from crewai.constants import DEFAULT_LLM_MODEL
from crewai.flow import Flow, human_feedback, listen, persist, start
@@ -880,6 +888,274 @@ class TestHumanFeedbackLearn:
assert config.learn_strict is True
+class _Draft(BaseModel):
+ title: str
+ body: str
+
+
+def _lesson(content: str) -> Any:
+ from crewai.memory.types import MemoryMatch, MemoryRecord
+
+ return MemoryMatch(
+ record=MemoryRecord(content=content, embedding=[]),
+ score=0.9,
+ match_reasons=["semantic"],
+ )
+
+
+def _review_llm(*responses: Any, function_calling: bool = True) -> MagicMock:
+ llm = MagicMock()
+ llm.supports_function_calling.return_value = function_calling
+ llm.call.side_effect = list(responses)
+ return llm
+
+
+def _run_learning_flow(
+ output: Any, llm: MagicMock, feedback: str = "", *, learn_strict: bool = False
+) -> tuple[Flow, dict[str, Any]]:
+ """Kick off a one-step learn=True flow whose method returns ``output``."""
+
+ class LearningFlow(Flow):
+ @start()
+ @human_feedback(
+ message="Review:", llm="gpt-4o-mini", learn=True, learn_strict=learn_strict
+ )
+ def draft(self):
+ return output
+
+ flow = LearningFlow()
+ flow.memory = MagicMock()
+ flow.memory.recall.return_value = [_lesson("Titles use sentence case")]
+ seen: dict[str, Any] = {}
+
+ def show(message, output, metadata=None, emit=None, method_name=""):
+ seen["shown"] = output
+ return feedback
+
+ with (
+ patch.object(flow, "_request_human_feedback", side_effect=show),
+ patch("crewai.llm.LLM", return_value=llm),
+ ):
+ flow.kickoff()
+ return flow, seen
+
+
+class TestPreReviewKeepsOutputType:
+ """learn=True pre-review must not change the type of what the human approves."""
+
+ def test_pydantic_output_is_rewritten_into_the_same_model(self):
+ llm = _review_llm(_Draft(title="Launch day", body="We shipped."))
+
+ flow, seen = _run_learning_flow(_Draft(title="Launch Day", body="We shipped."), llm)
+
+ assert seen["shown"] == _Draft(title="Launch day", body="We shipped.")
+ assert flow.last_human_feedback.output == seen["shown"]
+ assert llm.call.call_args.kwargs["response_model"] is _Draft
+ assert '"title":"Launch Day"' in llm.call.call_args.args[0][1]["content"]
+
+ def test_pydantic_output_parses_a_json_string_response(self):
+ # LiteLLM-routed models return structured output as a JSON string.
+ llm = _review_llm('{"title": "Launch day", "body": "We shipped."}')
+
+ _, seen = _run_learning_flow(_Draft(title="Launch Day", body="We shipped."), llm)
+
+ assert seen["shown"] == _Draft(title="Launch day", body="We shipped.")
+
+ def test_dict_output_reaches_the_reviewer_unchanged(self):
+ output = {"title": "Launch Day", "body": "We shipped."}
+ llm = _review_llm()
+
+ flow, seen = _run_learning_flow(output, llm)
+
+ assert seen["shown"] is output
+ assert flow.last_human_feedback.output is output
+ flow.memory.recall.assert_not_called()
+ llm.call.assert_not_called()
+
+ def test_text_output_parses_a_json_string_response(self):
+ llm = _review_llm('{"improved_output": "Launch day"}')
+
+ _, seen = _run_learning_flow("Launch Day", llm)
+
+ assert seen["shown"] == "Launch day"
+
+ def test_invalid_structured_result_falls_back_to_the_original(self):
+ original = _Draft(title="Launch Day", body="We shipped.")
+ llm = _review_llm('{"title": "Launch day"}')
+
+ _, seen = _run_learning_flow(original, llm)
+
+ assert seen["shown"] is original
+
+ def test_invalid_structured_result_raises_when_strict(self):
+ llm = _review_llm('{"title": "Launch day"}')
+
+ with pytest.raises(ValidationError, match="body"):
+ _run_learning_flow(
+ _Draft(title="Launch Day", body="We shipped."), llm, learn_strict=True
+ )
+
+
+class _Handle:
+ """An object pydantic cannot serialize to JSON."""
+
+
+class _WithHandle(BaseModel):
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ title: str
+ handle: _Handle
+
+
+class _Account(BaseModel):
+ name: str
+ token: SecretStr
+
+
+class _Tagged(BaseModel):
+ title: str
+ row_id: str = Field(default="", exclude=True)
+
+
+class _Report(BaseModel):
+ title: str
+ owner: _Account | None = None
+ accounts: list[_Account] = []
+ by_name: dict[str, _Account] = {}
+ tagged: list[_Tagged] = []
+
+
+class _Aliased(BaseModel):
+ model_config = ConfigDict(serialize_by_alias=True, validate_by_name=True)
+
+ title: str = Field(alias="headline")
+ body: str
+
+
+_ACCOUNT = _Account(name="ana", token=SecretStr("tok-123"))
+
+
+class _Session(BaseModel):
+ title: str
+ _token: str = PrivateAttr(default="")
+
+
+class _Workspace(BaseModel):
+ title: str
+ owner: _Session | None = None
+ sessions: list[_Session] = []
+ by_name: dict[str, _Session] = {}
+
+
+def _session(token: str = "tok-live") -> _Session:
+ session = _Session(title="Launch Day")
+ session._token = token
+ return session
+
+
+class TestPreReviewKeepsWhatItCannotSee:
+ def test_unserializable_output_falls_back_to_the_original(self):
+ original = _WithHandle(title="Launch Day", handle=_Handle())
+ llm = _review_llm()
+
+ _, seen = _run_learning_flow(original, llm)
+
+ assert seen["shown"] is original
+ llm.call.assert_not_called()
+
+ def test_unserializable_output_raises_when_strict(self):
+ from pydantic_core import PydanticSerializationError
+
+ with pytest.raises(PydanticSerializationError, match="Unable to serialize"):
+ _run_learning_flow(
+ _WithHandle(title="Launch Day", handle=_Handle()),
+ _review_llm(),
+ learn_strict=True,
+ )
+
+ @pytest.mark.parametrize(
+ "original",
+ [
+ _Account(name="ana", token=SecretStr("tok-123")),
+ _Tagged(title="Launch Day", row_id="r1"),
+ _Report(title="Q3", owner=_ACCOUNT),
+ _Report(title="Q3", accounts=[_ACCOUNT]),
+ _Report(title="Q3", by_name={"ana": _ACCOUNT}),
+ _Report(title="Q3", tagged=[_Tagged(title="t", row_id="r1")]),
+ _session(),
+ _Workspace(title="Q3", owner=_session()),
+ _Workspace(title="Q3", sessions=[_session()]),
+ _Workspace(title="Q3", by_name={"ana": _session()}),
+ ],
+ ids=[
+ "secret",
+ "excluded",
+ "nested",
+ "list",
+ "dict",
+ "list-excluded",
+ "private",
+ "nested-private",
+ "list-private",
+ "dict-private",
+ ],
+ )
+ def test_model_with_hidden_values_reaches_the_reviewer_unchanged(self, original):
+ # The LLM would see secrets masked and excluded fields not at all, so a
+ # model rebuilt from its review would lose them.
+ llm = _review_llm()
+
+ flow, seen = _run_learning_flow(original, llm)
+
+ assert seen["shown"] is original
+ assert flow.last_human_feedback.output is original
+ flow.memory.recall.assert_not_called()
+ llm.call.assert_not_called()
+
+ def test_model_without_hidden_values_is_still_reviewed(self):
+ llm = _review_llm('{"title": "Q3 report", "owner": null}')
+
+ _, seen = _run_learning_flow(_Report(title="Q3"), llm)
+
+ assert seen["shown"] == _Report(title="Q3 report")
+
+ def test_private_attribute_left_at_its_default_is_still_reviewed(self):
+ # A rebuilt model resets private attributes to their defaults, which only
+ # loses data when the value had changed.
+ llm = _review_llm('{"title": "Launch day"}')
+
+ _, seen = _run_learning_flow(_Session(title="Launch Day"), llm)
+
+ assert seen["shown"] == _Session(title="Launch day")
+ llm.call.assert_called_once()
+
+ def test_edits_to_aliased_fields_are_kept(self):
+ llm = _review_llm('{"headline": "Launch day", "body": "We shipped."}')
+
+ _, seen = _run_learning_flow(
+ _Aliased(title="Launch Day", body="We shipped."), llm
+ )
+
+ assert seen["shown"].title == "Launch day"
+ assert '"headline":"Launch Day"' in llm.call.call_args.args[0][1]["content"]
+
+
+class TestDistillationParsesJsonStrings:
+ def test_lessons_returned_as_a_json_string_are_stored(self):
+ from crewai.flow.human_feedback import PreReviewResult
+
+ llm = _review_llm(
+ PreReviewResult(improved_output="Launch day"),
+ '{"lessons": ["Titles use sentence case"]}',
+ )
+
+ flow, _ = _run_learning_flow("Launch Day", llm, feedback="Sentence case, please")
+
+ flow.memory.remember_many.assert_called_once_with(
+ ["Titles use sentence case"], source="hitl"
+ )
+
+
class TestHumanFeedbackFinalOutputPreservation:
"""Tests for preserving method return value as flow's final output when @human_feedback with emit is terminal.