Skip to content

armgit / dify

Public Imported from GitHub
0

Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.

test(e2e): align Agent v2 publish scenarios with guidance (#42787)

yyh1aafa481f32c
diff --git a/e2e/features/agent-v2/access-point.feature b/e2e/features/agent-v2/access-point.feature
index 2870e5270f9..cd784aa0589 100644
--- a/e2e/features/agent-v2/access-point.feature
+++ b/e2e/features/agent-v2/access-point.feature
@@ -151,7 +151,7 @@ Feature: Agent v2 Access Point
And a runnable Agent v2 test agent has been created via API
When I open the Agent v2 configure page
And I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
+ Then the Agent v2 first publication should succeed
Given an Agent v2 Backend service API key has been created via API
When I send the Agent v2 Backend service API minimal request
Then the Agent v2 Backend service API request should succeed with the normal E2E marker
diff --git a/e2e/features/agent-v2/knowledge.feature b/e2e/features/agent-v2/knowledge.feature
index bb235815945..4d7162cb8b3 100644
--- a/e2e/features/agent-v2/knowledge.feature
+++ b/e2e/features/agent-v2/knowledge.feature
@@ -36,7 +36,7 @@ Feature: Agent v2 Knowledge Retrieval
Then the Agent v2 Agent decide Knowledge Retrieval should be saved in the Agent v2 draft
And the Agent v2 configuration should be saved automatically
When I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
+ Then the Agent v2 first publication should succeed
Given an Agent v2 Backend service API key has been created via API
When I send the Agent v2 Backend service API knowledge request
Then the Agent v2 Backend service API response should include the knowledge E2E marker
@@ -53,7 +53,7 @@ Feature: Agent v2 Knowledge Retrieval
Then the Agent v2 Custom query Knowledge Retrieval should be saved in the Agent v2 draft
And the Agent v2 configuration should be saved automatically
When I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
+ Then the Agent v2 first publication should succeed
Given an Agent v2 Backend service API key has been created via API
When I send the Agent v2 Backend service API knowledge request
Then the Agent v2 Backend service API response should include the knowledge E2E marker
diff --git a/e2e/features/agent-v2/publish.feature b/e2e/features/agent-v2/publish.feature
index 30a8eabb799..975d6cce08d 100644
--- a/e2e/features/agent-v2/publish.feature
+++ b/e2e/features/agent-v2/publish.feature
@@ -1,110 +1,95 @@
@agent-v2 @authenticated @publish
Feature: Agent v2 publish
- @core
- Scenario: Publish is blocked until a model is configured
+ Background:
Given I am signed in as the default E2E admin
- And an Agent v2 test agent has been created via API
- And the Agent v2 composer draft uses the normal E2E prompt
- When I open the Agent v2 configure page
- And I try to publish the Agent v2 draft without a model
- Then Agent v2 publish should be blocked until a model is configured
- And the Agent v2 draft should remain unpublished
@core @prepared @stable-model
Scenario: Publish a configured Agent v2 draft
- Given I am signed in as the default E2E admin
- And the Agent Builder stable chat model is available
+ Given the Agent Builder stable chat model is available
And a runnable Agent v2 test agent has been created via API
When I open the Agent v2 configure page
And I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
- When I switch to the Agent v2 Access Point section
- Then Agent v2 Web app access should be in service
- And Agent v2 Backend service API access should be in service
+ Then the Agent v2 first publication should succeed
@core @prepared @stable-model
- Scenario: Publish action follows unpublished changes
- Given I am signed in as the default E2E admin
- And the Agent Builder stable chat model is available
+ Scenario: Publish guidance opens Agent v2 access methods
+ Given the Agent Builder stable chat model is available
+ And a runnable Agent v2 test agent has been created via API
+ When I open the Agent v2 configure page
+ And I publish the Agent v2 draft
+ And I follow the access methods link in the Agent v2 publish guidance
+ Then the Agent v2 Access Point should open
+
+ @core @prepared @stable-model @published-web-app
+ Scenario: Publish guidance opens the Agent v2 Web app
+ Given the Agent Builder stable chat model is available
And a runnable Agent v2 test agent has been created via API
When I open the Agent v2 configure page
- Then the Agent v2 publish action should be available for unpublished changes
- When I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
- And the Agent v2 publish action should be unavailable while up to date
- When I fill the Agent v2 prompt editor with the updated E2E prompt
- Then the Agent v2 configuration should be saved automatically
- And the Agent v2 publish action should be available for unpublished changes
+ And I publish the Agent v2 draft
+ And I follow the Web app link in the Agent v2 publish guidance
+ Then the Agent v2 Web app should open in a new tab
@core @prepared @stable-model
- Scenario: Restoring a published Agent v2 version shows the restored configuration in Builder
- Given I am signed in as the default E2E admin
- And the Agent Builder stable chat model is available
+ Scenario: Editing a published Agent v2 enables Publish update
+ Given the Agent Builder stable chat model is available
And a runnable Agent v2 test agent has been created via API
+ And the Agent v2 draft has been published via API
When I open the Agent v2 configure page
+ And I fill the Agent v2 prompt editor with the updated E2E prompt
+ Then the Agent v2 Publish update action should be available
+
+ @core @prepared @stable-model
+ Scenario: Publishing Agent v2 draft changes shows update guidance
+ Given the Agent Builder stable chat model is available
+ And a runnable Agent v2 test agent has been created via API
+ And the Agent v2 draft has been published via API
+ When I open the Agent v2 configure page
+ And I fill the Agent v2 prompt editor with the updated E2E prompt
And I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
- When I fill the Agent v2 prompt editor with the updated E2E prompt
- Then the Agent v2 configuration should be saved automatically
- And the normal Agent v2 draft should use the updated E2E prompt
- When I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
- When I open the Agent v2 version history
- And I select Agent v2 published version 1
- Then the selected Agent v2 version should be displayed in view-only mode
- And I should see the normal E2E prompt in the Agent v2 prompt editor
- When I restore the selected Agent v2 version
- Then I should see the normal E2E prompt in the Agent v2 prompt editor
- And the normal Agent v2 draft should use the normal E2E prompt
- And the Agent v2 publish action should be available for unpublished changes
+ Then the Agent v2 update publication should succeed
+
+ @core @prepared @stable-model
+ Scenario: Viewing a published Agent v2 version is read-only
+ Given the Agent Builder stable chat model is available
+ And an Agent v2 has original and updated published prompts
+ When I view the original Agent v2 published version in version history
+ Then the original Agent v2 version should be view-only
+
+ @core @prepared @stable-model
+ Scenario: Restoring a published Agent v2 version shows the restored configuration in Builder
+ Given the Agent Builder stable chat model is available
+ And an Agent v2 has original and updated published prompts
+ When I view the original Agent v2 published version in version history
+ And I restore the selected Agent v2 version
+ Then the original Agent v2 configuration should be an unpublished draft
@web-app-runtime @external-model @agent-backend-runtime @published-web-app @stable-model
Scenario: Published Agent v2 answers through Web app
- Given I am signed in as the default E2E admin
- And the Agent Builder stable chat model is available
+ Given the Agent Builder stable chat model is available
And the Agent v2 runtime backend is available
And a runnable Agent v2 test agent has been created via API
- When I open the Agent v2 configure page
- And I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
+ And the Agent v2 draft has been published via API
When I open the Agent v2 Web app URL
And I send an E2E message in the Agent v2 Web app
Then the Agent v2 Web app response should include the normal E2E marker
- When I close the Agent v2 Web app
@web-app-runtime @external-model @agent-backend-runtime @published-web-app @stable-model
Scenario: Published Web app remains isolated from unpublished Agent v2 draft edits
- Given I am signed in as the default E2E admin
- And the Agent Builder stable chat model is available
+ Given the Agent Builder stable chat model is available
And the Agent v2 runtime backend is available
And a runnable Agent v2 test agent has been created via API
- When I open the Agent v2 configure page
- And I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
- When I fill the Agent v2 prompt editor with the updated E2E prompt
- Then the Agent v2 configuration should be saved automatically
- And the normal Agent v2 draft should use the updated E2E prompt
+ And the Agent v2 draft has been published via API
+ And the Agent v2 draft has unpublished updated prompt changes
When I open the Agent v2 Web app URL
And I send an E2E message in the Agent v2 Web app
Then the Agent v2 Web app response should include the normal E2E marker
And the Agent v2 Web app response should not include the updated E2E marker
- When I close the Agent v2 Web app
@web-app-runtime @external-model @agent-backend-runtime @published-web-app @stable-model
Scenario: Published Web app uses the latest Agent v2 published configuration
- Given I am signed in as the default E2E admin
- And the Agent Builder stable chat model is available
+ Given the Agent Builder stable chat model is available
And the Agent v2 runtime backend is available
- And a runnable Agent v2 test agent has been created via API
- When I open the Agent v2 configure page
- And I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
- When I fill the Agent v2 prompt editor with the updated E2E prompt
- Then the Agent v2 configuration should be saved automatically
- And the normal Agent v2 draft should use the updated E2E prompt
- When I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
+ And an Agent v2 has original and updated published prompts
When I open the Agent v2 Web app URL
And I send an E2E message in the Agent v2 Web app
Then the Agent v2 Web app response should include the updated E2E marker
- When I close the Agent v2 Web app
diff --git a/e2e/features/agent-v2/tools.feature b/e2e/features/agent-v2/tools.feature
index c84f4274148..d05fd65c2de 100644
--- a/e2e/features/agent-v2/tools.feature
+++ b/e2e/features/agent-v2/tools.feature
@@ -22,7 +22,7 @@ Feature: Agent v2 tools
When I open the Agent v2 configure page
Then the Agent v2 JSON Replace tool should be saved in the Agent v2 draft
When I publish the Agent v2 draft
- Then the Agent v2 draft should be published and up to date
+ Then the Agent v2 first publication should succeed
Given an Agent v2 Backend service API key has been created via API
When I send the Agent v2 Backend service API JSON Replace request
Then the Agent v2 Backend service API response should include the JSON Replace E2E marker
diff --git a/e2e/features/step-definitions/agent-v2/publish.steps.ts b/e2e/features/step-definitions/agent-v2/publish.steps.ts
index 42d08e58c65..510271e3919 100644
--- a/e2e/features/step-definitions/agent-v2/publish.steps.ts
+++ b/e2e/features/step-definitions/agent-v2/publish.steps.ts
@@ -1,18 +1,58 @@
import type { DifyWorld } from '../../support/world.ts'
-import { Then, When } from '@cucumber/cucumber'
+import { Given, Then, When } from '@cucumber/cucumber'
import { expect } from '@playwright/test'
import { waitForAgentConfigureAutosaved } from '../../../support/agent-configure.ts'
-import { expectAgentModelRequiredFeedback, getCurrentAgentId } from './configure-helpers.ts'
+import {
+ createAgentSoulConfigWithModel,
+ normalAgentPrompt,
+ normalAgentSoulConfig,
+ updatedAgentSoulConfig,
+} from '../../agent-v2/support/agent-soul.ts'
+import {
+ createTestAgent,
+ getAgentConfigurePath,
+ publishAgentWithPublishableDraft,
+ saveAgentComposerDraft,
+} from '../../agent-v2/support/agent.ts'
+import { getCurrentAgentId } from './configure-helpers.ts'
+
+Given('an Agent v2 has original and updated published prompts', async function (this: DifyWorld) {
+ const stableModel = this.agentBuilder.fixtures.stableModel
+ if (!stableModel)
+ throw new Error('Create published Agent v2 versions after stable model fixture setup.')
+
+ const client = this.getConsoleClient()
+ const agent = await createTestAgent(client)
+ this.createdAgentIds.push(agent.id)
+ await saveAgentComposerDraft(
+ client,
+ agent.id,
+ createAgentSoulConfigWithModel(normalAgentSoulConfig, stableModel),
+ )
+ await publishAgentWithPublishableDraft(client, agent.id, 'E2E original prompt')
+ await saveAgentComposerDraft(
+ client,
+ agent.id,
+ createAgentSoulConfigWithModel(updatedAgentSoulConfig, stableModel),
+ )
+ await publishAgentWithPublishableDraft(client, agent.id, 'E2E updated prompt')
+})
-When('I publish the Agent v2 draft', async function (this: DifyWorld) {
- const page = this.getPage()
- const publishButton = page.getByRole('button', { name: /^Publish(?: update)?$/ })
+Given(
+ 'the Agent v2 draft has unpublished updated prompt changes',
+ async function (this: DifyWorld) {
+ const stableModel = this.agentBuilder.fixtures.stableModel
+ if (!stableModel) throw new Error('Update the Agent v2 draft after stable model fixture setup.')
- await expect(publishButton).toBeEnabled({ timeout: 30_000 })
- await publishButton.click()
-})
+ await saveAgentComposerDraft(
+ this.getConsoleClient(),
+ getCurrentAgentId(this),
+ createAgentSoulConfigWithModel(updatedAgentSoulConfig, stableModel),
+ )
+ },
+)
-When('I try to publish the Agent v2 draft without a model', async function (this: DifyWorld) {
+When('I publish the Agent v2 draft', async function (this: DifyWorld) {
const page = this.getPage()
const publishButton = page.getByRole('button', { name: /^Publish(?: update)?$/ })
@@ -20,102 +60,94 @@ When('I try to publish the Agent v2 draft without a model', async function (this
await publishButton.click()
})
-Then(
- 'Agent v2 publish should be blocked until a model is configured',
- async function (this: DifyWorld) {
- await expectAgentModelRequiredFeedback(this.getPage())
- },
-)
-
-Then('the Agent v2 draft should remain unpublished', async function (this: DifyWorld) {
- await expect
- .poll(
- async () => {
- const agentId = getCurrentAgentId(this)
- const composer = await this.getConsoleClient().agent.byAgentId.composer.get({
- params: { agent_id: agentId },
- })
- return composer.active_config_is_published
- },
- { timeout: 30_000 },
- )
- .toBe(false)
-})
-
Then('the Agent v2 configuration should be saved automatically', async function (this: DifyWorld) {
await waitForAgentConfigureAutosaved(this.getPage())
})
-Then('the Agent v2 draft should be published and up to date', async function (this: DifyWorld) {
+Then('the Agent v2 first publication should succeed', async function (this: DifyWorld) {
const page = this.getPage()
- const agentId = getCurrentAgentId(this)
- await expect(page.getByRole('button', { name: 'Published' })).toBeVisible({ timeout: 30_000 })
- await expect(page.getByRole('status', { name: /^Up to date\./ })).toBeVisible()
- await expect(page.getByText('Up to date')).toBeVisible()
- await expect
- .poll(async () => {
- const composer = await this.getConsoleClient().agent.byAgentId.composer.get({
- params: { agent_id: agentId },
- })
- return composer.active_config_is_published
- })
- .toBe(true)
+ await expect(
+ page.getByRole('paragraph').filter({ hasText: /^Published successfully$/ }),
+ ).toBeVisible({ timeout: 30_000 })
})
-Then(
- 'the Agent v2 publish action should be available for unpublished changes',
+Then('the Agent v2 update publication should succeed', async function (this: DifyWorld) {
+ await expect(
+ this.getPage()
+ .getByRole('paragraph')
+ .filter({ hasText: /^Changes published$/ }),
+ ).toBeVisible({ timeout: 30_000 })
+})
+
+When(
+ 'I follow the access methods link in the Agent v2 publish guidance',
async function (this: DifyWorld) {
- const page = this.getPage()
+ const link = this.getPage().getByRole('link', { name: 'View all access methods' })
- await expect(
- page.getByRole('status', { name: /^(?:Draft|Unpublished changes)\./ }),
- ).toBeVisible({ timeout: 30_000 })
- await expect(page.getByRole('button', { name: /^Publish(?: update)?$/ })).toBeEnabled()
+ await expect(link).toBeVisible({ timeout: 30_000 })
+ await link.click()
},
)
-Then(
- 'the Agent v2 publish action should be unavailable while up to date',
+Then('the Agent v2 Access Point should open', async function (this: DifyWorld) {
+ const page = this.getPage()
+
+ await expect(page).toHaveURL(new RegExp(`/agents/${getCurrentAgentId(this)}/access(?:\\?.*)?$`))
+ await expect(page.getByRole('region', { name: 'Access Point' })).toBeVisible()
+})
+
+When(
+ 'I follow the Web app link in the Agent v2 publish guidance',
async function (this: DifyWorld) {
const page = this.getPage()
+ const link = page.getByRole('link', { name: 'Open Web App' })
+
+ await expect(link).toBeVisible({ timeout: 30_000 })
+ const href = await link.getAttribute('href')
+ if (!href) throw new Error('Agent v2 publish guidance Web app link has no destination.')
- await expect(page.getByRole('status', { name: /^Up to date\./ })).toBeVisible({
- timeout: 30_000,
- })
- await expect(page.getByRole('button', { name: 'Published' })).toBeDisabled()
+ const [webAppPage] = await Promise.all([page.waitForEvent('popup'), link.click()])
+ this.agentBuilder.accessPoint.webAppURL = href
+ this.agentBuilder.accessPoint.webAppPage = webAppPage
},
)
-When('I open the Agent v2 version history', async function (this: DifyWorld) {
+Then('the Agent v2 Publish update action should be available', async function (this: DifyWorld) {
const page = this.getPage()
- await page.getByRole('button', { name: 'Open version history' }).click()
- await expect(page.getByRole('heading', { name: 'Versions' })).toBeVisible({ timeout: 30_000 })
+ await expect(page.getByRole('status', { name: /^Unpublished changes\./ })).toBeVisible({
+ timeout: 30_000,
+ })
+ await expect(page.getByRole('button', { name: 'Publish update' })).toBeEnabled()
})
When(
- 'I select Agent v2 published version {int}',
- async function (this: DifyWorld, versionNumber: number) {
+ 'I view the original Agent v2 published version in version history',
+ async function (this: DifyWorld) {
const page = this.getPage()
- const versionButton = page.getByRole('button', {
- name: new RegExp(`^Version ${versionNumber}\\b`),
- })
+
+ await page.goto(getAgentConfigurePath(getCurrentAgentId(this)))
+ await page.getByRole('button', { name: 'Open version history' }).click()
+ await expect(page.getByRole('heading', { name: 'Versions' })).toBeVisible({ timeout: 30_000 })
+ const versionButton = page.getByRole('button', { name: /^E2E original prompt\b/ })
await expect(versionButton).toBeVisible({ timeout: 30_000 })
await versionButton.click()
},
)
-Then(
- 'the selected Agent v2 version should be displayed in view-only mode',
- async function (this: DifyWorld) {
- const page = this.getPage()
-
- await expect(page.getByText('View Only')).toBeVisible({ timeout: 30_000 })
- await expect(page.getByRole('button', { name: 'Restore' })).toBeEnabled()
- },
-)
+Then('the original Agent v2 version should be view-only', async function (this: DifyWorld) {
+ const page = this.getPage()
+ const promptEditor = page
+ .getByRole('region', { name: 'Prompt' })
+ .getByRole('textbox', { name: 'Prompt' })
+
+ await expect(page.getByText('View Only')).toBeVisible({ timeout: 30_000 })
+ await expect(promptEditor).toContainText(normalAgentPrompt)
+ await expect(promptEditor).not.toBeEditable()
+ await expect(page.getByRole('button', { name: 'Restore' })).toBeEnabled()
+})
When('I restore the selected Agent v2 version', async function (this: DifyWorld) {
const page = this.getPage()
@@ -136,3 +168,18 @@ When('I restore the selected Agent v2 version', async function (this: DifyWorld)
const response = await restoreResponse
expect(response.ok()).toBe(true)
})
+
+Then(
+ 'the original Agent v2 configuration should be an unpublished draft',
+ async function (this: DifyWorld) {
+ const page = this.getPage()
+ const promptEditor = page
+ .getByRole('region', { name: 'Prompt' })
+ .getByRole('textbox', { name: 'Prompt' })
+
+ await expect(promptEditor).toContainText(normalAgentPrompt, { timeout: 30_000 })
+ await expect(promptEditor).toBeEditable()
+ await expect(page.getByRole('status', { name: /^Unpublished changes\./ })).toBeVisible()
+ await expect(page.getByRole('button', { name: 'Publish update' })).toBeEnabled()
+ },
+)