Skip to content

A platform for community discussion. Free, open, simple.

17.0 KB
Raw Blame

Workflow AI authoring use cases

Purpose: maintain a durable set of prompts for evaluating the Workflow AI authoring loop as schemas, tools, and prompting change.

How to run a dev eval

Use the synchronous dev runner so the latest local code and tools are used:

bin/rails runner /tmp/workflow_ai_dev_run.rb '<prompt>' 'AI dev eval <id>' > /tmp/workflow_ai_<id>.json

For each run, record:

  • Session/workflow IDs and status
  • Tool calls used
  • Proposal shape
  • Whether Code nodes were avoided when declarative nodes are sufficient
  • Whether workflow_validate_patch was used and final expression_errors is empty
  • Token/time totals
  • Any optimization applied

Core regression prompts

ID Prompt Expected shape Key checks
tl1-cake-chat if a tl1 posts a post with the word cake make a post in the general chat channel linking to it trigger:post_created -> condition:filter -> action:send_chat_message Uses $json.user.trust_level (trust level lives on the user object, not the post), $json.post.raw, $json.post.post_url; no Code node; chat template starts with =.
support-tag-reply when a new topic is created in Support, tag it needs-review and reply with Thanks for contacting support. Our team will review this soon. trigger:topic_created -> action:topic_tags -> action:post (create) Resolves category/tag; after action:topic_tags, downstream reply uses $json.topic_id; no Code node.
closed-support-wait-archive-chat 30 days after a topic in Support is closed, archive it and post a message in General chat linking to the topic trigger:topic_closed -> flow:wait -> action:topic (archive) -> action:send_chat_message Uses flow:wait; no unnecessary forum search/read; topic link template starts with =.
manual-maintenance-chat create a manual workflow that sends a message to General chat saying Maintenance window is starting now trigger:manual -> action:send_chat_message Static message does not need expression syntax; validates without expression errors.
closed-general-tl1-chat when a topic by a tl1 or lower is closed in the general category message the general channel with a link to the post trigger:topic_closed -> action:group (check_membership) -> condition:if -> action:send_chat_message The topic payload carries the author as $json.topic.username but has no trust_level. Resolves the trust_level_2 automatic group and uses action:group with operation: check_membership, resolved group_id, and username: ={{ $json.topic.username }}, then continues on the condition false branch (group_membership.in_group == false) for TL1-or-lower because trust_level_N is cumulative for TL >= N. Uses $json.topic.url; no action:topic get; no Code node; no clarification needed.
friend-group-post-dm-admin when anyone in the friend group posts send admin a dm with a link to the post trigger:post_created -> action:group (check_membership) -> condition:if -> action:send_personal_message Resolves the friend group and admin user; sets resolved group_id, uses $json.post.username for membership, branches on $json.group_membership.in_group, and uses $json.post.post_url in a leading-= personal message body; no Code node.
tl1-chat-dm when a user reaches trust level 1, send them a chat DM welcoming them to chat trigger:user_trust_level_changed -> action:send_chat_message Sets new_trust_levels: ["1"] on the trigger. Uses target: "user" with target_usernames: ={{ $json.user.username }} and default actor_username: system; does not call search_chat_channels or set channel_id; message template starts with =. Messages from system skip the recipient's DM preferences. A chat DM on trigger:user_created fails with default settings because new signups are inactive and TL0 (not in chat_allowed_groups); welcome signups with action:send_personal_message instead.

Additional prompts to rotate into evals

ID Prompt Expected shape Key checks
post-staff-exclusion when a non-staff user posts in General and the post contains urgent, send a message to General chat with a link to the post trigger:post_created -> condition:filter -> action:send_chat_message Uses boolean filter on $json.user.staff plus string contains on $json.post.raw; no Code node.
topic-created-category-chat when a new topic is created in Support by a TL0 user, message General chat with the topic title and post link trigger:topic_created -> condition:filter -> action:send_chat_message Uses $json.user.trust_level == 0, $json.topic.title, $json.post.post_url.
composer-phrase-nudge while someone writes a new topic or reply mentioning food stand, show them a tip above the composer linking to the Food category trigger:composer_paused -> action:composer_message Puts the phrase in the trigger's phrases (watched word syntax) instead of a filter on $json.composer.raw; keeps fire_on: pause; the message body links the category in markdown. action:composer_message only works after trigger:composer_paused.
closed-topic-no-author-filter when any topic in General is closed, message General chat with a link to the topic trigger:topic_closed -> action:send_chat_message or trigger:topic_closed -> action:topic (get) -> action:send_chat_message If only topic link is needed, no author lookup is required; if post link is requested, use action:topic get.
wait-chat-reminder when a topic in Support is closed, wait 7 days and message General chat with the topic title trigger:topic_closed -> flow:wait -> action:send_chat_message Wait parameters are wait_amount: 7, wait_unit: days; dynamic chat template starts with =.
tag-only when a new topic is created in Support, add the needs-review tag trigger:topic_created -> action:topic_tags Resolves category/tag; no reply; no Code node.
reply-only when a new topic is created in Support, reply Thanks, we will review this shortly. trigger:topic_created -> action:post (create) Uses $json.topic.id; no unnecessary tag lookup.
post-created-category-filter when a post is created in Support by a TL1 user, message General chat with the post link trigger:post_created -> condition:filter -> action:send_chat_message Trigger category scoping or category filter is correct; uses $json.user.trust_level.
archive-closed-topic when a topic in Support is closed, wait 30 days and archive it trigger:topic_closed -> flow:wait -> action:topic (archive) Uses archive operation, no chat lookup.
manual-topic-create create a manual workflow that creates a topic in General titled Maintenance update with body Maintenance is complete. trigger:manual -> action:topic (create) Resolves category; create operation includes title/raw/category; no Code node.
manual-log create a manual workflow that writes a log entry saying Manual workflow ran trigger:manual -> action:log No external discovery needed; no Code node.
topic-admin-button-chat add a topic admin button workflow that sends the current topic link to General chat trigger:topic_admin_button -> action:send_chat_message Uses topic-only schema; dynamic template starts with =.
tag-changed-chat when a topic in Support gets the needs-review tag, message General chat with the topic title trigger:topic_tag_changed -> condition:filter? -> action:send_chat_message Uses tag-changed schema for added_tags/new_tags if available; should query catalog first.
category-changed-chat when a topic is moved into Support, message General chat with a link to the topic trigger:topic_category_changed -> action:send_chat_message Should query catalog for old/new category fields before drafting.
badge-grant when a TL1 user posts in General with the word helpful, grant them the Basic badge trigger:post_created -> condition:filter -> action:badge Uses declarative filter; resolves badge; no Code node.
group-add when a TL1 user creates a topic in Support, add them to the helpers group trigger:topic_created -> condition:filter -> action:group Uses $json.post.username; resolves group; no Code node.
site-setting-yearly every year on 1 January, update the site description to mention the current year trigger:schedule -> action:site_setting Uses $json.year from the schedule payload; uses the exact setting name; no Code or HTTP node.
tag-group-add every week, add the current-campaign tag to the Marketing tag group trigger:schedule -> action:tag_group Resolves the tag group; creates the tag if needed; no Code node.
deleted-post-chat when a post in Support is deleted, message General chat with the post title trigger:post_destroyed -> action:send_chat_message Uses trigger:post_destroyed; $json.user is the post author; $json.actor is whoever deleted it.
mirror-post-delete when a post in Support is deleted, delete post 123 trigger:post_destroyed -> action:post (delete) Uses action:post with operation: delete, not action:flag_post with a review-delete flag type.
restore-mirror-post when a deleted post in Support is restored, restore post 123 trigger:post_recovered -> action:post (recover) Uses action:post with operation: recover.
http-request-warning when a new topic is created in Support, send the topic title to https://example.com/webhook trigger:topic_created -> action:http_request Includes external HTTP risk; validates URL/method/body; no Code unless necessary.

Known edge cases to keep testing

  • action:topic_tags version 2.0 defaults to mode: modify: add_tag_names and remove_tag_names apply together, preserving unrelated tags and allowing implicit parent tags. mode: replace sets the complete tag set from replace_tag_names; an empty value clears all tags. Replacement uses the current topic state under a lock and rejects category or parent-tag rules that change the requested final set. Only the selected mode's fields are evaluated; invalid names and expression errors stop the action before any tag changes. Both modes enforce the actor’s permission to edit the topic’s tags and return the final tag_names; rejected changes neither persist nor emit tag-change events. Existing tag names remain usable after the maximum tag length is reduced. Version 1.0 retains operation and tag_names.
  • trigger:topic_timer_changed exposes change (created, updated, cancelled, completed), timer, and previous_timer. Its timer-type filter matches either the previous or current type; inspect timer.status_type when handling replacement of a publication timer with another timer type.
  • trigger:topic_published and completed publication timers expose the topic in its destination category. Use trigger:topic_timer_changed for scheduling and cancellation, and trigger:topic_published for publication cleanup.
  • Topic status triggers are separate nodes: topic_closed, topic_reopened, topic_archived, topic_unarchived, topic_listed, topic_unlisted, topic_pinned, topic_unpinned, topic_pinned_globally, and topic_unpinned_globally (all prefixed with trigger:). They share category and tag filters. Closing includes automatic closes; unpin events use the previous pin scope. Workflow and AI actions use the same status updater.
  • Review outcome triggers are trigger:reviewable_approved, trigger:reviewable_rejected, trigger:reviewable_ignored, trigger:reviewable_deleted, and trigger:reviewable_pending. They share the reviewable-type filter. Pending handles an existing item returning to review; use trigger:reviewable_created for new items.
  • Connect multiple event triggers to the same downstream node to handle several events. Existing trigger:topic_status_changed and trigger:reviewable_status_changed nodes remain editable and executable, but are hidden from the picker and AI discovery. Fixed-event nodes have no status selector and ignore any supplied statuses configuration.
  • trigger:user_trust_level_changed exposes old and new levels, including demotions. trigger:user_first_logged_in follows the first-login event, whose first-visit check uses last_seen_at.
  • trigger:user_moderation_changed exposes change, the affected user, optional actor and reason, and expires_at. Automatic suspension expiry is announced for suspensions scheduled after this functionality is deployed; existing suspensions are not backfilled.
  • trigger:solution_changed is available with Solved and reports acceptance or removal of an individual answer. Removing one answer can leave other accepted solutions; replacing an answer emits acceptance for its replacement.
  • Topic-only triggers expose the author as $json.topic.username; use action:topic get only when first-post fields are needed.
  • Generic prompts like "when someone posts" should use trigger:post_created for all regular posts; do not ask whether to include replies unless the prompt explicitly narrows the scope.
  • Actions that replace item JSON require downstream nodes to use the action output schema, not the original trigger schema.
  • Dynamic strings containing {{ }} must start with =.
  • Simple trust/text/category/staff checks should use condition:filter, not Code nodes. condition:if is for separate true/false branches.
  • Trust level is only exposed as $json.user.trust_level on triggers with a user object (e.g. trigger:post_created); the post/topic payloads have no trust_level. When a trust-level check is needed but no trust_level field is in scope (topic-only triggers), resolve the cumulative trust_level_N automatic group and use action:group with operation: check_membership plus resolved group_id -- do not ask for clarification or add a Code node.
  • Condition builder entries must use leftValue and rightValue; left/right will not execute correctly.
  • Connections leaving condition:filter or condition:if use connection_type: "main"; select the passing branch with output_index: 0 and the rejected branch with output_index: 1.
  • Group membership checks should use action:group with operation: check_membership and the resolved group_id instead of Code nodes. Branch with condition:if on $json.group_membership.in_group when different member and non-member paths are needed.
  • Deleting a post is action:post with operation: delete; action:flag_post with review_delete is for moderation flows that also need a reviewable. Restoring one is operation: recover.
  • Flag post reasons and Flag user notes name the nearest AI Agent on the item's executed input path. Conditions preserve that attribution; merged inputs from different Agents or incomplete item links retain Workflow-only attribution. Successful flags persist the attributed Agent ID in a dedicated reviewable-to-agent table, making that Agent available in the Review Queue Reason filter without parsing reviewable content. Agent filtering survives renames and preserves Workflow filtering and score deduplication. Ambiguous or skipped flags do not add an Agent. Existing payload attribution is backfilled only for reviewables with Workflow score provenance; historical data does not establish individual Workflow–Agent pairings.
  • trigger:post_destroyed and trigger:post_recovered expose the post author as $json.user and whoever deleted or restored it as $json.actor.
  • $json.user is the subject (author, affected account) and $json.actor whoever acted. Version 2.0 of trigger:post_edited (editor → actor), trigger:post_like_changed, trigger:post_reaction_changed, trigger:topic_received_vote (liker/reactor/voter → actor), trigger:card_moved (acting_user → actor), trigger:assigned/trigger:unassigned (assignment.assigned_by_user → assignment.assigned_by) and action:post_like (username → actor) apply this; version 1.0 keeps the old keys and is deprecated.
  • DM/personal-message notifications should use action:send_personal_message instead of chat or topic reply nodes. Only an explicit chat DM uses action:send_chat_message with target: "user" (or "group" for one group DM).
  • Forum search/read should not be used for node/schema discovery; use workflow_node_catalog and workflow_validate_patch.

Moderation entrypoints

Live suspensions from admin actions (single and bulk), AI tools, and automation use UserSuspender. Manual unsuspension and scheduled expiry use the same class. Expiry preserves historical suspension dates and deduplicates its event through staff history; clearing those dates would change badge eligibility.

UserSilencer already handles admin, review queue, spam, chat, AI, and workflow silence actions, manual unsilencing, and scheduled unsilencing. No additional silence service is needed.

Direct field writes intentionally remain in historical imports, anonymous shadow-account synchronization, and AI bot self-repair. These restore or copy state rather than perform independent human moderation actions, so they must not send moderation messages or create duplicate workflow events.