Doubao-powered voice input for Linux, Wayland and Omarchy
Midscene Test Node Reference
This document describes the nodes registered by the current Test Project.
Available Nodes
This table lists all available Nodes. See Node Details below for each Node's usage and input schema.
| Node | Description |
|---|---|
| aiAct | Perform a natural-language task with a Midscene UI Agent. |
| aiAssert | Assert a natural-language condition with a Midscene UI Agent. |
| aiAsk | Run aiAsk with a Midscene UI Agent and store its value. |
| aiBoolean | Run aiBoolean with a Midscene UI Agent and store its value. |
| aiNumber | Run aiNumber with a Midscene UI Agent and store its value. |
| aiString | Run aiString with a Midscene UI Agent and store its value. |
| aiTap | Locate and tap an element with a Midscene UI Agent. |
| computer.inputText | Replace text in a visually located input through the active Midscene Computer Agent. |
| computer.keyPress | Press a desktop shortcut through the active Midscene Computer Agent. |
| fixture.prepare | Select deterministic synthetic state for this test case. |
| recordToReport | Add text or screenshots to the current Midscene report. |
| shell.closeSystemMenu | Close the Omarchy system menu. |
| shell.moveBarLeft | Back up Omarchy bar settings and dock the bar left. |
| shell.openSystemMenu | Open the real Omarchy system menu and focus one row. |
| wait | Wait for a fixed duration while honoring cancellation. |
Where to use these Nodes
Use these Nodes in Workflow Documents (.yaml or .yml files) selected by an Execution Project. Case file patterns and the config file path below are relative to this reference file. Case file patterns use glob syntax. A custom --config path does not change where case files are selected.
Projects created by midscene-test create select cases/**/*.{yaml,yml} by default, so you can write test cases in cases/example.yaml or subdirectories of cases/. If an Execution Project has no files configuration, Midscene Test searches **/*.{yaml,yml} recursively instead. Check projects[].files in your configuration before choosing a file location.
Config file: midscene.config.ts
Case files: cases/polishing.yaml (Execution Project: ubuntu-polishing)
Case files: cases/onboarding.yaml, cases/onboarding-regressions.yaml, cases/runtime.yaml (Execution Project: ubuntu-product)
Case files: cases/onboarding.yaml, cases/onboarding-regressions.yaml, cases/runtime.yaml (Execution Project: omarchy-product)
Case files: cases/omarchy-shell.yaml (Execution Project: omarchy-shell)
Files matching files.exclude are not selected. Midscene Test always ignores .git/, .midscene/, midscene_run/, and node_modules/ directories, including nested ones.
Call a Node in cases[].steps, or in the Workflow Document lifecycle steps beforeAll, beforeEach, afterEach, and afterAll. Root Nodes are shared defaults; projects[].nodes adds local Nodes and overrides same-named root Nodes only within that Execution Project. Each Execution Project supplies its own runtime environment.
Workflow Step Contract
Each step uses a node name as its key. The $ object contains engine metadata and is not part of the node input.
cases:
- name: Example case
steps:
- node.name:
prompt: Describe the task.
$:
timeout: 30000
continue-on-error: false
A Node may map a string step value to one declared input field. Each Node section states whether it supports this shorthand and names the target field.
Node Details
aiAct
Perform a natural-language task with a Midscene UI Agent.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"cacheable": {
"type": "boolean"
},
"context": {
"type": "string"
},
"deepLocate": {
"type": "boolean"
},
"deepThink": {
"anyOf": [
{
"const": "unset",
"type": "string"
},
{
"type": "boolean"
}
]
},
"fileChooserAccept": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
]
},
"fileChooserAllowedDir": {
"type": "string"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
aiAssert
Assert a natural-language condition with a Midscene UI Agent.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"message": {
"type": "string"
},
"options": {
"additionalProperties": false,
"properties": {
"context": {
"type": "string"
},
"domIncluded": {
"anyOf": [
{
"type": "boolean"
},
{
"const": "visible-only",
"type": "string"
}
]
},
"keepRawResponse": {
"type": "boolean"
},
"screenshotIncluded": {
"type": "boolean"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
aiAsk
Run aiAsk with a Midscene UI Agent and store its value.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"context": {
"type": "string"
},
"domIncluded": {
"anyOf": [
{
"type": "boolean"
},
{
"const": "visible-only",
"type": "string"
}
]
},
"screenshotIncluded": {
"type": "boolean"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
aiBoolean
Run aiBoolean with a Midscene UI Agent and store its value.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"context": {
"type": "string"
},
"domIncluded": {
"anyOf": [
{
"type": "boolean"
},
{
"const": "visible-only",
"type": "string"
}
]
},
"screenshotIncluded": {
"type": "boolean"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
aiNumber
Run aiNumber with a Midscene UI Agent and store its value.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"context": {
"type": "string"
},
"domIncluded": {
"anyOf": [
{
"type": "boolean"
},
{
"const": "visible-only",
"type": "string"
}
]
},
"screenshotIncluded": {
"type": "boolean"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
aiString
Run aiString with a Midscene UI Agent and store its value.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"context": {
"type": "string"
},
"domIncluded": {
"anyOf": [
{
"type": "boolean"
},
{
"const": "visible-only",
"type": "string"
}
]
},
"screenshotIncluded": {
"type": "boolean"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
aiTap
Locate and tap an element with a Midscene UI Agent.
String shorthand: Maps to { prompt: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"cacheable": {
"type": "boolean"
},
"context": {
"type": "string"
},
"deepLocate": {
"type": "boolean"
},
"deepThink": {
"type": "boolean"
},
"fileChooserAccept": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
]
},
"xpath": {
"type": "string"
}
},
"type": "object"
},
"prompt": {
"anyOf": [
{
"pattern": "\\S",
"type": "string"
},
{
"additionalProperties": false,
"properties": {
"convertHttpImage2Base64": {
"type": "boolean"
},
"images": {
"items": {
"additionalProperties": false,
"properties": {
"name": {
"pattern": "\\S",
"type": "string"
},
"url": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"name",
"url"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"prompt": {
"pattern": "\\S",
"type": "string"
}
},
"required": [
"prompt"
],
"type": "object"
}
]
}
},
"required": [
"prompt"
],
"type": "object"
}
computer.inputText
Replace text in a visually located input through the active Midscene Computer Agent. An optional point focuses a fixed-size Omarchy fixture input when visual localization is unreliable.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"point": {
"additionalProperties": false,
"properties": {
"x": { "type": "number" },
"y": { "type": "number" }
},
"required": ["x", "y"],
"type": "object"
},
"target": {
"minLength": 1,
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"target",
"value"
],
"type": "object"
}
computer.keyPress
Press a desktop shortcut through the active Midscene Computer Agent.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"keyName": {
"enum": [
"F8",
"Escape"
],
"type": "string"
}
},
"required": [
"keyName"
],
"type": "object"
}
fixture.prepare
Select deterministic synthetic state for this test case.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"mode": {
"enum": [
"microphone-gate",
"voice-test",
"volcengine",
"microphone-change",
"shortcut-capture",
"runtime-delivery",
"runtime-cancel"
],
"type": "string"
}
},
"required": [
"mode"
],
"type": "object"
}
recordToReport
Add text or screenshots to the current Midscene report.
String shorthand: Maps to { title: value }.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"options": {
"additionalProperties": false,
"properties": {
"content": {
"type": "string"
},
"screenshotBase64": {
"type": "string"
},
"screenshots": {
"items": {
"additionalProperties": false,
"properties": {
"base64": {
"minLength": 1,
"type": "string"
},
"description": {
"type": "string"
}
},
"required": [
"base64"
],
"type": "object"
},
"minItems": 1,
"type": "array"
}
},
"type": "object"
},
"title": {
"type": "string"
}
},
"type": "object"
}
shell.closeSystemMenu
Close the Omarchy system menu.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {},
"type": "object"
}
shell.moveBarLeft
Back up Omarchy bar settings and dock the bar left.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {},
"type": "object"
}
shell.openSystemMenu
Open the real Omarchy system menu and focus one row.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {},
"type": "object"
}
wait
Wait for a fixed duration while honoring cancellation.
String shorthand: Not supported by this Node.
Input Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"properties": {
"duration": {
"exclusiveMinimum": 0,
"type": "number"
},
"unit": {
"default": "ms",
"enum": [
"ms",
"s",
"min"
],
"type": "string"
}
},
"required": [
"duration"
],
"type": "object"
}