Skip to content

Doubao-powered voice input for Linux, Wayland and Omarchy

21.6 KB
Raw Blame

Midscene Test Node Reference

This document describes the nodes registered by the current Test Project.

Available Nodes

This table lists all available Nodes. See Node Details below for each Node's usage and input schema.

Node Description
aiAct Perform a natural-language task with a Midscene UI Agent.
aiAssert Assert a natural-language condition with a Midscene UI Agent.
aiAsk Run aiAsk with a Midscene UI Agent and store its value.
aiBoolean Run aiBoolean with a Midscene UI Agent and store its value.
aiNumber Run aiNumber with a Midscene UI Agent and store its value.
aiString Run aiString with a Midscene UI Agent and store its value.
aiTap Locate and tap an element with a Midscene UI Agent.
computer.inputText Replace text in a visually located input through the active Midscene Computer Agent.
computer.keyPress Press a desktop shortcut through the active Midscene Computer Agent.
fixture.prepare Select deterministic synthetic state for this test case.
recordToReport Add text or screenshots to the current Midscene report.
shell.closeSystemMenu Close the Omarchy system menu.
shell.moveBarLeft Back up Omarchy bar settings and dock the bar left.
shell.openSystemMenu Open the real Omarchy system menu and focus one row.
wait Wait for a fixed duration while honoring cancellation.

Where to use these Nodes

Use these Nodes in Workflow Documents (.yaml or .yml files) selected by an Execution Project. Case file patterns and the config file path below are relative to this reference file. Case file patterns use glob syntax. A custom --config path does not change where case files are selected.

Projects created by midscene-test create select cases/**/*.{yaml,yml} by default, so you can write test cases in cases/example.yaml or subdirectories of cases/. If an Execution Project has no files configuration, Midscene Test searches **/*.{yaml,yml} recursively instead. Check projects[].files in your configuration before choosing a file location.

Config file: midscene.config.ts

Case files: cases/polishing.yaml (Execution Project: ubuntu-polishing)

Case files: cases/onboarding.yaml, cases/onboarding-regressions.yaml, cases/runtime.yaml (Execution Project: ubuntu-product)

Case files: cases/onboarding.yaml, cases/onboarding-regressions.yaml, cases/runtime.yaml (Execution Project: omarchy-product)

Case files: cases/omarchy-shell.yaml (Execution Project: omarchy-shell)

Files matching files.exclude are not selected. Midscene Test always ignores .git/, .midscene/, midscene_run/, and node_modules/ directories, including nested ones.

Call a Node in cases[].steps, or in the Workflow Document lifecycle steps beforeAll, beforeEach, afterEach, and afterAll. Root Nodes are shared defaults; projects[].nodes adds local Nodes and overrides same-named root Nodes only within that Execution Project. Each Execution Project supplies its own runtime environment.

Workflow Step Contract

Each step uses a node name as its key. The $ object contains engine metadata and is not part of the node input.

cases:
  - name: Example case
    steps:
      - node.name:
          prompt: Describe the task.
          $:
            timeout: 30000
            continue-on-error: false

A Node may map a string step value to one declared input field. Each Node section states whether it supports this shorthand and names the target field.

Node Details

aiAct

Perform a natural-language task with a Midscene UI Agent.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "cacheable": {
          "type": "boolean"
        },
        "context": {
          "type": "string"
        },
        "deepLocate": {
          "type": "boolean"
        },
        "deepThink": {
          "anyOf": [
            {
              "const": "unset",
              "type": "string"
            },
            {
              "type": "boolean"
            }
          ]
        },
        "fileChooserAccept": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            }
          ]
        },
        "fileChooserAllowedDir": {
          "type": "string"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

aiAssert

Assert a natural-language condition with a Midscene UI Agent.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "message": {
      "type": "string"
    },
    "options": {
      "additionalProperties": false,
      "properties": {
        "context": {
          "type": "string"
        },
        "domIncluded": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "const": "visible-only",
              "type": "string"
            }
          ]
        },
        "keepRawResponse": {
          "type": "boolean"
        },
        "screenshotIncluded": {
          "type": "boolean"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

aiAsk

Run aiAsk with a Midscene UI Agent and store its value.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "context": {
          "type": "string"
        },
        "domIncluded": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "const": "visible-only",
              "type": "string"
            }
          ]
        },
        "screenshotIncluded": {
          "type": "boolean"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

aiBoolean

Run aiBoolean with a Midscene UI Agent and store its value.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "context": {
          "type": "string"
        },
        "domIncluded": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "const": "visible-only",
              "type": "string"
            }
          ]
        },
        "screenshotIncluded": {
          "type": "boolean"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

aiNumber

Run aiNumber with a Midscene UI Agent and store its value.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "context": {
          "type": "string"
        },
        "domIncluded": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "const": "visible-only",
              "type": "string"
            }
          ]
        },
        "screenshotIncluded": {
          "type": "boolean"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

aiString

Run aiString with a Midscene UI Agent and store its value.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "context": {
          "type": "string"
        },
        "domIncluded": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "const": "visible-only",
              "type": "string"
            }
          ]
        },
        "screenshotIncluded": {
          "type": "boolean"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

aiTap

Locate and tap an element with a Midscene UI Agent.

String shorthand: Maps to { prompt: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "cacheable": {
          "type": "boolean"
        },
        "context": {
          "type": "string"
        },
        "deepLocate": {
          "type": "boolean"
        },
        "deepThink": {
          "type": "boolean"
        },
        "fileChooserAccept": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            }
          ]
        },
        "xpath": {
          "type": "string"
        }
      },
      "type": "object"
    },
    "prompt": {
      "anyOf": [
        {
          "pattern": "\\S",
          "type": "string"
        },
        {
          "additionalProperties": false,
          "properties": {
            "convertHttpImage2Base64": {
              "type": "boolean"
            },
            "images": {
              "items": {
                "additionalProperties": false,
                "properties": {
                  "name": {
                    "pattern": "\\S",
                    "type": "string"
                  },
                  "url": {
                    "pattern": "\\S",
                    "type": "string"
                  }
                },
                "required": [
                  "name",
                  "url"
                ],
                "type": "object"
              },
              "minItems": 1,
              "type": "array"
            },
            "prompt": {
              "pattern": "\\S",
              "type": "string"
            }
          },
          "required": [
            "prompt"
          ],
          "type": "object"
        }
      ]
    }
  },
  "required": [
    "prompt"
  ],
  "type": "object"
}

computer.inputText

Replace text in a visually located input through the active Midscene Computer Agent. An optional point focuses a fixed-size Omarchy fixture input when visual localization is unreliable.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "point": {
      "additionalProperties": false,
      "properties": {
        "x": { "type": "number" },
        "y": { "type": "number" }
      },
      "required": ["x", "y"],
      "type": "object"
    },
    "target": {
      "minLength": 1,
      "type": "string"
    },
    "value": {
      "type": "string"
    }
  },
  "required": [
    "target",
    "value"
  ],
  "type": "object"
}

computer.keyPress

Press a desktop shortcut through the active Midscene Computer Agent.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "keyName": {
      "enum": [
        "F8",
        "Escape"
      ],
      "type": "string"
    }
  },
  "required": [
    "keyName"
  ],
  "type": "object"
}

fixture.prepare

Select deterministic synthetic state for this test case.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "mode": {
      "enum": [
        "microphone-gate",
        "voice-test",
        "volcengine",
        "microphone-change",
        "shortcut-capture",
        "runtime-delivery",
        "runtime-cancel"
      ],
      "type": "string"
    }
  },
  "required": [
    "mode"
  ],
  "type": "object"
}

recordToReport

Add text or screenshots to the current Midscene report.

String shorthand: Maps to { title: value }.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "options": {
      "additionalProperties": false,
      "properties": {
        "content": {
          "type": "string"
        },
        "screenshotBase64": {
          "type": "string"
        },
        "screenshots": {
          "items": {
            "additionalProperties": false,
            "properties": {
              "base64": {
                "minLength": 1,
                "type": "string"
              },
              "description": {
                "type": "string"
              }
            },
            "required": [
              "base64"
            ],
            "type": "object"
          },
          "minItems": 1,
          "type": "array"
        }
      },
      "type": "object"
    },
    "title": {
      "type": "string"
    }
  },
  "type": "object"
}

shell.closeSystemMenu

Close the Omarchy system menu.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {},
  "type": "object"
}

shell.moveBarLeft

Back up Omarchy bar settings and dock the bar left.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {},
  "type": "object"
}

shell.openSystemMenu

Open the real Omarchy system menu and focus one row.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {},
  "type": "object"
}

wait

Wait for a fixed duration while honoring cancellation.

String shorthand: Not supported by this Node.

Input Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "duration": {
      "exclusiveMinimum": 0,
      "type": "number"
    },
    "unit": {
      "default": "ms",
      "enum": [
        "ms",
        "s",
        "min"
      ],
      "type": "string"
    }
  },
  "required": [
    "duration"
  ],
  "type": "object"
}