Apache DataFusion SQL Query Engine
ci: run the Rust workflow for fork pull requests on the fork (#25591)
Kumar Ujjawal00c8ecbead60
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex f3c069f5d9..bdcabb8deb 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -1,3 +1,8 @@+> [!IMPORTANT]+>+> **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push.+> See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).+## Which issue does this PR close?<!--@@ -10,7 +15,7 @@ We generally require a GitHub issue to be filed for all bug fixes and enhancemen<!--Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.- Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.+ Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.Please explain the problem you are trying to solve in terms of the user-visiblebehavior, rather than the implementation.diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.ymlindex bee4cf5ec1..a34ef5b50f 100644--- a/.github/workflows/dev.yml+++ b/.github/workflows/dev.yml@@ -83,6 +83,12 @@ jobs:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1- run: pip install pyyaml- run: python3 ci/scripts/check_asf_yaml_status_checks.py+ - run: python3 ci/scripts/check_rust_workflow_skips.py+ - run: python3 ci/scripts/test_rust_workflow_skips.py+ - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0+ with:+ node-version: "24"+ - run: node --test ci/scripts/test_fork_ci_status.cjstypos:name: Spell Check with Typosdiff --git a/.github/workflows/fork_ci_status.yml b/.github/workflows/fork_ci_status.ymlnew file mode 100644index 0000000000..56213480b4--- /dev/null+++ b/.github/workflows/fork_ci_status.yml@@ -0,0 +1,319 @@+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++# # Fork CI status+#+# A pull request from a fork to `main` runs `rust.yml` on the fork. This+# workflow mirrors that run as one `Rust (fork)` check on the pull request, so+# reviewers see the result without opening the fork. It never checks out or+# runs code from the pull request: on `pull_request_target` it reads the fork's+# public run list and creates the check, and on a schedule it updates the+# active or missing runs of recently updated pull requests plus a rotating+# slice of all open pull requests. The check is informational. The merge+# queue runs the full workflow on the merged result and stays the gate.+#+# See docs/source/contributor-guide/index.md, "Pull Requests from Forks".+name: Fork CI status++on:+ pull_request_target:+ types: [opened, reopened, synchronize]+ schedule:+ - cron: "*/15 * * * *"+ workflow_dispatch:+ inputs:+ hours:+ description: "Update every pull request updated within this many hours"+ type: number+ default: 6++permissions:+ actions: read+ checks: write+ pull-requests: read++concurrency:+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || 'update' }}+ cancel-in-progress: ${{ github.event_name == 'pull_request_target' }}++jobs:+ bridge:+ name: Mirror the fork run+ runs-on: ubuntu-slim+ timeout-minutes: 10+ steps:+ - name: Create or update the check+ uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0+ env:+ UPDATE_HOURS: ${{ inputs.hours || 6 }}+ with:+ script: |+ const CHECK = 'Rust (fork)';+ const WORKFLOW = 'rust.yml';+ const upstream = { owner: context.repo.owner, repo: context.repo.repo };+ const GUIDE = 'https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks';+ // Older pull requests are revisited this many at a time, so a rerun+ // or a late run on a quiet pull request is mirrored within hours.+ const SWEEP_SIZE = 25;++ // A check run accepts a fixed set of conclusions; a workflow run can+ // report others, such as startup_failure.+ const CONCLUSIONS = new Set([+ 'action_required', 'cancelled', 'failure', 'neutral',+ 'skipped', 'stale', 'success', 'timed_out',+ ]);+ const conclusionOf = (run) =>+ CONCLUSIONS.has(run.conclusion) ? run.conclusion : 'failure';+ const statusOf = (run) => {+ if (run.status === 'completed') return 'completed';+ return run.status === 'in_progress' ? 'in_progress' : 'queued';+ };+ const isForkPrToMain = (pr) =>+ Boolean(pr.head.repo) && pr.head.repo.fork === true && pr.base.ref === 'main';++ const HOW_UPDATED =+ 'Active or missing runs on recently updated pull requests are checked every 15 minutes. ' ++ 'Completed runs and older pull requests are checked by a rotating sweep, which can take a few hours. ' ++ 'The merge queue runs the full workflow on the merged result before this pull request merges.';++ // The check for a run. `text` is read back by the scheduled update.+ const runPayload = (fork, sha, run) => {+ const status = statusOf(run);+ return {+ status,+ ...(status === 'completed' ? { conclusion: conclusionOf(run) } : {}),+ details_url: run.html_url,+ output: {+ title: `Fork run ${status === 'completed' ? run.conclusion : status}`,+ summary: [+ `[Rust workflow run on ${fork.owner}/${fork.repo}](${run.html_url}) for this commit.`,+ '',+ 'If the fork run failed for reasons unrelated to this pull request, rerun it on the fork.',+ HOW_UPDATED,+ ].join('\n'),+ text: JSON.stringify({+ owner: fork.owner,+ repo: fork.repo,+ head_sha: sha,+ run_id: run.id,+ run_attempt: run.run_attempt,+ }),+ },+ };+ };++ // The check when the fork has no run for the commit. The workflow's+ // state on the fork says whether workflows are enabled there; a+ // push that changes only documentation or Markdown does not start+ // rust.yml even when they are. The scheduled update keeps looking+ // for a run in every case.+ const missingPayload = (fork, sha, state) => {+ const disabled = typeof state === 'string' && state.startsWith('disabled');+ const active = state === 'active';+ const advice = disabled+ ? [+ '1. Enable workflows on your fork: open its **Actions** tab and confirm the prompt.',+ '2. Push again. A push that changes only documentation or Markdown does not start',+ ' the Rust workflow.',+ ]+ : active+ ? [+ 'A push that changes only documentation or Markdown does not start the Rust',+ 'workflow, and a run can take a moment to appear. This check keeps looking for one.',+ ]+ : [+ 'GitHub reports no Rust workflow on the fork, which usually means workflows are not',+ 'enabled there. Enable them on the fork\'s **Actions** tab and push again. A push that',+ 'changes only documentation or Markdown does not start the Rust workflow.',+ ];+ return {+ status: 'completed',+ conclusion: disabled ? 'action_required' : active ? 'skipped' : 'neutral',+ output: {+ title: disabled ? 'Workflows are disabled on the fork' : 'No fork run for this commit',+ summary: [+ `No Rust workflow run was found on ${fork.owner}/${fork.repo} for ${sha}.`,+ '',+ ...advice,+ '',+ HOW_UPDATED,+ '',+ `See ${GUIDE}`,+ ].join('\n'),+ text: JSON.stringify({ owner: fork.owner, repo: fork.repo, head_sha: sha, run_id: null }),+ },+ };+ };++ // The fork creates its run a few seconds after the push.+ async function findRun(fork, sha, attempts) {+ for (let attempt = 0; attempt < attempts; attempt++) {+ try {+ const { data } = await github.rest.actions.listWorkflowRuns({+ owner: fork.owner,+ repo: fork.repo,+ workflow_id: WORKFLOW,+ head_sha: sha,+ per_page: 10,+ });+ // A newer manual run can replace a cancelled push run.+ const run = data.workflow_runs[0];+ if (run) return run;+ } catch (error) {+ // 404 when the fork has Actions disabled or lacks the workflow.+ core.info(`No run list for ${fork.owner}/${fork.repo}: ${error.message}`);+ }+ if (attempt + 1 < attempts) {+ await new Promise((resolve) => setTimeout(resolve, 5000));+ }+ }+ return null;+ }++ // `active`, or `disabled_fork` until the owner enables workflows;+ // null when the workflow cannot be read.+ async function workflowState(fork) {+ try {+ const { data } = await github.rest.actions.getWorkflow({+ owner: fork.owner,+ repo: fork.repo,+ workflow_id: WORKFLOW,+ });+ return data.state || null;+ } catch (error) {+ core.info(`No workflow state for ${fork.owner}/${fork.repo}: ${error.message}`);+ return null;+ }+ }++ async function createCheck(pr, attempts) {+ const fork = { owner: pr.head.repo.owner.login, repo: pr.head.repo.name };+ const sha = pr.head.sha;+ const run = await findRun(fork, sha, attempts);+ const payload = run+ ? runPayload(fork, sha, run)+ : missingPayload(fork, sha, await workflowState(fork));+ await github.rest.checks.create({ ...upstream, name: CHECK, head_sha: sha, ...payload });+ core.info(`Created ${CHECK} for ${sha}: ${payload.conclusion || payload.status}`);+ }++ async function notify() {+ const pr = context.payload.pull_request;+ // Only fork PRs to main skip jobs upstream; release branches have+ // no merge queue and keep the full run.+ if (!isForkPrToMain(pr)) {+ core.info('Not a fork pull request to main; nothing to mirror.');+ return;+ }+ await createCheck(pr, 12);+ }++ // Discover the latest run again, since a manual run can replace the+ // recorded run, and the fork can delete old runs.+ async function syncCheck(pr, includeCompleted = true) {+ const { data } = await github.rest.checks.listForRef({+ ...upstream,+ ref: pr.head.sha,+ check_name: CHECK,+ per_page: 5,+ });+ const check = data.check_runs[0];+ if (!check) {+ // Recover if the initial notification failed before creating a check.+ await createCheck(pr, 1);+ return;+ }+ if (!check.output || !check.output.text) return;+ let params;+ try {+ params = JSON.parse(check.output.text);+ } catch {+ return;+ }+ // Completed runs only need the rotating sweep to discover reruns.+ // A completed check with no run must keep looking for a late run.+ if (!includeCompleted && check.status === 'completed' && params.run_id) return;+ const fork = { owner: pr.head.repo.owner.login, repo: pr.head.repo.name };+ const sha = pr.head.sha;+ const run = await findRun(fork, sha, 1);+ if (!run) return;+ const payload = runPayload(fork, sha, run);+ if (+ payload.status === check.status &&+ (payload.conclusion || null) === check.conclusion &&+ run.id === params.run_id &&+ run.run_attempt === params.run_attempt+ ) {+ return;+ }+ await github.rest.checks.update({ ...upstream, check_run_id: check.id, ...payload });+ core.info(`#${pr.number}: ${check.status}/${check.conclusion} -> ${payload.status}/${payload.conclusion || ''}`);+ }++ const listing = { ...upstream, state: 'open', sort: 'updated', direction: 'desc' };++ // The number of open pull requests, from the last-page link of a+ // one-item page: one request, whatever the count.+ async function openPullRequestCount() {+ const response = await github.rest.pulls.list({ ...listing, per_page: 1 });+ const last = /[?&]page=(\d+)>;\s*rel="last"/.exec(response.headers.link || '');+ return last ? Number(last[1]) : response.data.length;+ }++ async function update() {+ const hours = Number(process.env.UPDATE_HOURS) || 6;+ const cutoff = Date.now() - hours * 3600 * 1000;+ const seen = new Set();++ // Select the sweep first so a recent completed PR in this slice+ // is checked once, even though it also appears in the window.+ const slices = Math.max(1, Math.ceil((await openPullRequestCount()) / SWEEP_SIZE));+ const slot = Math.floor(Date.now() / (15 * 60 * 1000));+ const page = (slot % slices) + 1;+ const { data: slice } = await github.rest.pulls.list({ ...listing, per_page: SWEEP_SIZE, page });+ const sweepNumbers = new Set(slice.map((pr) => pr.number));++ // Pull requests updated within the window, newest first. Stop at+ // the window: GITHUB_TOKEN allows 1,000 requests per hour per+ // repository.+ const pages = github.paginate.iterator(github.rest.pulls.list, { ...listing, per_page: 100 });+ for await (const page of pages) {+ let stale = false;+ for (const pr of page.data) {+ if (new Date(pr.updated_at).getTime() < cutoff) {+ stale = true;+ break;+ }+ seen.add(pr.number);+ if (isForkPrToMain(pr)) await syncCheck(pr, sweepNumbers.has(pr.number));+ }+ if (stale) break;+ }++ // Check the rest of the slice, including older pull requests.+ // A fork rerun does not update the upstream pull request.+ for (const pr of slice) {+ if (seen.has(pr.number) || !isForkPrToMain(pr)) continue;+ await syncCheck(pr);+ }+ }++ if (context.eventName === 'pull_request_target') {+ await notify();+ } else {+ await update();+ }diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.ymlindex 383d268854..1b68535fdd 100644--- a/.github/workflows/rust.yml+++ b/.github/workflows/rust.yml@@ -38,13 +38,25 @@# The other jobs carry an `if:` that skips them for that event alone. PRs, the# merge queue, `workflow_dispatch`, and every other branch run all jobs. A# direct push that bypasses the queue is not re-checked; dispatch the workflow-# on `main` for a full run. `ci/scripts/check_asf_yaml_status_checks.py` keeps+# on `main` for a full run. `ci/scripts/check_rust_workflow_skips.py` keeps# the skipped set explicit.#+# # Pull requests from forks+#+# A PR from a fork runs this workflow on the fork, on the contributor's own+# runners, and `fork_ci_status.yml` mirrors that run as the `Rust (fork)` check+# on the PR. Upstream runs `check-fmt`, `check-workflow-tool-installs`, and+# `examples-check-datafusion-crates` for such a PR; the other jobs carry an+# `if:` that skips them. The merge queue runs every job on the merged result,+# so it stays the gate. PRs from branches in this repository, and fork PRs to+# release branches, which have no merge queue, run every job here, as before.+#name: Rustconcurrency:- group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}+ # Fork pushes cancel superseded runs on the same branch. Upstream main and+ # merge queue runs keep separate groups for each SHA.+ group: ${{ github.repository }}-${{ github.head_ref || (github.repository == 'apache/datafusion' && github.sha || github.ref) }}-${{ github.workflow }}cancel-in-progress: trueon:@@ -71,6 +83,7 @@ jobs:# Check crate compiles and base cargo check passeslinux-build-lib:name: linux build test+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=8,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -121,7 +134,7 @@ jobs:linux-datafusion-common-features:name: cargo check datafusion-common featuresneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rust@@ -152,7 +165,7 @@ jobs:linux-datafusion-substrait-features:name: cargo check datafusion-substrait featuresneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}env:CARGO_INCREMENTAL: "0"runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}@@ -200,7 +213,7 @@ jobs:linux-datafusion-proto-features:name: cargo check datafusion-proto featuresneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -237,6 +250,7 @@ jobs:linux-datafusion-ffi-features:name: cargo check datafusion-ffi featuresneeds: linux-build-lib+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -263,7 +277,7 @@ jobs:linux-cargo-check-datafusion:name: cargo check datafusion featuresneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}env:CARGO_INCREMENTAL: "0"runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}@@ -339,6 +353,7 @@ jobs:linux-cargo-check-datafusion-functions:name: cargo check datafusion-functions featuresneeds: linux-build-lib+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rust@@ -384,7 +399,7 @@ jobs:linux-cargo-check-datafusion-spark:name: cargo check datafusion-spark featuresneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rust@@ -413,6 +428,7 @@ jobs:linux-test:name: cargo test (amd64)needs: linux-build-lib+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -466,7 +482,7 @@ jobs:linux-test-datafusion-cli:name: cargo test datafusion-cli (amd64)needs: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}steps:- uses: runs-on/action@17b59092eb37ebfdbb034460892887c09100eaa3 # v2.5.0@@ -512,6 +528,7 @@ jobs:linux-test-example:name: cargo examples (amd64)needs: linux-build-lib+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -543,7 +560,7 @@ jobs:linux-test-doc:name: cargo test doc (amd64)needs: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -566,7 +583,7 @@ jobs:linux-rustdoc:name: cargo docneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -582,7 +599,7 @@ jobs:linux-wasm-pack:name: build and run with wasm-pack- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-24.04env:# Use the preinstalled Clang 18 even if the runner's default Clang changes.@@ -614,7 +631,7 @@ jobs:verify-benchmark-results:name: verify benchmark results (amd64)needs: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -647,7 +664,7 @@ jobs:sqllogictest-postgres:name: "Run sqllogictest with Postgres runner"needs: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -685,7 +702,7 @@ jobs:sqllogictest-substrait:name: "Run sqllogictest in Substrait round-trip mode"needs: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -734,6 +751,7 @@ jobs:# flaky sqllogictest metrics.macos-aarch64:name: cargo test (macos-aarch64)+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: macos-15steps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1@@ -748,7 +766,7 @@ jobs:vendor:name: Verify Vendored Code- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rust@@ -793,6 +811,7 @@ jobs:clippy:name: clippyneeds: linux-build-lib+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -820,7 +839,7 @@ jobs:cargo-toml-formatting-checks:name: check Cargo.toml formattingneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rust@@ -844,7 +863,7 @@ jobs:config-docs-check:name: check configs.md and ***_functions.md is up-to-dateneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}container:image: amd64/rust@@ -874,7 +893,7 @@ jobs:examples-docs-check:name: check example README is up-to-dateneeds: linux-build-lib- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rust@@ -917,7 +936,7 @@ jobs:# - datafusion-climsrv:name: Verify MSRV (Min Supported Rust Version)- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}runs-on: ubuntu-latestcontainer:image: amd64/rustdiff --git a/ci/scripts/check_asf_yaml_status_checks.py b/ci/scripts/check_asf_yaml_status_checks.pyindex 1bd2852f4e..1356541590 100755--- a/ci/scripts/check_asf_yaml_status_checks.py+++ b/ci/scripts/check_asf_yaml_status_checks.py@@ -24,11 +24,6 @@ check from running on some PRs, blocking merges).A typo or stale entry in required_status_checks will block allmerges for the project, so this check catches that early.--Also validate the post-merge conditions in rust.yml: exactly the jobs-in POST_MERGE_SKIPPED_JOBS carry the condition that skips them on-pushes to upstream `main`, and the Cargo check artifact steps are-guarded the same way."""import glob@@ -37,57 +32,6 @@ import sysimport yaml-# Commits reach upstream `main` through the merge queue, which already ran-# rust.yml at the same SHA. On a push to upstream `main`, rust.yml skips the-# jobs below and keeps only the jobs that save caches, publish coverage, or run-# the non-required FFI check. A job is skipped only when it is listed here and-# carries exactly POST_MERGE_SKIP_CONDITION, so a new job runs by default and-# changing the set is a deliberate edit to both places.-RUST_WORKFLOW = "rust.yml"-POST_MERGE_PUSH = (- "github.event_name == 'push'"- " && github.ref == 'refs/heads/main'"- " && github.repository == 'apache/datafusion'"-)-POST_MERGE_SKIP_CONDITION = "${{ !(" + POST_MERGE_PUSH + ") }}"-POST_MERGE_SKIPPED_JOBS = frozenset(- {- "cargo-toml-formatting-checks",- "check-fmt",- "check-workflow-tool-installs",- "config-docs-check",- "examples-docs-check",- "linux-cargo-check-datafusion",- "linux-cargo-check-datafusion-spark",- "linux-datafusion-common-features",- "linux-datafusion-proto-features",- "linux-datafusion-substrait-features",- "linux-rustdoc",- "linux-test-datafusion-cli",- "linux-test-doc",- "linux-wasm-pack",- "msrv",- "sqllogictest-postgres",- "sqllogictest-substrait",- "vendor",- "verify-benchmark-results",- }-)-# `linux-build-lib` uploads its Cargo check artifacts for two skipped jobs, so-# the archive and upload steps skip on the same pushes.-ARTIFACT_JOB = "linux-build-lib"-ARTIFACT_ARCHIVE_STEP = "archive-check"-ARTIFACT_NAME = "cargo-check"-ARTIFACT_UPLOAD_CONDITION = (- "${{ steps." + ARTIFACT_ARCHIVE_STEP + ".outcome == 'success'"- " && !(" + POST_MERGE_PUSH + ") }}"-)---def load_workflow(path):- with open(path) as f:- return yaml.safe_load(f)-def get_required_checks(asf_yaml_path):"""Extract all required_status_checks contexts from .asf.yaml."""@@ -114,7 +58,8 @@ def get_workflow_jobs(workflows_dir):"""jobs = {} # identifier -> [(workflow_file, has_path_filters)]for workflow_file in sorted(glob.glob(os.path.join(workflows_dir, "*.yml"))):- workflow = load_workflow(workflow_file)+ with open(workflow_file) as f:+ workflow = yaml.safe_load(f)if not workflow or "jobs" not in workflow:continue@@ -140,90 +85,6 @@ def get_workflow_jobs(workflows_dir):return jobs-def iter_steps(job_config):- """Yield the steps of a job, flattening `parallel:` groups."""- for step in job_config.get("steps") or []:- if not isinstance(step, dict):- continue- if "parallel" in step:- yield from (s for s in step["parallel"] or [] if isinstance(s, dict))- else:- yield step---def uses_action(step, action):- return str(step.get("uses", "")).startswith(action)---def check_post_merge_conditions(workflow):- """Return the problems with rust.yml's post-merge job and artifact conditions."""- errors = []- jobs = {- key: config- for key, config in (workflow.get("jobs") or {}).items()- if isinstance(config, dict)- }-- for job in sorted(POST_MERGE_SKIPPED_JOBS - set(jobs)):- errors.append(- f" - job `{job}` is listed in POST_MERGE_SKIPPED_JOBS "- f"but does not exist in {RUST_WORKFLOW}"- )- for job, config in sorted(jobs.items()):- condition = config.get("if")- if job in POST_MERGE_SKIPPED_JOBS:- if condition != POST_MERGE_SKIP_CONDITION:- errors.append(- f" - job `{job}` must carry exactly "- f"`if: {POST_MERGE_SKIP_CONDITION}`, found {condition!r}"- )- elif condition is not None and any(- token in str(condition)- for token in ("github.event_name", "github.ref", "refs/heads/")- ):- errors.append(- f" - job `{job}` is not listed in POST_MERGE_SKIPPED_JOBS "- f"but carries an event condition: {condition!r}"- )-- build = jobs.get(ARTIFACT_JOB)- if build is None:- errors.append(f" - job `{ARTIFACT_JOB}` does not exist in {RUST_WORKFLOW}")- else:- steps = list(iter_steps(build))- archive = [s for s in steps if s.get("id") == ARTIFACT_ARCHIVE_STEP]- uploads = [s for s in steps if uses_action(s, "actions/upload-artifact")]- if len(archive) != 1 or archive[0].get("if") != POST_MERGE_SKIP_CONDITION:- found = [s.get("if") for s in archive]- errors.append(- f" - the step `id: {ARTIFACT_ARCHIVE_STEP}` in `{ARTIFACT_JOB}` "- f"must exist once and carry exactly "- f"`if: {POST_MERGE_SKIP_CONDITION}`, found {found!r}"- )- if len(uploads) != 1 or uploads[0].get("if") != ARTIFACT_UPLOAD_CONDITION:- found = [s.get("if") for s in uploads]- errors.append(- f" - the upload-artifact step in `{ARTIFACT_JOB}` must exist "- f"once and carry exactly `if: {ARTIFACT_UPLOAD_CONDITION}`, "- f"found {found!r}"- )-- for job, config in sorted(jobs.items()):- downloads = [- s- for s in iter_steps(config)- if uses_action(s, "actions/download-artifact")- and (s.get("with") or {}).get("name") == ARTIFACT_NAME- ]- if downloads and job not in POST_MERGE_SKIPPED_JOBS:- errors.append(- f" - job `{job}` downloads the `{ARTIFACT_NAME}` artifact, which "- f"is not uploaded on pushes to main, so it must be listed in "- f"POST_MERGE_SKIPPED_JOBS"- )- return errors--def main():repo_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))@@ -234,6 +95,7 @@ def main():required_checks = get_required_checks(asf_yaml)if not required_checks:print("No required_status_checks found in .asf.yaml — nothing to validate.")+ returnjobs = get_workflow_jobs(workflows_dir)errors = []@@ -264,10 +126,6 @@ def main():f"and will block merging")- post_merge_errors = check_post_merge_conditions(- load_workflow(os.path.join(workflows_dir, RUST_WORKFLOW))- )-if errors:print("ERROR: Problems found with required_status_checks in .asf.yaml:\n")print("\n".join(errors))@@ -275,23 +133,11 @@ def main():print("Available job names across all workflows:")for name in sorted(jobs):print(f" - {name}")- if post_merge_errors:- print(- f"ERROR: Problems found with post-merge conditions in {RUST_WORKFLOW}:\n"- )- print("\n".join(post_merge_errors))- print()- if errors or post_merge_errors:sys.exit(1)- if required_checks:- print(- f"OK: All {len(required_checks)} required_status_checks "- "match existing GitHub Actions jobs."- )print(- f"OK: {RUST_WORKFLOW} skips exactly {len(POST_MERGE_SKIPPED_JOBS)} "- "jobs on pushes to main."+ f"OK: All {len(required_checks)} required_status_checks "+ "match existing GitHub Actions jobs.")diff --git a/ci/scripts/check_rust_workflow_skips.py b/ci/scripts/check_rust_workflow_skips.pynew file mode 100755index 0000000000..4d9ea090d3--- /dev/null+++ b/ci/scripts/check_rust_workflow_skips.py@@ -0,0 +1,284 @@+#!/usr/bin/env python3+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++"""+Validate the job-level skip conditions in rust.yml.++rust.yml skips jobs on two events. Each skip set below pairs a clause with+the jobs that carry it: a job's `if:` is exactly `!(clause)` for every set+it belongs to, joined with ` && ` in the order of SKIP_POLICIES, and a job+in no set carries no job-level `if:`. A new job therefore runs everywhere+by default, and changing a set is a deliberate edit to both files.++The Cargo check artifact steps in `linux-build-lib` and the jobs that+download the artifact must follow the same sets.+"""++import os+import sys++import yaml++sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))++from check_asf_yaml_status_checks import get_required_checks # noqa: E402++RUST_WORKFLOW = "rust.yml"++# Skipped on a push to upstream main; see "Pushes to main" in rust.yml.+POST_MERGE_PUSH = (+ "github.event_name == 'push'"+ " && github.ref == 'refs/heads/main'"+ " && github.repository == 'apache/datafusion'"+)+POST_MERGE_SKIPPED_JOBS = frozenset(+ {+ "cargo-toml-formatting-checks",+ "check-fmt",+ "check-workflow-tool-installs",+ "config-docs-check",+ "examples-docs-check",+ "linux-cargo-check-datafusion",+ "linux-cargo-check-datafusion-spark",+ "linux-datafusion-common-features",+ "linux-datafusion-proto-features",+ "linux-datafusion-substrait-features",+ "linux-rustdoc",+ "linux-test-datafusion-cli",+ "linux-test-doc",+ "linux-wasm-pack",+ "msrv",+ "sqllogictest-postgres",+ "sqllogictest-substrait",+ "vendor",+ "verify-benchmark-results",+ }+)++# Skipped on a pull request from a fork; see "Pull requests from forks" in+# rust.yml and fork_ci_status.yml.+FORK_PULL_REQUEST = (+ "github.event_name == 'pull_request'"+ " && github.event.pull_request.head.repo.fork"+ " && github.base_ref == 'main'"+)+FORK_PR_SKIPPED_JOBS = frozenset(+ {+ "cargo-toml-formatting-checks",+ "clippy",+ "config-docs-check",+ "examples-docs-check",+ "linux-build-lib",+ "linux-cargo-check-datafusion",+ "linux-cargo-check-datafusion-functions",+ "linux-cargo-check-datafusion-spark",+ "linux-datafusion-common-features",+ "linux-datafusion-ffi-features",+ "linux-datafusion-proto-features",+ "linux-datafusion-substrait-features",+ "linux-rustdoc",+ "linux-test",+ "linux-test-datafusion-cli",+ "linux-test-doc",+ "linux-test-example",+ "linux-wasm-pack",+ "macos-aarch64",+ "msrv",+ "sqllogictest-postgres",+ "sqllogictest-substrait",+ "vendor",+ "verify-benchmark-results",+ }+)++SKIP_POLICIES = (+ ("POST_MERGE_SKIPPED_JOBS", POST_MERGE_SKIPPED_JOBS, POST_MERGE_PUSH),+ ("FORK_PR_SKIPPED_JOBS", FORK_PR_SKIPPED_JOBS, FORK_PULL_REQUEST),+)+++def expected_condition(job):+ """The exact job-level `if:` for a job, or None when it must carry none."""+ clauses = [clause for _, jobs, clause in SKIP_POLICIES if job in jobs]+ if not clauses:+ return None+ return "${{ " + " && ".join(f"!({clause})" for clause in clauses) + " }}"+++# `linux-build-lib` uploads Cargo check artifacts for two consumers. Its steps+# skip on a push to main and the whole job skips on a fork PR.+ARTIFACT_JOB = "linux-build-lib"+ARTIFACT_ARCHIVE_STEP = "archive-check"+ARTIFACT_NAME = "cargo-check"+ARTIFACT_ARCHIVE_CONDITION = "${{ !(" + POST_MERGE_PUSH + ") }}"+ARTIFACT_UPLOAD_CONDITION = (+ "${{ steps." + ARTIFACT_ARCHIVE_STEP + ".outcome == 'success'"+ " && !(" + POST_MERGE_PUSH + ") }}"+)+++def iter_steps(job_config):+ """Yield the steps of a job, flattening `parallel:` groups."""+ for step in job_config.get("steps") or []:+ if not isinstance(step, dict):+ continue+ if "parallel" in step:+ yield from (s for s in step["parallel"] or [] if isinstance(s, dict))+ else:+ yield step+++def uses_action(step, action):+ return str(step.get("uses", "")).startswith(action)+++def job_needs(config):+ needs = config.get("needs") or []+ return [needs] if isinstance(needs, str) else list(needs)+++def check_skip_conditions(workflow, required_checks):+ """Return the problems with rust.yml's skip conditions and artifact steps."""+ errors = []+ jobs = {+ key: config+ for key, config in (workflow.get("jobs") or {}).items()+ if isinstance(config, dict)+ }+ listed = frozenset().union(*(members for _, members, _ in SKIP_POLICIES))++ for job in sorted(listed - set(jobs)):+ errors.append(+ f" - job `{job}` is listed in a skip set but does not exist "+ f"in {RUST_WORKFLOW}"+ )+ for job, config in sorted(jobs.items()):+ expected = expected_condition(job)+ found = config.get("if")+ if found == expected:+ continue+ if expected is None:+ errors.append(+ f" - job `{job}` is in no skip set, so it must carry no "+ f"job-level `if:`; found {found!r}. List the job in a skip "+ f"set or extend this check."+ )+ else:+ errors.append(+ f" - job `{job}` must carry exactly `if: {expected}`, "+ f"found {found!r}"+ )++ for set_name, members, _ in SKIP_POLICIES:+ for job, config in sorted(jobs.items()):+ if job in members:+ continue+ blocked = sorted(set(job_needs(config)) & members)+ if blocked:+ errors.append(+ f" - job `{job}` is not in {set_name} but needs "+ f"{blocked}; GitHub would skip it with them"+ )+ # Only the merge queue gates a job that skips after the merge. A job+ # that skips on a fork PR still runs on the fork, in the queue, and on+ # main.+ if required_checks and set_name == "POST_MERGE_SKIPPED_JOBS":+ for job in sorted(members & set(jobs)):+ name = jobs[job].get("name", job)+ if name not in required_checks:+ errors.append(+ f" - job `{job}` ({name!r}) is in {set_name} but is "+ f"not a required status check in .asf.yaml, so no "+ f"run would gate it before it lands on main"+ )++ build = jobs.get(ARTIFACT_JOB)+ if build is None:+ errors.append(f" - job `{ARTIFACT_JOB}` does not exist in {RUST_WORKFLOW}")+ else:+ steps = list(iter_steps(build))+ archive = [s for s in steps if s.get("id") == ARTIFACT_ARCHIVE_STEP]+ uploads = [+ s+ for s in steps+ if uses_action(s, "actions/upload-artifact")+ and (s.get("with") or {}).get("name") == ARTIFACT_NAME+ ]+ if len(archive) != 1 or archive[0].get("if") != ARTIFACT_ARCHIVE_CONDITION:+ found = [s.get("if") for s in archive]+ errors.append(+ f" - the step `id: {ARTIFACT_ARCHIVE_STEP}` in `{ARTIFACT_JOB}` "+ f"must exist once and carry exactly "+ f"`if: {ARTIFACT_ARCHIVE_CONDITION}`, found {found!r}"+ )+ if len(uploads) != 1 or uploads[0].get("if") != ARTIFACT_UPLOAD_CONDITION:+ found = [s.get("if") for s in uploads]+ errors.append(+ f" - the `{ARTIFACT_NAME}` upload step in `{ARTIFACT_JOB}` must "+ f"exist once and carry exactly `if: {ARTIFACT_UPLOAD_CONDITION}`, "+ f"found {found!r}"+ )++ # The artifact is missing on both events, so every consumer must skip on both.+ consumer_sets = [+ (name, members)+ for name, members, _ in SKIP_POLICIES+ if name == "POST_MERGE_SKIPPED_JOBS" or ARTIFACT_JOB in members+ ]+ for job, config in sorted(jobs.items()):+ downloads = [+ s+ for s in iter_steps(config)+ if uses_action(s, "actions/download-artifact")+ and (s.get("with") or {}).get("name") == ARTIFACT_NAME+ ]+ if not downloads:+ continue+ for set_name, members in consumer_sets:+ if job not in members:+ errors.append(+ f" - job `{job}` downloads the `{ARTIFACT_NAME}` artifact "+ f"but is not in {set_name}, where its producer skips"+ )+ return errors+++def main():+ repo_root = os.path.dirname(+ os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+ )+ with open(os.path.join(repo_root, ".github", "workflows", RUST_WORKFLOW)) as f:+ workflow = yaml.safe_load(f)+ required_checks = get_required_checks(os.path.join(repo_root, ".asf.yaml"))++ errors = check_skip_conditions(workflow, required_checks)+ if errors:+ print(f"ERROR: Problems found with skip conditions in {RUST_WORKFLOW}:\n")+ print("\n".join(errors))+ print()+ sys.exit(1)++ print(+ f"OK: {RUST_WORKFLOW} skips exactly {len(POST_MERGE_SKIPPED_JOBS)} jobs "+ f"on pushes to main and {len(FORK_PR_SKIPPED_JOBS)} jobs on pull "+ "requests from forks."+ )+++if __name__ == "__main__":+ main()diff --git a/ci/scripts/test_fork_ci_status.cjs b/ci/scripts/test_fork_ci_status.cjsnew file mode 100755index 0000000000..46e55c1776--- /dev/null+++ b/ci/scripts/test_fork_ci_status.cjs@@ -0,0 +1,373 @@+#!/usr/bin/env node+// Licensed to the Apache Software Foundation (ASF) under one+// or more contributor license agreements. See the NOTICE file+// distributed with this work for additional information+// regarding copyright ownership. The ASF licenses this file+// to you under the Apache License, Version 2.0 (the+// "License"); you may not use this file except in compliance+// with the License. You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing,+// software distributed under the License is distributed on an+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+// KIND, either express or implied. See the License for the+// specific language governing permissions and limitations+// under the License.++// Test fork CI workflows with stubbed GitHub APIs. No API calls are made.+const assert = require("node:assert/strict");+const { execFileSync } = require("node:child_process");+const { readFileSync } = require("node:fs");+const path = require("node:path");+const { test } = require("node:test");++function loadWorkflow(name) {+ return JSON.parse(+ execFileSync(+ "python3",+ [+ "-c",+ "import json, sys, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))",+ ],+ {+ input: readFileSync(+ path.join(__dirname, "../../.github/workflows", name)+ ),+ encoding: "utf8",+ }+ )+ );+}+const script =+ loadWorkflow("fork_ci_status.yml").jobs.bridge.steps[0].with.script;+const rustWorkflow = loadWorkflow("rust.yml");+const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor;+const bridge = new AsyncFunction("github", "context", "core", "Date", script);+const now = Date.UTC(2026, 0, 1);+class Clock extends Date {+ static now() {+ return now;+ }+}+const sha = "a".repeat(40);+const pr = {+ number: 1,+ updated_at: new Date(now).toISOString(),+ head: {+ sha,+ repo: { owner: { login: "contributor" }, name: "datafusion", fork: true },+ },+ base: { ref: "main" },+};+const run = (id, event = "push", conclusion = "success", run_attempt = 1) => ({+ id,+ event,+ conclusion,+ run_attempt,+ status: "completed",+ html_url: `https://github.com/contributor/datafusion/actions/runs/${id}`,+});+const checkFor = (run) => ({+ id: 10,+ status: run.status,+ conclusion: run.conclusion,+ output: {+ text: JSON.stringify({+ owner: "contributor",+ repo: "datafusion",+ head_sha: sha,+ run_id: run.id,+ run_attempt: run.run_attempt,+ }),+ },+});++async function execute({+ check,+ runs = [],+ recordedRun,+ event = "schedule",+ workflowState = "active",+ recentPrs = [pr],+ slice = [pr],+ calls = [],+}) {+ const writes = [];+ const github = {+ rest: {+ checks: {+ listForRef: async () => {+ calls.push("checks.listForRef");+ return { data: { check_runs: check ? [check] : [] } };+ },+ create: async (payload) =>+ writes.push({ method: "create", ...payload }),+ update: async (payload) =>+ writes.push({ method: "update", ...payload }),+ },+ actions: {+ listWorkflowRuns: async (params) => {+ calls.push("actions.listWorkflowRuns");+ assert.equal(params.owner, "contributor");+ assert.equal(params.repo, "datafusion");+ assert.equal(params.workflow_id, "rust.yml");+ assert.equal(params.head_sha, sha);+ return { data: { workflow_runs: runs } };+ },+ getWorkflowRun: async () => {+ if (!recordedRun) throw new Error("Not Found");+ return { data: recordedRun };+ },+ getWorkflow: async () => ({ data: { state: workflowState } }),+ },+ pulls: {+ list: async ({ per_page }) => {+ if (per_page === 1) return { data: [pr], headers: {} };+ assert.equal(per_page, 25);+ return { data: slice };+ },+ },+ },+ paginate: {+ iterator: async function* () {+ yield { data: recentPrs };+ },+ },+ };+ await bridge(+ github,+ {+ repo: { owner: "apache", repo: "datafusion" },+ eventName: event,+ payload: { pull_request: pr },+ },+ { info: () => {} },+ Clock+ );+ return writes;+}++function assertRun(write, method, expected) {+ assert.equal(write.method, method);+ assert.equal(write.status, expected.status);+ assert.equal(write.conclusion, expected.conclusion);+ assert.equal(write.details_url, expected.html_url);+ assert.equal(JSON.parse(write.output.text).run_id, expected.id);+ assert.equal(JSON.parse(write.output.text).run_attempt, expected.run_attempt);+}++test("notification selects a newer manual run instead of a push run", async () => {+ const latest = run(2, "workflow_dispatch");+ const writes = await execute({+ event: "pull_request_target",+ runs: [latest, run(1)],+ });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "create", latest);+});++for (const conclusion of ["cancelled", "failure"]) {+ test(`sweep replaces a ${conclusion} push run with a newer manual run`, async () => {+ const previous = run(1, "push", conclusion);+ const latest = run(2, "workflow_dispatch");+ const writes = await execute({+ check: checkFor(previous),+ recordedRun: previous,+ runs: [latest, previous],+ });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+ });+}++test("sweep recovers when the recorded run was deleted", async () => {+ const latest = run(2, "workflow_dispatch");+ const writes = await execute({ check: checkFor(run(1)), runs: [latest] });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+});++test("sweep creates a missing check from the PR head", async () => {+ const latest = run(2);+ const writes = await execute({ runs: [latest] });+ assert.equal(writes.length, 1);+ assert.equal(writes[0].head_sha, sha);+ assertRun(writes[0], "create", latest);+});++for (const [state, conclusion] of [+ ["active", "skipped"],+ ["disabled_fork", "action_required"],+]) {+ test(`sweep creates a missing check when the workflow is ${state} and has no run`, async () => {+ const writes = await execute({ workflowState: state });+ assert.equal(writes.length, 1);+ assert.equal(writes[0].method, "create");+ assert.equal(writes[0].head_sha, sha);+ assert.equal(writes[0].conclusion, conclusion);+ assert.equal(JSON.parse(writes[0].output.text).run_id, null);+ });+}++test("sweep picks up a rerun of the same workflow run", async () => {+ const previous = run(1, "push", "failure");+ const latest = run(1, "push", "success", 2);+ const writes = await execute({+ check: checkFor(previous),+ recordedRun: latest,+ runs: [latest],+ });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+});++test("sweep does not rewrite an unchanged check", async () => {+ const latest = run(1);+ assert.deepEqual(+ await execute({+ check: checkFor(latest),+ recordedRun: latest,+ runs: [latest],+ }),+ []+ );+});++for (const conclusion of ["success", "failure", "cancelled"]) {+ test(`recent completed ${conclusion} checks wait for their sweep slice`, async () => {+ const calls = [];+ const previous = run(1, "push", conclusion);+ const writes = await execute({+ check: checkFor(previous),+ runs: [run(2, "workflow_dispatch")],+ slice: [],+ calls,+ });+ assert.deepEqual(writes, []);+ assert.deepEqual(calls, ["checks.listForRef"]);+ });+}++test("the sweep finds a rerun on a recently updated completed PR", async () => {+ const calls = [];+ const latest = run(2, "workflow_dispatch");+ const writes = await execute({+ check: checkFor(run(1)),+ runs: [latest],+ calls,+ });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+ assert.deepEqual(calls, ["checks.listForRef", "actions.listWorkflowRuns"]);+});++test("the sweep still checks completed runs on older PRs", async () => {+ const oldPr = {+ ...pr,+ updated_at: new Date(now - 24 * 3600 * 1000).toISOString(),+ };+ const latest = run(2, "workflow_dispatch");+ const writes = await execute({+ check: checkFor(run(1)),+ runs: [latest],+ recentPrs: [oldPr],+ slice: [oldPr],+ });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+});++for (const status of ["queued", "in_progress"]) {+ test(`recent ${status} runs are polled outside the sweep slice`, async () => {+ const previous = { ...run(1), status, conclusion: null };+ const latest = run(1);+ const writes = await execute({+ check: checkFor(previous),+ runs: [latest],+ slice: [],+ });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+ });+}++test("recent checks with no fork run keep looking outside the sweep slice", async () => {+ const missing = checkFor(run(null, "push", "skipped"));+ const latest = run(2);+ const writes = await execute({ check: missing, runs: [latest], slice: [] });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "update", latest);+});++test("recent missing checks are recovered outside the sweep slice", async () => {+ const latest = run(2);+ const writes = await execute({ runs: [latest], slice: [] });+ assert.equal(writes.length, 1);+ assertRun(writes[0], "create", latest);+});++test("a recent active PR in the sweep slice is only polled once", async () => {+ const calls = [];+ const active = { ...run(1), status: "in_progress", conclusion: null };+ await execute({ check: checkFor(active), runs: [active], calls });+ assert.deepEqual(calls, ["checks.listForRef", "actions.listWorkflowRuns"]);+});++// The concurrency expressions use string lookups and &&/||, whose semantics+// match JavaScript for these non-empty strings and booleans.+function concurrencyGroup(github) {+ return rustWorkflow.concurrency.group.replace(+ /\$\{\{\s*(.*?)\s*\}\}/g,+ (_, expression) => new Function("github", `return (${expression});`)(github)+ );+}+const forkPush = {+ repository: "contributor/datafusion",+ head_ref: "",+ sha: "first",+ ref: "refs/heads/feature",+ workflow: "Rust",+};++test("successive pushes to a fork branch cancel the previous run", () => {+ assert.equal(rustWorkflow.concurrency["cancel-in-progress"], true);+ assert.equal(+ concurrencyGroup(forkPush),+ concurrencyGroup({ ...forkPush, sha: "second" })+ );+});++test("different fork branches have separate concurrency groups", () => {+ assert.notEqual(+ concurrencyGroup(forkPush),+ concurrencyGroup({ ...forkPush, ref: "refs/heads/other" })+ );+});++for (const ref of [+ "refs/heads/main",+ "refs/heads/gh-readonly-queue/main/pr-1",+]) {+ test(`upstream ${ref} runs keep separate groups for each SHA`, () => {+ const upstream = { ...forkPush, repository: "apache/datafusion", ref };+ assert.notEqual(+ concurrencyGroup(upstream),+ concurrencyGroup({ ...upstream, sha: "second" })+ );+ });+}++test("upstream PR updates still share a concurrency group", () => {+ const upstream = {+ ...forkPush,+ repository: "apache/datafusion",+ head_ref: "feature",+ ref: "refs/pull/1/merge",+ };+ assert.equal(+ concurrencyGroup(upstream),+ concurrencyGroup({ ...upstream, sha: "second" })+ );+});diff --git a/ci/scripts/test_rust_workflow_skips.py b/ci/scripts/test_rust_workflow_skips.pynew file mode 100755index 0000000000..2170439d98--- /dev/null+++ b/ci/scripts/test_rust_workflow_skips.py@@ -0,0 +1,71 @@+#!/usr/bin/env python3+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++"""Regression tests for the Rust workflow skip policies."""++import unittest+from pathlib import Path++import yaml++from check_asf_yaml_status_checks import get_required_checks+from check_rust_workflow_skips import check_skip_conditions++ROOT = Path(__file__).resolve().parents[2]+++class RustWorkflowSkipsTest(unittest.TestCase):+ def setUp(self):+ self.workflow = yaml.safe_load((ROOT / ".github/workflows/rust.yml").read_text())+ self.required_checks = get_required_checks(ROOT / ".asf.yaml")++ def errors(self):+ return check_skip_conditions(self.workflow, self.required_checks)++ def test_current_workflow(self):+ self.assertEqual(self.errors(), [])++ def test_missing_skip_guard(self):+ del self.workflow["jobs"]["linux-test"]["if"]+ self.assertTrue(+ any("job `linux-test` must carry exactly" in error for error in self.errors())+ )++ def test_unskipped_job_needs_skipped_job(self):+ self.workflow["jobs"]["check-fmt"]["needs"] = "linux-test"+ self.assertIn(+ " - job `check-fmt` is not in FORK_PR_SKIPPED_JOBS but needs "+ "['linux-test']; GitHub would skip it with them",+ self.errors(),+ )++ def test_missing_artifact_upload_guard(self):+ upload = next(+ step+ for step in self.workflow["jobs"]["linux-build-lib"]["steps"]+ if step.get("uses", "").startswith("actions/upload-artifact")+ and step.get("with", {}).get("name") == "cargo-check"+ )+ del upload["if"]+ self.assertTrue(+ any("`cargo-check` upload step" in error for error in self.errors())+ )+++if __name__ == "__main__":+ unittest.main()diff --git a/dev/rust_lint.sh b/dev/rust_lint.shindex dbb6454302..b1dddaac3b 100755--- a/dev/rust_lint.sh+++ b/dev/rust_lint.sh@@ -88,10 +88,10 @@ doneSCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-# `ci/scripts/check_asf_yaml_status_checks.py` runs with `python3` from PATH-# and imports PyYAML. Report a missing prerequisite before any tool is-# installed or any formatter runs, and point at `uv run`, which sets up the-# Python dependencies from the uv workspace.+# `ci/scripts/check_asf_yaml_status_checks.py` and `check_rust_workflow_skips.py`+# run with `python3` from PATH and import PyYAML. Report a missing prerequisite+# before any tool is installed or any formatter runs, and point at `uv run`,+# which sets up the Python dependencies from the uv workspace.ensure_python_with_yaml() {if ! command -v python3 &> /dev/null; thenecho "[${SCRIPT_NAME}] python3 was not found on PATH. Please run the suite through uv, which provides Python and its packages: uv run ./dev/rust_lint.sh" >&2@@ -136,6 +136,9 @@ declare -a WRITE_STEPS=(declare -a READONLY_STEPS=("ci/scripts/check_no_cargo_install_in_workflows.sh|false""ci/scripts/check_asf_yaml_status_checks.py|false"+ "ci/scripts/check_rust_workflow_skips.py|false"+ "ci/scripts/test_rust_workflow_skips.py|false"+ "ci/scripts/test_fork_ci_status.cjs|false""ci/scripts/check_large_files.sh|false""ci/scripts/markdown_link_check.sh|false""ci/scripts/security_audit.sh|false"diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.mdindex 81c432013a..2efd13921d 100644--- a/docs/source/contributor-guide/index.md+++ b/docs/source/contributor-guide/index.md@@ -239,3 +239,26 @@ According to Runs-On, spot instance termination is extremely rare for instances### GitHub RunnersWe also use standard GitHub runners for some actions in the main repository; these are also runnable in forks.++#### Pull Requests from Forks++DataFusion and its subprojects use substantial CI runner time. ASF Infrastructure+identified them among its largest GitHub Actions users in+[#25148](https://github.com/apache/datafusion/issues/25148). Fork CI reduces+duplicate work on ASF's shared runners.++A pull request from a fork to `main` runs the Rust workflow on your fork, with+your own GitHub Actions runners. The main repository runs only the fast lint jobs for+such a pull request and adds a `Rust (fork)` check that links to the run on+your fork. The merge queue runs the full workflow on the merged result before+a pull request merges, so a green fork run does not replace it.++To make this work:++1. Before you push your branch, open the **Actions** tab on your fork.+ Confirm the prompt to enable workflows.+2. Keep the `main` branch of your fork in sync with upstream. Runs on your+ branches restore the build caches that a run on your `main` saves, so a+ synced fork builds much faster.+3. Rebase your branch before you push. The fork run tests your branch as it+ is, while the merge queue tests it merged with `main`.diff --git a/docs/source/contributor-guide/testing.md b/docs/source/contributor-guide/testing.mdindex 3e39ef2eb1..f4b5c1cbc0 100644--- a/docs/source/contributor-guide/testing.md+++ b/docs/source/contributor-guide/testing.md@@ -163,6 +163,22 @@ For local SQLite test instructions, see the [instructions in the documentation].[instructions in the documentation]: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest#running-tests-sqlite[extended.yml]: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml+## CI Workflow Tests++The Rust workflow validator and fork status bridge have regression tests.+These tests cover skip conditions, fork run recovery, polling frequency, and+concurrency groups for fork pushes and merge queue runs.+They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`.+Run them locally with Python, PyYAML, and Node.js 24:++```shell+python3 ci/scripts/test_rust_workflow_skips.py+node --test ci/scripts/test_fork_ci_status.cjs+```++The validator tests remove guards and add an invalid job dependency in memory.+The bridge tests use stubbed GitHub API responses and make no network requests.+## Rust Integration TestsThere are several public interface tests for the DataFusion library in the [tests](https://github.com/apache/datafusion/tree/main/datafusion/core/tests) directory.@@ -261,18 +277,21 @@ bash ci/scripts/rust_docs.sh## ASF Status Check Validation`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status-check in `.asf.yaml` matches a job in `.github/workflows`, and that `rust.yml`-skips only its listed jobs on pushes to `main`. `./dev/rust_lint.sh` runs it-and needs `python3` with [PyYAML]. The [uv] workspace provides both:+check in `.asf.yaml` matches a job in `.github/workflows`.+`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its+listed jobs on pushes to `main` and on pull requests from forks.+`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv]+workspace provides both:```shelluv run ./dev/rust_lint.sh```-To run the check on its own:+To run the checks on their own:```shelluv run python3 ci/scripts/check_asf_yaml_status_checks.py+uv run python3 ci/scripts/check_rust_workflow_skips.py```[pyyaml]: https://pypi.org/project/PyYAML/