Apache DataFusion SQL Query Engine
chore: add internal markdown link check (#21831)
Geethapranay126762977499c
diff --git a/.asf.yaml b/.asf.yamlindex 73adb1c058..d953f9c585 100644--- a/.asf.yaml+++ b/.asf.yaml@@ -55,6 +55,7 @@ github:contexts:- "Check License Header"- "Use prettier to check formatting of documents"+ - "Check Markdown Links"- "Validate required_status_checks in .asf.yaml"- "Spell Check with Typos"# needs to be updated as part of the release processdiff --git a/.github/workflows/dev.yml b/.github/workflows/dev.ymlindex a247f07333..55e8572408 100644--- a/.github/workflows/dev.yml+++ b/.github/workflows/dev.yml@@ -23,6 +23,9 @@ on:pull_request:merge_group:+permissions:+ contents: read+concurrency:group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}cancel-in-progress: true@@ -51,6 +54,22 @@ jobs:# if you encounter error, see instructions inside the scriptrun: ci/scripts/doc_prettier_check.sh+ markdown-link-check:+ name: Check Markdown Links+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2+ - name: Load tool versions+ run: |+ source ci/scripts/utils/tool_versions.sh+ echo "LYCHEE_VERSION=${LYCHEE_VERSION}" >> "$GITHUB_ENV"+ - name: Install lychee+ uses: taiki-e/install-action@055f5df8c3f65ea01cd41e9dc855becd88953486 # v2.75.18+ with:+ tool: lychee@${{ env.LYCHEE_VERSION }}+ - name: Run markdown link check+ run: bash ci/scripts/markdown_link_check.sh+asf-yaml-check:name: Validate required_status_checks in .asf.yamlruns-on: ubuntu-latestdiff --git a/ci/scripts/markdown_link_check.sh b/ci/scripts/markdown_link_check.shnew file mode 100644index 0000000000..65fede4111--- /dev/null+++ b/ci/scripts/markdown_link_check.sh@@ -0,0 +1,33 @@+#!/usr/bin/env bash+#+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++set -euo pipefail++ROOT_DIR="$(git rev-parse --show-toplevel)"++cd "${ROOT_DIR}"++MARKDOWN_FILES=()+while IFS= read -r file; do+ MARKDOWN_FILES+=("${file}")+done < <(+ git -C "${ROOT_DIR}" ls-files 'README.md' 'CONTRIBUTING.md' 'docs/**/*.md' 'datafusion-cli/README.md' 'datafusion-examples/README.md' 'dev/**/*.md'+)++lychee --no-progress --config "${ROOT_DIR}/lychee.toml" "${MARKDOWN_FILES[@]}"diff --git a/ci/scripts/utils/tool_versions.sh b/ci/scripts/utils/tool_versions.shindex ac731ed0d5..237b18b62e 100644--- a/ci/scripts/utils/tool_versions.sh+++ b/ci/scripts/utils/tool_versions.sh@@ -21,3 +21,4 @@# It is intended to be sourced by other scripts and should not be executed directly.PRETTIER_VERSION="2.7.1"+LYCHEE_VERSION="0.23.0"diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.mdindex aac0710dad..bfaf398d3f 100644--- a/docs/source/contributor-guide/roadmap.md+++ b/docs/source/contributor-guide/roadmap.md@@ -19,7 +19,7 @@ under the License.# Roadmap and Improvement Proposals-The [project introduction](../user-guide/introduction) explains the+The [project introduction](../user-guide/introduction.md) explains theoverview and goals of DataFusion, and our development efforts largelyalign to that vision.diff --git a/docs/source/contributor-guide/testing.md b/docs/source/contributor-guide/testing.mdindex 6b8e4568ec..3b644f610b 100644--- a/docs/source/contributor-guide/testing.md+++ b/docs/source/contributor-guide/testing.md@@ -186,6 +186,34 @@ tested in the same way using the [doc_comment] crate. See the end of[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583+## Documentation Link Checks++Run the internal markdown link check locally:++```shell+source ci/scripts/utils/tool_versions.sh+cargo install lychee --locked --version "${LYCHEE_VERSION}"+bash ci/scripts/markdown_link_check.sh+```++Notes:++- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency).+- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures.++When a link is broken, lychee prints the file and URL/path that failed. For example:++```text+[docs/source/user-guide/cli/overview.md]:+ [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct+```++Rust doc comments are validated by rustdoc in CI and can be checked locally with:++```shell+bash ci/scripts/rust_docs.sh+```+## Benchmarks### Criterion Benchmarksdiff --git a/docs/source/library-user-guide/upgrading/49.0.0.md b/docs/source/library-user-guide/upgrading/49.0.0.mdindex 92dee81355..92267a80fa 100644--- a/docs/source/library-user-guide/upgrading/49.0.0.md+++ b/docs/source/library-user-guide/upgrading/49.0.0.md@@ -123,7 +123,7 @@ Or via SQL:SET datafusion.execution.spill_compression = 'zstd';```-For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../../user-guide/configs) documentation.+For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../../user-guide/configs.md) documentation.### Deprecated `map_varchar_to_utf8view` configuration optiondiff --git a/docs/source/user-guide/cli/overview.md b/docs/source/user-guide/cli/overview.mdindex 86beea0e82..e0228d3ea0 100644--- a/docs/source/user-guide/cli/overview.md+++ b/docs/source/user-guide/cli/overview.md@@ -41,5 +41,5 @@ DataFusion CLI v37.0.0Elapsed 1.969 seconds.```-For more information, see the [Installation](installation), [Usage Guide](usage)-and [Data Sources](datasources) sections.+For more information, see the [Installation](installation.md), [Usage Guide](usage.md)+and [Data Sources](datasources.md) sections.diff --git a/docs/source/user-guide/dataframe.md b/docs/source/user-guide/dataframe.mdindex 85724a7239..d35b543bfd 100644--- a/docs/source/user-guide/dataframe.md+++ b/docs/source/user-guide/dataframe.md@@ -122,4 +122,4 @@ async fn main() -> Result<()> {[`collect`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.collect[library users guide]: ../library-user-guide/using-the-dataframe-api.md[api reference on docs.rs]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html-[expressions reference]: expressions+[expressions reference]: expressions.mddiff --git a/docs/source/user-guide/sql/format_options.md b/docs/source/user-guide/sql/format_options.mdindex 3385080314..46d251c18e 100644--- a/docs/source/user-guide/sql/format_options.md+++ b/docs/source/user-guide/sql/format_options.md@@ -29,7 +29,7 @@ Format-related options can be specified in three ways, in decreasing order of pr- `COPY` option tuples- Session-level config defaults-For a list of supported session-level config defaults, see [Configuration Settings](../configs). These defaults apply to all operations but have the lowest level of precedence.+For a list of supported session-level config defaults, see [Configuration Settings](../configs.md). These defaults apply to all operations but have the lowest level of precedence.If creating an external table, table-specific format options can be specified when the table is created using the `OPTIONS` clause:diff --git a/lychee.toml b/lychee.tomlnew file mode 100644index 0000000000..74245ee7ec--- /dev/null+++ b/lychee.toml@@ -0,0 +1,32 @@+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++timeout = 20+max_retries = 2+retry_wait_time = 2++exclude_path = [+ "target",+ "docs/build",+ "datafusion/core/benches/tpch-csv",+]++exclude = [+ "^http://",+ "^https://",+ "^mailto:",+]