PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Add cpu & xpu nightly test (#2149)
Sun, Xuehao436dc247ad30
diff --git a/.azure-pipelines/nightly-test-cpu.yml b/.azure-pipelines/nightly-test-cpu.ymlnew file mode 100644index 000000000..e20ecc9c0--- /dev/null+++ b/.azure-pipelines/nightly-test-cpu.yml@@ -0,0 +1,121 @@+# Nightly pipeline: runs the slower CPU integration + e2e suites that are+# excluded from the fast PR unit-test pipeline. Scheduled daily; can also be+# triggered manually from the Azure DevOps UI.++trigger: none+pr:+ autoCancel: true+ drafts: false+ branches:+ include:+ - main+ paths:+ include:+ - test/integration/test_cpu+ - .azure-pipelines/nightly-test-cpu.yml+ - .azure-pipelines/scripts/ut/detect_changed_tests.sh++parameters:+ - name: unitTestParts+ type: object+ default:+ - 1+ - 2+ - 3+ - 4+ - 5+ - inc+ - llmc++schedules:+ - cron: "0 18 * * *" # 02:00 Asia/Shanghai every day+ displayName: Daily nightly run+ branches:+ include:+ - main+ always: true++pool: INTEL-CPU++variables:+ IMAGE_NAME: "auto-round"+ IMAGE_TAG: "py312"+ UPLOAD_PATH: $(Build.SourcesDirectory)/log_dir+ DOWNLOAD_PATH: $(Build.SourcesDirectory)/log_dir+ ARTIFACT_NAME: "Nightly_coverage_report"+ REPO: $(Build.Repository.Uri)++stages:+ - template: template/lib-build-template.yml+ parameters:+ enableChangeDetection: true++ - stage: Unit_test+ displayName: Unit Test+ dependsOn: [DetectArkChanges, BuildArkWheel]+ condition: and(succeeded('DetectArkChanges'), in(dependencies.BuildArkWheel.result, 'Succeeded', 'Skipped'))+ variables:+ BUILD_ARK_WHEEL: $[ stageDependencies.DetectArkChanges.Detect.outputs['SetArkWheelFlag.BUILD_ARK_WHEEL'] ]+ jobs:+ - job:+ timeoutInMinutes: 60+ strategy:+ matrix:+ ${{ each part in parameters.unitTestParts }}:+ part_${{ part }}:+ PART: ${{ part }}+ steps:+ - template: template/ut-template.yml+ parameters:+ dockerConfigName: "commonDockerConfig"+ utScriptFileName: "run_ut"+ uploadPath: $(UPLOAD_PATH)+ utArtifact: "ut-$(PART)"+ utTestMode: $(PART)+ utContainerName: "AutoRoundUnitTest$(NODE_LABEL)"+ buildARKWheel: $(BUILD_ARK_WHEEL)+ filterChangedTests: true++ - stage: Coverage+ displayName: "Collect Coverage"+ pool:+ vmImage: "ubuntu-latest"+ dependsOn: [Unit_test]+ jobs:+ - job: CollectDatafiles+ steps:+ - task: DownloadPipelineArtifact@2+ inputs:+ artifact:+ patterns: '*_coverage/.coverage.*'+ path: $(DOWNLOAD_PATH)++ - task: UsePythonVersion@0+ inputs:+ versionSpec: '3.12'+ displayName: 'Use Python 3.12'++ - script: |+ cd ${BUILD_SOURCESDIRECTORY}+ pip install -U pip setuptools uv+ uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu+ uv pip install .+ pip list+ cd ${BUILD_SOURCESDIRECTORY}/.azure-pipelines/scripts+ bash ut/collect_log.sh+ env:+ PYTHONUNBUFFERED: '1'+ UV_NO_PROGRESS: '1'+ UV_SYSTEM_PYTHON: '1'+ displayName: "Collect UT Coverage"++ - task: PublishPipelineArtifact@1+ condition: succeededOrFailed()+ inputs:+ targetPath: $(UPLOAD_PATH)/coverage_PR+ artifact: $(ARTIFACT_NAME)+ publishLocation: "pipeline"++ - task: PublishCodeCoverageResults@2+ inputs:+ summaryFileLocation: $(Build.SourcesDirectory)/log_dir/coverage_PR/coverage.xmldiff --git a/.azure-pipelines/nightly-test-xpu.yml b/.azure-pipelines/nightly-test-xpu.ymlindex fee2a8067..66896637d 100644--- a/.azure-pipelines/nightly-test-xpu.yml+++ b/.azure-pipelines/nightly-test-xpu.yml@@ -3,7 +3,24 @@# Scheduled daily; can also be triggered manually from the Azure DevOps UI.trigger: none-pr: none+pr:+ autoCancel: true+ drafts: false+ branches:+ include:+ - main+ paths:+ include:+ - test/integration/test_xpu+ - .azure-pipelines/nightly-test-xpu.yml+ - .azure-pipelines/scripts/ut/detect_changed_tests.sh++parameters:+ - name: xpuTestParts+ type: object+ default:+ - nightly+ - llmcschedules:- cron: "0 19 * * *" # 03:00 Asia/Shanghai every day@@ -28,7 +45,7 @@ variables:stages:- template: template/lib-build-template.ymlparameters:- enableChangeDetection: false+ enableChangeDetection: truepublishToTestPyPI: false- stage: Nightly_xpu_test@@ -39,19 +56,24 @@ stages:- job:displayName: Nightly XPU TesttimeoutInMinutes: 180+ strategy:+ matrix:+ ${{ each part in parameters.xpuTestParts }}:+ part_${{ part }}:+ PART: ${{ part }}steps:- template: template/ut-template.ymlparameters:dockerConfigName: "commonDockerConfig"utScriptFileName: "run_ut_xpu"- # "integration" enables the XPU LLMC integration suite in run_ut_xpu.sh- utTestMode: "integration"+ utTestMode: $(PART)uploadPath: $(UPLOAD_PATH)- utArtifact: "nightly-xpu"+ utArtifact: "$(PART)-xpu"imageTag: $(IMAGE_TAG)dockerFileName: $(DOCKERFILE_NAME)utContainerName: "AutoRoundNightlyXPU$(NODE_LABEL)"buildARKWheel: "false"+ filterChangedTests: true- task: UseDotNet@2displayName: 'Use .NET Core sdk 7.0.x'@@ -59,6 +81,6 @@ stages:version: 7.0.x- task: PublishCodeCoverageResults@2- condition: succeededOrFailed()+ condition: and(succeededOrFailed(), ne(variables['HAS_COVERAGE'], 'false'))inputs:summaryFileLocation: $(UPLOAD_PATH)/coverage.xmldiff --git a/.azure-pipelines/nightly-test.yml b/.azure-pipelines/nightly-test.ymldeleted file mode 100644index a4cbef256..000000000--- a/.azure-pipelines/nightly-test.yml+++ /dev/null@@ -1,51 +0,0 @@-# Nightly pipeline: runs the slower CPU integration + e2e suites that are-# excluded from the fast PR unit-test pipeline. Scheduled daily; can also be-# triggered manually from the Azure DevOps UI.--trigger: none-pr: none--schedules:- - cron: "0 18 * * *" # 02:00 Asia/Shanghai every day- displayName: Daily nightly run- branches:- include:- - main- always: true--pool: ICX-16C--variables:- IMAGE_NAME: "auto-round"- IMAGE_TAG: "py312"- UPLOAD_PATH: $(Build.SourcesDirectory)/log_dir- DOWNLOAD_PATH: $(Build.SourcesDirectory)/log_dir- ARTIFACT_NAME: "Nightly_coverage_report"- REPO: $(Build.Repository.Uri)--stages:- - template: template/lib-build-template.yml- parameters:- enableChangeDetection: false-- - stage: Nightly_test- displayName: Nightly Integration + E2E Test- dependsOn: [BuildArkWheel]- condition: in(dependencies.BuildArkWheel.result, 'Succeeded', 'Skipped')- jobs:- - job:- timeoutInMinutes: 180- steps:- - template: template/ut-template.yml- parameters:- dockerConfigName: "commonDockerConfig"- utScriptFileName: "run_nightly"- uploadPath: $(UPLOAD_PATH)- utArtifact: "nightly"- utContainerName: "AutoRoundNightly$(NODE_LABEL)"- buildARKWheel: "false"-- - task: PublishCodeCoverageResults@2- condition: succeededOrFailed()- inputs:- summaryFileLocation: $(UPLOAD_PATH)/coverage.xmldiff --git a/.azure-pipelines/scripts/cuda_unit_test/run_cuda_ut.sh b/.azure-pipelines/scripts/cuda_unit_test/run_cuda_ut.shindex 9125451a1..5c3579656 100644--- a/.azure-pipelines/scripts/cuda_unit_test/run_cuda_ut.sh+++ b/.azure-pipelines/scripts/cuda_unit_test/run_cuda_ut.sh@@ -19,10 +19,17 @@ for i in "$@"; dodonesource ${BUILD_SOURCESDIRECTORY}/.azure-pipelines/scripts/change_color.sh+# Change-based test selection helpers. REPO_DIR points the detector at the+# agent checkout instead of the container default (/auto-round).+REPO_DIR="${BUILD_SOURCESDIRECTORY}"+source ${BUILD_SOURCESDIRECTORY}/.azure-pipelines/scripts/ut/detect_changed_tests.shLOG_DIR="${BUILD_SOURCESDIRECTORY}/log_dir"mkdir -p "${LOG_DIR}"SUMMARY_LOG="${LOG_DIR}/results_summary.log"+# print_summary reads this file unconditionally; a matrix part that selects no+# test never writes it, so make sure it always exists.+touch "${SUMMARY_LOG}"function setup_environment() {export TZ='Asia/Shanghai'@@ -82,6 +89,12 @@ function run_unit_test() {fiend_line=$(( start_line + chunk_size - 1 ))selected_files=$(sed -n "${start_line},${end_line}p" all_tests.txt)+ selected_files=$(filter_changed_tests "test" "${selected_files}")++ if [ -z "${selected_files}" ]; then+ echo "No changed CUDA unit test file in part ${test_part}, skip."+ return 0+ fifor test_file in ${selected_files}; doecho "##[group]Running ${test_file}..."@@ -203,11 +216,15 @@ function run_unit_test_vllm() {function main() {setup_environment+ init_changed_testsif [ "${test_case}" == "nightly" ]; thenrun_unit_test_sglangrun_unit_test_llmcrun_unit_test_vllmelif [ "${test_case}" == "ci" ]; then+ # Mirror the selection below: tests excluded from the ci run (vlms,+ # llmc, sglang, vllm, multiple_card) must not enable filtering.+ scope_changed_tests "$(cd "${BUILD_SOURCESDIRECTORY}" && find test/unit/test_cuda -type f -name "test_*.py" | grep -Ev "vlms|llmc|sglang|vllm|multiple_card")"run_unit_testelseecho "##[error]Invalid test case specified: ${test_case}. Please use 'nightly' or 'ci'."diff --git a/.azure-pipelines/scripts/ut/collect_log.sh b/.azure-pipelines/scripts/ut/collect_log.shindex d6a7828d5..5113492ab 100644--- a/.azure-pipelines/scripts/ut/collect_log.sh+++ b/.azure-pipelines/scripts/ut/collect_log.sh@@ -7,6 +7,12 @@ cd "${BUILD_SOURCESDIRECTORY}/log_dir"echo "collect coverage for PR branch"mkdir -p coverage_PR+# Every matrix part may legitimately select no test (change-based filtering),+# in which case there is nothing to combine.+if ! compgen -G "ut-*/.coverage.*" > /dev/null; then+ echo "no coverage data found, skip coverage collection."+ exit 0+ficp ut-*/.coverage.* ./coverage_PR/cd coverage_PRcoverage combine --keep --rcfile=${COVERAGE_RCFILE}diff --git a/.azure-pipelines/scripts/ut/detect_changed_tests.sh b/.azure-pipelines/scripts/ut/detect_changed_tests.shnew file mode 100644index 000000000..f4124f635--- /dev/null+++ b/.azure-pipelines/scripts/ut/detect_changed_tests.sh@@ -0,0 +1,167 @@+#!/bin/bash+# Detect the test files modified by the current change set.+#+# Usage:+# source detect_changed_tests.sh+# detect_changed_tests # prints repo-relative test files, one per line+# returns 0 -> filtering applies (a change set was detected)+# returns 1 -> filtering does NOT apply, caller should run the full suite+#+# Detection order:+# 1. Filtering is opt-in: unless FILTER_CHANGED_TESTS is truthy the full suite+# runs. This keeps pipelines that do not request it running everything.+# 2. Only PR builds are filtered. When BUILD_REASON is set and is not+# "PullRequest" (e.g. a scheduled or manual run) the full suite runs.+# 3. Diff the working tree against BASE_REF (default origin/main, or the PR+# target branch when available). Filtering only kicks in when the change+# set is EXCLUSIVELY test files (test/**/test_*.py); if any other file+# changed (source, helpers, conftest, deletions, ...) the full suite runs.+#+# Only existing files matching test/**/test_*.py are reported; a deleted test+# file counts as a non-test change so the full suite runs.++REPO_DIR="${REPO_DIR:-/auto-round}"+# Prefer the PR target branch reported by Azure DevOps; fall back to origin/main.+_pr_target="${SYSTEM_PULLREQUEST_TARGETBRANCH#refs/heads/}"+BASE_REF="${BASE_REF:-origin/${_pr_target:-main}}"++detect_changed_tests() {+ # Filtering is opt-in; pipelines that do not request it run everything.+ case "${FILTER_CHANGED_TESTS:-}" in+ 1 | true | True | TRUE) ;;+ *)+ echo "FILTER_CHANGED_TESTS not enabled, run full suite." >&2+ return 1+ ;;+ esac++ # Only PR builds are filtered; scheduled/manual runs execute everything.+ if [ -n "${BUILD_REASON:-}" ] && [ "${BUILD_REASON}" != "PullRequest" ]; then+ echo "BUILD_REASON=${BUILD_REASON} is not a PR, run full suite." >&2+ return 1+ fi++ if ! command -v git > /dev/null 2>&1; then+ echo "git not available, run full suite." >&2+ return 1+ fi++ cd "${REPO_DIR}" 2>/dev/null || return 1+ git config --global --add safe.directory "${REPO_DIR}" > /dev/null 2>&1 || true++ # Best-effort fetch so BASE_REF resolves; ignore failures (offline runners).+ # CI checkouts are often shallow, which breaks merge-base; unshallow first.+ local base_branch="${BASE_REF#origin/}"+ if [ "$(git rev-parse --is-shallow-repository 2>/dev/null)" = "true" ]; then+ git fetch --quiet --unshallow origin > /dev/null 2>&1 || true+ fi+ git fetch --quiet origin "+refs/heads/${base_branch}:refs/remotes/origin/${base_branch}" > /dev/null 2>&1 || true++ local base_sha+ base_sha="$(git merge-base "${BASE_REF}" HEAD 2>/dev/null)" || base_sha=""+ if [ -z "${base_sha}" ]; then+ echo "cannot resolve base ref '${BASE_REF}', run full suite." >&2+ return 1+ fi++ local all_changed test_changed non_test+ # NOTE: no --diff-filter here. Deletions must stay visible, otherwise a PR+ # that removes a source file and touches a test would look test-only.+ all_changed="$(git diff --name-only "${base_sha}" HEAD 2>/dev/null)"+ if [ -z "${all_changed}" ]; then+ echo "no changed file detected, run full suite." >&2+ return 1+ fi++ # Added/copied/modified/renamed only: a deleted test cannot be executed.+ test_changed="$(git diff --name-only --diff-filter=ACMR "${base_sha}" HEAD -- 'test/**/test_*.py' 2>/dev/null)"+ if [ -z "${test_changed}" ]; then+ echo "no changed test file detected, run full suite." >&2+ return 1+ fi++ # Only filter when the change set is exclusively test files. Any other file+ # (source, test helpers, conftest, requirements, ...) may affect many tests,+ # so fall back to the full suite.+ non_test="$(comm -23 <(printf '%s\n' "${all_changed}" | sort -u) \+ <(printf '%s\n' "${test_changed}" | sort -u))"+ if [ -n "${non_test}" ]; then+ echo "non-test files changed, run full suite:" >&2+ printf ' %s\n' ${non_test} >&2+ return 1+ fi++ printf '%s\n' "${test_changed}"+ return 0+}++# ---------------------------------------------------------------------------+# Shared helpers for the run_ut*.sh scripts.+# ---------------------------------------------------------------------------++# Populated by init_changed_tests(): whitespace separated repo-relative test+# files touched by this change set. FILTER_TESTS=1 means only run those files.+CHANGED_TEST_FILES=""+FILTER_TESTS=0++# Resolve the change set once and cache it for filter_changed_tests().+init_changed_tests() {+ local changed+ if changed=$(detect_changed_tests); then+ CHANGED_TEST_FILES=$(printf '%s' "${changed}" | tr '\n' ' ')+ FILTER_TESTS=1+ echo "Only running changed test files: ${CHANGED_TEST_FILES}"+ else+ FILTER_TESTS=0+ echo "Running the full test suite."+ fi+}++# Disable filtering when none of the changed tests can be executed by this+# runner. Without this every matrix part could end up with an empty selection+# and the job would produce no results, no summary and no coverage at all.+# $@: repo-relative paths of every test this runner is able to execute.+scope_changed_tests() {+ [ "${FILTER_TESTS}" = "1" ] || return 0++ local scope="$*"+ local changed candidate+ for changed in ${CHANGED_TEST_FILES}; do+ for candidate in ${scope}; do+ if [ "${changed}" = "${candidate#./}" ]; then+ return 0+ fi+ done+ done++ echo "Changed tests are outside this runner's scope, run the full test suite."+ FILTER_TESTS=0+}++# Keep only the entries of a candidate list that were modified by this PR.+# $1: repo-relative directory the candidates are relative to (e.g. "test")+# $2: newline separated candidate test files (e.g. "./unit/test_cpu/test_a.py")+filter_changed_tests() {+ local base_dir=$1+ local candidates=$2++ if [ "${FILTER_TESTS}" != "1" ]; then+ printf '%s' "${candidates}"+ return 0+ fi++ local kept=""+ local candidate normalized changed+ while IFS= read -r candidate; do+ [ -z "${candidate}" ] && continue+ normalized="${base_dir}/${candidate#./}"+ for changed in ${CHANGED_TEST_FILES}; do+ if [ "${changed#./}" = "${normalized}" ]; then+ kept+="${candidate}"$'\n'+ break+ fi+ done+ done <<< "${candidates}"++ printf '%s' "${kept}"+}diff --git a/.azure-pipelines/scripts/ut/run_ut.sh b/.azure-pipelines/scripts/ut/run_ut.shindex 14e4db8cf..b82151096 100644--- a/.azure-pipelines/scripts/ut/run_ut.sh+++ b/.azure-pipelines/scripts/ut/run_ut.sh@@ -1,14 +1,18 @@#!/bin/bashset -e-test_part=$1+test_part=${UT_MODE}source /auto-round/.azure-pipelines/scripts/change_color.sh+source /auto-round/.azure-pipelines/scripts/ut/detect_changed_tests.shLOG_DIR=/auto-round/log_dirmkdir -p "${LOG_DIR}"SUMMARY_LOG="${LOG_DIR}/results_summary.log"+TIMEOUT=30+SESSION_TIMEOUT=600+function setup_environment() {echo "##[group]set up UT env..."echo "NUMA_NODE=${NUMA_NODE}"@@ -72,6 +76,12 @@ function run_unit_test() {fiend_line=$(( start_line + chunk_size - 1 ))selected_files=$(sed -n "${start_line},${end_line}p" all_tests.txt)+ selected_files=$(filter_changed_tests "test" "${selected_files}")++ if [ -z "${selected_files}" ]; then+ echo "No changed unit test file in part ${test_part}, skip."+ return 0+ fifor test_file in ${selected_files}; doecho "##[group]Running ${test_file}..."@@ -79,7 +89,7 @@ function run_unit_test() {local ut_log_name=${LOG_DIR}/unittest_${test_basename}.lognumactl --physcpubind="${NUMA_CPUSET:-0-15}" --membind="${NUMA_NODE:-0}" \- pytest -m "not skip_ci" --timeout=30 --session-timeout=600 \+ pytest -m "not skip_ci" --timeout=${TIMEOUT} --session-timeout=${SESSION_TIMEOUT} \--cov=auto_round --cov-report= --cov-append \-vs --junitxml="${ut_log_name%.log}.xml" ${test_file} 2>&1 | tee ${ut_log_name}echo "##[endgroup]"@@ -87,26 +97,41 @@ function run_unit_test() {}function run_inc_unit_test() {+ local selected_files+ selected_files=$(filter_changed_tests "test/integration" \+ "$(cd /auto-round/test/integration && find ./test_cpu -name "test_inc*.py" | sort)")+ if [ -z "${selected_files}" ]; then+ echo "No changed INC test file, skip."+ return 0+ fi+echo "##[group]set up INC UT env..."INC_PT_ONLY=1 uv pip install -r /auto-round/test/integration/test_cpu/requirements_inc.txt --extra-index-url https://download.pytorch.org/whl/cpuecho "##[endgroup]"cd /auto-round/test/integration || exit 1- for test_file in $(find ./test_cpu -name "test_inc*.py" | sort); do+ for test_file in ${selected_files}; doecho "##[group]Running ${test_file}..."local test_basename=$(basename ${test_file} .py)local ut_log_name=${LOG_DIR}/unittest_${test_basename}.lognumactl --physcpubind="${NUMA_CPUSET:-0-15}" --membind="${NUMA_NODE:-0}" \- pytest --timeout=30 --session-timeout=600 \- --cov=auto_round --cov-report= --cov-append \+ pytest --cov=auto_round --cov-report= --cov-append \-vs --junitxml="${ut_log_name%.log}.xml" ${test_file} 2>&1 | tee ${ut_log_name}echo "##[endgroup]"done}function run_llmc_unit_test() {+ local selected_files+ selected_files=$(filter_changed_tests "test/integration" \+ "$(cd /auto-round/test/integration && find ./test_cpu -name "test_llmc*.py" | sort)")+ if [ -z "${selected_files}" ]; then+ echo "No changed LLMC test file, skip."+ return 0+ fi+echo "##[group]set up LLMC UT env..."BUILD_TYPE="nightly" uv pip install -r /auto-round/test/integration/test_cpu/requirements_llmc.txt --extra-index-url https://download.pytorch.org/whl/cpuuv pip uninstall auto-round@@ -115,36 +140,42 @@ function run_llmc_unit_test() {cd /auto-round/test/integration || exit 1- for test_file in $(find ./test_cpu -name "test_llmc*.py" | sort); do+ for test_file in ${selected_files}; doecho "##[group]Running ${test_file}..."local test_basename=$(basename ${test_file} .py)local ut_log_name=${LOG_DIR}/unittest_${test_basename}.lognumactl --physcpubind="${NUMA_CPUSET:-0-15}" --membind="${NUMA_NODE:-0}" \- pytest --timeout=30 --session-timeout=600 \- --cov=auto_round --cov-report= --cov-append \+ pytest --cov=auto_round --cov-report= --cov-append \-vs --junitxml="${ut_log_name%.log}.xml" ${test_file} 2>&1 | tee ${ut_log_name}echo "##[endgroup]"done}function collect_log() {+ touch "${SUMMARY_LOG}"python /auto-round/.azure-pipelines/scripts/ut/collect_result.py \--test-type "Unit Tests" --log-pattern "unittest_test_*.log" --log-dir ${LOG_DIR} --summary-log ${SUMMARY_LOG}- cp .coverage "${LOG_DIR}/.coverage.part${test_part}"+ if [ -f .coverage ]; then+ cp .coverage "${LOG_DIR}/.coverage.part${test_part}"+ fi}function main() {setup_environment- run_unit_test- if [ "$test_part" -eq 5 ] && [ "$NIGHTLY_TEST" = 1 ]; then+ init_changed_tests+ scope_changed_tests "$(cd /auto-round && find test/unit/test_cpu test/integration/test_cpu -name "test_*.py" 2>/dev/null)"+ if [ "$test_part" = "inc" ]; thenrun_inc_unit_test+ elif [ "$test_part" = "llmc" ]; thenrun_llmc_unit_test+ else+ run_unit_testficollect_logcheck_storage_usageprint_summary}-main+main "$@"diff --git a/.azure-pipelines/scripts/ut/run_ut_hpu.sh b/.azure-pipelines/scripts/ut/run_ut_hpu.shindex 77f88d346..7e821eb72 100644--- a/.azure-pipelines/scripts/ut/run_ut_hpu.sh+++ b/.azure-pipelines/scripts/ut/run_ut_hpu.sh@@ -1,6 +1,7 @@#!/bin/bashset -esource /auto-round/.azure-pipelines/scripts/change_color.sh+source /auto-round/.azure-pipelines/scripts/ut/detect_changed_tests.shfunction setup_environment() {# install requirements@@ -28,7 +29,14 @@ function setup_environment() {function run_unit_test() {auto_round_path=$(python -c 'import auto_round; print(auto_round.__path__[0])')- for test_file in $(find ./unit/test_hpu -name "test*.py" | sort); do+ local hpu_tests+ hpu_tests=$(filter_changed_tests "test" "$(find ./unit/test_hpu -name "test*.py" | sort)")+ if [ -z "${hpu_tests}" ]; then+ echo "No changed HPU test file, skip."+ return 0+ fi++ for test_file in ${hpu_tests}; dolocal test_basename=$(basename ${test_file} .py)echo "##[group]Running ${test_file} in HPU lazy mode..."@@ -55,21 +63,29 @@ function print_summary() {}function collect_log() {+ touch "${SUMMARY_LOG}"python /auto-round/.azure-pipelines/scripts/ut/collect_result.py \--test-type "Unit Tests" --log-pattern "unittest_*.log" --log-dir ${LOG_DIR} --summary-log ${SUMMARY_LOG}- cp .coverage "${LOG_DIR}/.coverage"- python -m coverage xml -o "${LOG_DIR}/coverage.xml"- python -m coverage html -d "${LOG_DIR}/htmlcov"+ if [ -f .coverage ]; then+ cp .coverage "${LOG_DIR}/.coverage"+ python -m coverage xml -o "${LOG_DIR}/coverage.xml"+ python -m coverage html -d "${LOG_DIR}/htmlcov"+ else+ echo "No coverage data (no test selected), skip coverage report."+ echo "##vso[task.setvariable variable=HAS_COVERAGE]false"+ fi}function print_coverage() {echo "##[group]overall code coverage..."- python -m coverage report+ [ -f .coverage ] && python -m coverage report || echo "No coverage data."echo "##[endgroup]"}function main() {setup_environment+ init_changed_tests+ scope_changed_tests "$(cd /auto-round && find test/unit/test_hpu -name "test_*.py" 2>/dev/null)"run_unit_testcollect_logprint_coveragediff --git a/.azure-pipelines/scripts/ut/run_ut_xpu.sh b/.azure-pipelines/scripts/ut/run_ut_xpu.shindex 642f1172f..edc06dfc3 100644--- a/.azure-pipelines/scripts/ut/run_ut_xpu.sh+++ b/.azure-pipelines/scripts/ut/run_ut_xpu.sh@@ -2,6 +2,10 @@set -esource /auto-round/.azure-pipelines/scripts/change_color.sh+source /auto-round/.azure-pipelines/scripts/ut/detect_changed_tests.sh++TIMEOUT=30+SESSION_TIMEOUT=600function setup_environment() {echo "##[group]set up UT env..."@@ -33,25 +37,29 @@ function setup_environment() {function run_unit_test() {auto_round_path=$(python -c 'import auto_round; print(auto_round.__path__[0])')- for test_file in $(find ./unit/test_ark -name "test*.py" | sort); do+ local ark_tests xpu_tests+ ark_tests=$(filter_changed_tests "test" "$(find ./unit/test_ark -name "test*.py" | sort)")+ xpu_tests=$(filter_changed_tests "test" "$(find ./unit/test_xpu -name "test*.py" | sort)")++ for test_file in ${ark_tests}; dolocal test_basename=$(basename ${test_file} .py)echo "##[group]Running ark ${test_file}..."local ut_log_name="${LOG_DIR}/unittest_ark_${test_basename}.log"numactl --physcpubind="${NUMA_CPUSET:-0-27}" --membind="${NUMA_NODE:-0}" \- pytest --timeout=30 --session-timeout=600 \+ pytest --timeout=${TIMEOUT} --session-timeout=${SESSION_TIMEOUT} \--cov="${auto_round_path}" --cov-report= --cov-append -vs \--junitxml="${ut_log_name%.log}.xml" ${test_file} 2>&1 | tee ${ut_log_name}echo "##[endgroup]"done- for test_file in $(find ./unit/test_xpu -name "test*.py" | sort); do+ for test_file in ${xpu_tests}; dolocal test_basename=$(basename ${test_file} .py)echo "##[group]Running xpu ${test_file}..."local ut_log_name="${LOG_DIR}/unittest_xpu_${test_basename}.log"numactl --physcpubind="${NUMA_CPUSET:-0-27}" --membind="${NUMA_NODE:-0}" \- pytest --timeout=30 --session-timeout=600 \+ pytest --timeout=${TIMEOUT} --session-timeout=${SESSION_TIMEOUT} \--cov="${auto_round_path}" --cov-report= --cov-append -vs \--junitxml="${ut_log_name%.log}.xml" ${test_file} 2>&1 | tee ${ut_log_name}echo "##[endgroup]"@@ -59,6 +67,13 @@ function run_unit_test() {}function run_unit_test_llmc() {+ local llmc_tests+ llmc_tests=$(filter_changed_tests "test" "$(find ./integration/test_xpu -name "test_llmc_integration.py" | sort)")+ if [ -z "${llmc_tests}" ]; then+ echo "No changed XPU LLMC test file, skip."+ return 0+ fi+echo "##[group]set up llmc UT env..."BUILD_TYPE="nightly" uv pip install -r ./unit/test_xpu/requirements_llmc.txtuv pip list@@ -66,13 +81,13 @@ function run_unit_test_llmc() {auto_round_path=$(python -c 'import auto_round; print(auto_round.__path__[0])')- for test_file in $(find ./integration/test_xpu -name "test_llmc_integration.py" | sort); do+ for test_file in ${llmc_tests}; dolocal test_basename=$(basename ${test_file} .py)echo "##[group]Running xpu llmc ${test_file}..."local ut_log_name="${LOG_DIR}/unittest_xpu_${test_basename}.log"numactl --physcpubind="${NUMA_CPUSET:-0-27}" --membind="${NUMA_NODE:-0}" \- pytest --timeout=30 --session-timeout=600 \+ pytest --timeout=${TIMEOUT} --session-timeout=${SESSION_TIMEOUT} \--cov="${auto_round_path}" --cov-report= --cov-append -vs \--junitxml="${ut_log_name%.log}.xml" ${test_file} 2>&1 | tee ${ut_log_name}echo "##[endgroup]"@@ -85,23 +100,34 @@ function print_summary() {}function collect_log() {+ touch "${SUMMARY_LOG}"python /auto-round/.azure-pipelines/scripts/ut/collect_result.py \--test-type "Unit Tests" --log-pattern "unittest_*.log" --log-dir ${LOG_DIR} --summary-log ${SUMMARY_LOG}- cp .coverage "${LOG_DIR}/.coverage"- python -m coverage xml -o "${LOG_DIR}/coverage.xml"- python -m coverage html -d "${LOG_DIR}/htmlcov"+ if [ -f .coverage ]; then+ cp .coverage "${LOG_DIR}/.coverage"+ python -m coverage xml -o "${LOG_DIR}/coverage.xml"+ python -m coverage html -d "${LOG_DIR}/htmlcov"+ else+ echo "No coverage data (no test selected), skip coverage report."+ echo "##vso[task.setvariable variable=HAS_COVERAGE]false"+ fi}function print_coverage() {echo "##[group]overall code coverage..."- python -m coverage report+ [ -f .coverage ] && python -m coverage report || echo "No coverage data."echo "##[endgroup]"}function main() {setup_environment- run_unit_test- run_unit_test_llmc+ init_changed_tests+ scope_changed_tests "$(cd /auto-round && find test/unit/test_ark test/unit/test_xpu test/integration/test_xpu -name "test_*.py" 2>/dev/null)"+ if [[ "${UT_MODE}" == "llmc" ]]; then+ run_unit_test_llmc+ else+ run_unit_test+ ficollect_logprint_coverageprint_summarydiff --git a/.azure-pipelines/template/ut-template.yml b/.azure-pipelines/template/ut-template.ymlindex 7a3a57794..0c87ce50b 100644--- a/.azure-pipelines/template/ut-template.yml+++ b/.azure-pipelines/template/ut-template.yml@@ -32,6 +32,11 @@ parameters:- name: buildARKWheeltype: stringdefault: "false"+ # Opt-in: on PR builds only run the test files touched by the PR. Off by+ # default so the fast PR unit-test pipeline keeps running the full suite.+ - name: filterChangedTests+ type: boolean+ default: falsesteps:@@ -100,9 +105,15 @@ steps:- script: |set -eo pipefail- docker exec -e NUMA_NODE=${NUMA_NODE} -e NUMA_CPUSET=${NUMA_CPUSET} -e ZE_AFFINITY_MASK=${ZE_AFFINITY_MASK} ${{ parameters.utContainerName }} \- bash -c "cd /auto-round/.azure-pipelines/scripts \- && bash ut/${{ parameters.utScriptFileName }}.sh ${{ parameters.utTestMode }}"+ docker exec -e NUMA_NODE=${NUMA_NODE} \+ -e NUMA_CPUSET=${NUMA_CPUSET} \+ -e ZE_AFFINITY_MASK=${ZE_AFFINITY_MASK} \+ -e UT_MODE=${{ parameters.utTestMode }} \+ -e BUILD_REASON=${BUILD_REASON} \+ -e SYSTEM_PULLREQUEST_TARGETBRANCH=${SYSTEM_PULLREQUEST_TARGETBRANCH} \+ -e FILTER_CHANGED_TESTS=${{ lower(parameters.filterChangedTests) }} \+ ${{ parameters.utContainerName }} \+ bash -c "cd /auto-round/.azure-pipelines/scripts && bash ut/${{ parameters.utScriptFileName }}.sh"displayName: "Run UT"- task: PublishPipelineArtifact@1diff --git a/.azure-pipelines/unit-test-cuda.yml b/.azure-pipelines/unit-test-cuda.ymlindex 5f4420304..9db081f61 100644--- a/.azure-pipelines/unit-test-cuda.yml+++ b/.azure-pipelines/unit-test-cuda.yml@@ -14,6 +14,7 @@ pr:- setup.py- requirements.txt- .azure-pipelines/scripts/cuda_unit_test+ - .azure-pipelines/scripts/ut/detect_changed_tests.sh- .azure-pipelines/unit-test-cuda.ymlexclude:@@ -140,6 +141,7 @@ stages:- script: |export UV_NO_CACHE=0+ export FILTER_CHANGED_TESTS=truebash .azure-pipelines/scripts/cuda_unit_test/run_cuda_ut.sh \--test_case=ci \--test_part=${{ pair.value.PART }}diff --git a/.azure-pipelines/unit-test-hpu.yml b/.azure-pipelines/unit-test-hpu.ymlindex 2dfd3c67d..dd07f78dc 100644--- a/.azure-pipelines/unit-test-hpu.yml+++ b/.azure-pipelines/unit-test-hpu.yml@@ -14,6 +14,7 @@ pr:- setup.py- requirements-hpu.txt- .azure-pipelines/scripts/ut/run_ut_hpu.sh+ - .azure-pipelines/scripts/ut/detect_changed_tests.sh- .azure-pipelines/template/docker-template.yml- .azure-pipelines/template/ut-template.ymlexclude:@@ -46,6 +47,7 @@ stages:utScriptFileName: "run_ut_hpu"uploadPath: $(UPLOAD_PATH)utArtifact: "ut"+ filterChangedTests: true- task: UseDotNet@2displayName: 'Use .NET Core sdk 7.0.x'@@ -53,5 +55,6 @@ stages:version: 7.0.x- task: PublishCodeCoverageResults@2+ condition: ne(variables['HAS_COVERAGE'], 'false')inputs:summaryFileLocation: $(UPLOAD_PATH)/coverage.xmldiff --git a/.azure-pipelines/unit-test-xpu.yml b/.azure-pipelines/unit-test-xpu.ymlindex 21758e837..29acaf623 100644--- a/.azure-pipelines/unit-test-xpu.yml+++ b/.azure-pipelines/unit-test-xpu.yml@@ -16,6 +16,7 @@ pr:- .azure-pipelines/template/ut-template.yml- .azure-pipelines/unit-test-xpu.yml- .azure-pipelines/scripts/ut/run_ut_xpu.sh+ - .azure-pipelines/scripts/ut/detect_changed_tests.sh- .azure-pipelines/docker/Dockerfile_xpu.devel- .azure-pipelines/build-auto-round-lib.yml- .azure-pipelines/template/lib-build-template.yml@@ -61,6 +62,7 @@ stages:dockerFileName: $(DOCKERFILE_NAME)utContainerName: "AutoRoundUnitTest$(NODE_LABEL)"buildARKWheel: $(BUILD_ARK_WHEEL)+ filterChangedTests: true- task: UseDotNet@2displayName: 'Use .NET Core sdk 7.0.x'@@ -68,5 +70,6 @@ stages:version: 7.0.x- task: PublishCodeCoverageResults@2+ condition: ne(variables['HAS_COVERAGE'], 'false')inputs:summaryFileLocation: $(UPLOAD_PATH)/coverage.xmldiff --git a/.azure-pipelines/unit-test.yml b/.azure-pipelines/unit-test.ymlindex eb10a27bd..18a7bc356 100644--- a/.azure-pipelines/unit-test.yml+++ b/.azure-pipelines/unit-test.yml@@ -84,6 +84,7 @@ stages:utTestMode: $(PART)utContainerName: "AutoRoundUnitTest$(NODE_LABEL)"buildARKWheel: $(BUILD_ARK_WHEEL)+ filterChangedTests: true- stage: CoveragedisplayName: "Collect Coverage"diff --git a/.azure-pipelines/weekly-test-cuda.yml b/.azure-pipelines/weekly-test-cuda.ymldeleted file mode 100644index a53949126..000000000--- a/.azure-pipelines/weekly-test-cuda.yml+++ /dev/null@@ -1,195 +0,0 @@-# Weekly CUDA pipeline: runs the heavy GPU integration + e2e suites-# (vLLM / SGLang / LLMCompressor and the throughput e2e tests) that are-# excluded from the PR CUDA unit-test pipeline. Scheduled weekly; can also be-# triggered manually from the Azure DevOps UI.--trigger: none-pr: none--schedules:- - cron: "0 18 * * 0" # 02:00 Asia/Shanghai every Sunday- displayName: Weekly CUDA integration + e2e run- branches:- include:- - main- always: true--variables:- - name: POOL_NAME- value: "RunPod-GPU-Pool"- - name: AGENT_NAME- value: "runpod-agent-$(Build.BuildId)"--parameters:- - name: matrix- type: object- default:- part0:- PART: 0--stages:- - ${{ each pair in parameters.matrix }}:- - stage: Start_Pod_${{ pair.key }}- displayName: "Start Pod ${{ pair.value.PART }}"- dependsOn: []- jobs:- - job: Start_Pod_Job- displayName: "Launch Pod & Run CUDA Tests"- timeoutInMinutes: 60- pool:- vmImage: "ubuntu-latest"- steps:- - checkout: self-- - task: UsePythonVersion@0- inputs:- versionSpec: "3.13"- displayName: "Use Python 3.13"-- - task: Bash@3- displayName: "Install dependencies"- inputs:- targetType: "inline"- script: |- pip install requests-- - task: Bash@3- displayName: "Create RunPod Instance"- name: create_runpod- inputs:- targetType: "inline"- script: |- export PYTHONUNBUFFERED=1- cd ${BUILD_SOURCESDIRECTORY}- CUDA_VERSION="13.0"- python .azure-pipelines/scripts/cuda_unit_test/runpod_manager.py \- --action create \- --api_key $(RUNPOD_API_KEY) \- --name "weekly-$(Build.BuildId)-${{ pair.value.PART }}" \- --container_disk_size "100" \- --part ${{ pair.value.PART }} \- --cuda_version "${CUDA_VERSION}" \- --env AZP_URL=$(System.CollectionUri) AZP_TOKEN=$(AZP_DEVOPS_PAT) AZP_POOL=$(POOL_NAME) AZP_AGENT_NAME="$(AGENT_NAME)-part${{ pair.value.PART }}"-- - task: Bash@3- displayName: "Wait for Pod to be Online"- name: wait_for_pod- inputs:- targetType: "inline"- script: |- cd ${BUILD_SOURCESDIRECTORY}- export PYTHONUNBUFFERED=1- python .azure-pipelines/scripts/cuda_unit_test/runpod_manager.py \- --action wait \- --api_key $(RUNPOD_API_KEY) \- --name "weekly-$(Build.BuildId)-${{ pair.value.PART }}"-- - task: Bash@3- displayName: "Wait for Agent to be Online"- name: wait_for_agent- inputs:- targetType: "inline"- script: |- cd ${BUILD_SOURCESDIRECTORY}- export PYTHONUNBUFFERED=1- python .azure-pipelines/scripts/cuda_unit_test/azure_agent.py \- --action wait \- --url "$(System.CollectionUri)" \- --pat "$(AZP_DEVOPS_PAT)" \- --pool "$(POOL_NAME)" \- --agent "$(AGENT_NAME)-part${{ pair.value.PART }}"-- - stage: GPU_Test_${{ pair.key }}- displayName: "GPU Test ${{ pair.value.PART }}"- dependsOn: Start_Pod_${{ pair.key }}- jobs:- - job: GpuTests- timeoutInMinutes: 180- pool:- name: "$(POOL_NAME)"- demands: Agent.Name -equals runpod-agent-$(Build.BuildId)-part${{ pair.value.PART }}- steps:- - checkout: self- clean: true- displayName: "Checkout Repo"-- - script: |- echo "Running on GPU Agent: $(Agent.Name)"- nvidia-smi- lscpu- cd .azure-pipelines/scripts/cuda_unit_test- nohup uv run monitor_gpu.py daemon > monitor.log 2>&1 &- displayName: "Verify GPU Access"-- - script: |- echo "##[group]Installing Python dependencies..."- uv pip install -U "pytest-cov" "pytest-html" "requests" "huggingface_hub"- hf auth login --token "$(HF_TOKEN)"- echo "##[endgroup]"- displayName: "Install Python Dependencies"-- - script: |- export UV_NO_CACHE=0- bash .azure-pipelines/scripts/cuda_unit_test/run_cuda_ut.sh \- --test_case=specific- displayName: "Run GPU Integration + E2E Tests"-- - script: |- cd .azure-pipelines/scripts/cuda_unit_test- uv run monitor_gpu.py stop- condition: always()- displayName: "GPU Monitor"-- - stage: Cleanup_Verification_${{ pair.key }}- displayName: "Verify Pod Cleanup ${{ pair.value.PART }}"- dependsOn:- - Start_Pod_${{ pair.key }}- - GPU_Test_${{ pair.key }}- condition: always()- jobs:- - job: Verify_Cleanup- pool:- vmImage: "ubuntu-latest"- steps:- - checkout: self-- - task: UsePythonVersion@0- inputs:- versionSpec: "3.13"- displayName: "Use Python 3.13"-- - task: Bash@3- displayName: "Install dependencies"- condition: always()- inputs:- targetType: "inline"- script: |- pip install requests-- - task: Bash@3- displayName: "Terminate RunPod Instance"- condition: always()- inputs:- targetType: "inline"- script: |- echo "Terminating Pod: weekly-$(Build.BuildId)-${{ pair.value.PART }}"- export PYTHONUNBUFFERED=1- python .azure-pipelines/scripts/cuda_unit_test/runpod_manager.py \- --action terminate \- --api_key $(RUNPOD_API_KEY) \- --name "weekly-$(Build.BuildId)-${{ pair.value.PART }}"-- - task: Bash@3- displayName: "Deregister Agent from Pool"- condition: always()- inputs:- targetType: "inline"- script: |- cd ${BUILD_SOURCESDIRECTORY}- export PYTHONUNBUFFERED=1- python .azure-pipelines/scripts/cuda_unit_test/azure_agent.py \- --action deregister \- --url "$(System.CollectionUri)" \- --pat "$(AZP_DEVOPS_PAT)" \- --pool "$(POOL_NAME)" \- --agent "$(AGENT_NAME)-part${{ pair.value.PART }}"diff --git a/test/README.md b/test/README.mdindex 28947fb10..f9dddd78c 100644--- a/test/README.md+++ b/test/README.md@@ -3,7 +3,7 @@This project uses `pytest` for unit testing. All test cases are under the `test/` directory.## 1. Environment Setup-- Recommended Python 3.8 or above.+- Recommended Python 3.10 or above.- Install dependencies:```shpip install -r ../requirements.txt@@ -17,9 +17,9 @@ Tests are split into three **tiers**, then by hardware backend- **`unit/`** — fast, self-contained tests. Run on **every PR** (`unit-test*.yml`).- **`integration/`** — tests against third-party frameworks (vLLM, SGLang,- LLMCompressor, INC, HuggingFace). Run **nightly** (`nightly-test*.yml`).-- **`e2e/`** — full-model / real inference-engine tests. Run **weekly**- (`weekly-test*.yml`).+ LLMCompressor, INC, HuggingFace). Run **nightly** — CPU via+ `nightly-test-cpu.yml`, XPU via `nightly-test-xpu.yml`.+- **`e2e/`** — full-model / real inference-engine tests. Run **manually**Within `unit/test_cpu/` and `unit/test_cuda/`, tests are grouped by functionality:diff --git a/test/integration/README.md b/test/integration/README.mdindex 6aa8b6773..100e30d67 100644--- a/test/integration/README.md+++ b/test/integration/README.md@@ -19,4 +19,3 @@ These tests run in the **nightly** CI pipelines:- CPU integration → `.azure-pipelines/nightly-test.yml`- XPU integration → `.azure-pipelines/nightly-test-xpu.yml`-- CUDA integration (vLLM / SGLang / LLMCompressor) → `.azure-pipelines/weekly-test-cuda.yml`diff --git a/test/unit/test_cpu/utils/test_model_scope.py b/test/unit/test_cpu/utils/test_model_scope.pyindex 17cfd497b..5d15a1247 100644--- a/test/unit/test_cpu/utils/test_model_scope.py+++ b/test/unit/test_cpu/utils/test_model_scope.py@@ -30,15 +30,13 @@ class TestModelScope:if os.path.exists(self.cache_path):shutil.rmtree(self.cache_path, ignore_errors=True)- @pytest.mark.skip_ci(reason="https://github.com/intel/auto-round/issues/2127")@pytest.mark.timeout(120)def test_llm(self, dataloader):model_name = get_model_path("Qwen/Qwen2.5-0.5B-Instruct")autoround = AutoRound(model_name, platform="model_scope", scheme="w4a16", iters=0, seqlen=2, dataset=dataloader)autoround.quantize_and_save()- @pytest.mark.skip_ci(reason="https://github.com/intel/auto-round/issues/2127")- @pytest.mark.timeout(360)+ @pytest.mark.timeout(480)def test_mllm(self, dataloader):model_name = get_model_path("Qwen/Qwen2-VL-2B-Instruct")autoround = AutoRound(